Solving Gymnasium's Car Racing with Reinforcement Learning

Soft-Actor Critic (SAC)

Deep Q Learning (DQN)

Proximal Policy Optimization (PPO)

Results

Hardware: Google Colab T4

Model Type	Discrete	Average Reward	Training Time	Total Training Steps
PPO	No	809.98	3:39:20	751,615
SAC	No	610.67	6:29:16	333,116
DQN	Yes	897.77	5:41:22	750,000

Training Notes

• Set ent_coef for PPO as it encourages exploration of other actions. Stable Baselines3 defaults the value to 0.0. More Information
• Do not set your eval_freq too low, as it can sometimes cause instability during learning due to being interrupted by evaluation. (e.g. >=10,000)
• buffer_size defaults to 1,000,000, which requires a significant memory for DQN and SAC. Try setting it to a more practical value when using the original observation space (e.g., 200,000)
• Set the gray_scale flag in the notebooks to True to allow DQN and SAC to run without using the High-RAM option in Google Colab (buffer size <= 150,000). This converts the observation space from (96 x 96 x 3) images to (84 x 84) grayscale images.

Finding Theta Blog Posts:

• Solving Gymnasium's Car Racing with Reinforcement Learning