High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC, PPG)
Alternatives To Cleanrl
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Baselines13,5483722 months ago6February 26, 2018490mitPython
OpenAI Baselines: high-quality implementations of reinforcement learning algorithms
Reinforcement Learning With Tensorflow7,469
8 months ago58mitPython
Simple Reinforcement learning tutorials, 莫烦Python 中文AI教学
Easy Rl6,137
16 hours ago41otherJupyter Notebook
Tianshou5,99644 days ago29July 04, 202242mitPython
An elegant PyTorch deep reinforcement learning library.
Deep Reinforcement Learning4,419
21 days ago2mitJupyter Notebook
Repo for the Deep Reinforcement Learning Nanodegree program
Reinforcement Learning3,637
3 years ago2mitJupyter Notebook
Learn Deep Reinforcement Learning in 60 days! Lectures & Code in Python. Reinforcement Learning + Deep Learning
5 months ago5mitPython
Modularized Implementation of Deep RL Algorithms in PyTorch
Deep Reinforcement Learning With Pytorch2,741
5 days ago26mitPython
PyTorch implementation of DQN, AC, ACER, A2C, A3C, PG, DDPG, TRPO, PPO, SAC, TD3 and ....
Elegantrl2,715110 days ago3January 08, 202287otherPython
Cloud-native Deep Reinforcement Learning. 🔥
6 hours ago53otherPython
High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC, PPG)
Alternatives To Cleanrl
Select To Compare

Alternative Project Comparisons

CleanRL (Clean Implementation of RL Algorithms)

tests docs Code style: black Imports: isort Open In Colab

CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:

  • 📜 Single-file implementation
    • Every detail about an algorithm variant is put into a single standalone file.
    • For example, our only has 340 lines of code but contains all implementation details on how PPO works with Atari games, so it is a great reference implementation to read for folks who do not wish to read an entire modular library.
  • 📊 Benchmarked Implementation (7+ algorithms and 34+ games at
  • 📈 Tensorboard Logging
  • 🪛 Local Reproducibility via Seeding
  • 🎮 Videos of Gameplay Capturing
  • 🧫 Experiment Management with Weights and Biases
  • 💸 Cloud Integration with docker and AWS

You can read more about CleanRL in our JMLR paper and documentation.

CleanRL only contains implementations of online deep reinforcement learning algorithms. If you are looking for offline algorithms, please check out tinkoff-ai/CORL, which shares a similar design philosophy as CleanRL.

ℹ️ Support for Gymnasium: Farama-Foundation/Gymnasium is the next generation of openai/gym that will continue to be maintained and introduce new features. Please see their announcement for further detail. We are migrating to gymnasium and the progress can be tracked in vwxyzjn/cleanrl#277.

⚠️ NOTE: CleanRL is not a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's varaint or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

Get started


To run experiments locally, give the following a try:

git clone && cd cleanrl
poetry install

# alternatively, you could use `poetry shell` and do
# `python run cleanrl/`
poetry run python cleanrl/ \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

# open another temrminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runs

To use experiment tracking with wandb, run

wandb login # only required for the first time
poetry run python cleanrl/ \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name cleanrltest

To run training scripts in other games:

poetry shell

# classic control
python cleanrl/ --env-id CartPole-v1
python cleanrl/ --env-id CartPole-v1
python cleanrl/ --env-id CartPole-v1

# atari
poetry install --with atari
python cleanrl/ --env-id BreakoutNoFrameskip-v4
python cleanrl/ --env-id BreakoutNoFrameskip-v4
python cleanrl/ --env-id BreakoutNoFrameskip-v4
python cleanrl/ --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
poetry install --with envpool
python cleanrl/ --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
poetry run python --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

# pybullet
poetry install --with pybullet
python cleanrl/ --env-id MinitaurBulletDuckEnv-v0
python cleanrl/ --env-id MinitaurBulletDuckEnv-v0
python cleanrl/ --env-id MinitaurBulletDuckEnv-v0

# procgen
poetry install --with procgen
python cleanrl/ --env-id starpilot
python cleanrl/ --env-id starpilot

# ppo + lstm
python cleanrl/ --env-id BreakoutNoFrameskip-v4

You may also use a prebuilt development environment hosted in Gitpod:

Open in Gitpod

Algorithms Implemented

Algorithm Variants Implemented
Proximal Policy Gradient (PPO), docs, docs, docs, docs, docs, docs, docs), docs, docs, docs, docs
Deep Q-Learning (DQN), docs, docs, docs, docs
Categorical DQN (C51), docs, docs, docs, docs
Soft Actor-Critic (SAC), docs, docs
Deep Deterministic Policy Gradient (DDPG), docs, docs
Twin Delayed Deep Deterministic Policy Gradient (TD3), docs, docs
Phasic Policy Gradient (PPG), docs
Random Network Distillation (RND), docs

Open RL Benchmark

To make our experimental data transparent, CleanRL participates in a related project called Open RL Benchmark, which contains tracked experiments from popular DRL libraries such as ours, Stable-baselines3, openai/baselines, jaxrl, and others.

Check out for a collection of Weights and Biases reports showcasing tracked DRL experiments. The reports are interactive, and researchers can easily query information such as GPU utilization and videos of an agent's gameplay that are normally hard to acquire in other RL benchmarks. In the future, Open RL Benchmark will likely provide an dataset API for researchers to easily access the data (see repo).

Support and get involved

We have a Discord Community for support. Feel free to ask questions. Posting in Github Issues and PRs are also welcome. Also our past video recordings are available at YouTube

Citing CleanRL

If you use CleanRL in your work, please cite our technical paper:

  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {}
Popular Dqn Projects
Popular Ppo Projects
Popular Machine Learning Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Video Game
Machine Learning
Deep Learning
Reinforcement Learning
Deep Reinforcement Learning
Actor Critic
Proximal Policy Optimization