Introduction
Reinforcement Learning is the third paradigm of Machine Learning which is conceptually quite different from the other supervised and unsupervised learning. Although we had a good number of libraries for supervised and unsupervised learning for a long time, it was not the case with reinforcement learning a few years back. Its algorithms had to be coded from scratch, but with its growing popularity many reinforcement learning libraries have come up that can make life easier for RL developers.
In this article, we will list down some useful reinforcement learning libraries that you should know.
Reinforcement Learning Libraries
OpenAI Baselines
OpenAI released a reinforcement learning library Baselines in 2017 to offer implementations of various RL algorithms. It supports the following RL algorithms – A2C, ACER, ACKTR, DDPG, DQN, GAIL, HER, PPO, TRPO.
Baselines let you train the model and also support a logger to help you visualize the training metrics. And after training the model can be saved for reuse. One drawback of the OpenAI Baseline is that it does not have a proper document to follow.
Its GitHub repository has 11K+ stars and is in maintenance mode only and has not seen development for quite some time.
Stable Baselines
Stable Baseline is a fork of OpenAI Baseline library with huge improvements over it. Stable Baseline has refactored and cleaned up the OpenAI Baseline code to bring a common structure and interface to the algorithms.
Apart from all the algorithms of OpenAI Baseline, Stable Baseline offers two additional algorithms Soft Actor-Critic (SAC) and Twin Delayed DDPG (TD3) plus it supports Tensorboard.
Most importantly, Stable Baseline has proper documentation and a good collection of examples to make it easy for people to get started with it.
Its GitHub repository has 3K stars and is actively maintained and developed.
TF Agents
TF Agents is a Tensorflow library for reinforcement learning that provides various RL components that can be easily used or modified as per needs. It provides the following RL algorithms – DQN, DDQN, DDPG, TD3, REINFORCE, PPO, SAC. TF Agents has built-in wrappers for various RL environments like the OpenAI Gym, DeepMind-control, and Atari.
Being a Tensorflow library, it is very well documented, and its GitHub repository with 1.8K stars is also quite active.
Keras-RL
Keras-RL is a deep reinforcement learning library for Keras that has implementations of state-of-art RL algorithms. It supports algorithms like DQN, DDPG, NAF, CEM, SARSA and has good documentation to explain their working in the library.
Keras-RL integrates with OpenAI Gym out of the box hence enables you can work with the Gym environments seamlessly.
Its GitHub repository has more than 5K stars but has not seen activities for quite some time now.
Keras-RL2
Keras-RL2 is the fork of the Keras-RL library that we discussed above. It came to existence thanks to the community to support Tensorflow 2 in the original Keras-RL library. It however does not have its own documentation but the documentation of Keras-RL can be used here.
Its GitHub repository has 130+ starts but is still active with recent updates.
PyQlearning
PyQlearning is a reinforcement learning library that focuses only on Q Learning as its name suggests. It supports both deep Q learning and multi-agent deep Q learning that can be optimized by Annealing models such as Simulated Annealing, Adaptive Simulated Annealing, and Quantum Monte Carlo Method.
This library was released in 2020 and its GitHub library has 150+ stars with active maintenance as of now. The documentation extensively covers the theory and its implementation in the library along with some examples to help you get started.
Tensorforce
Tensorforce is an open-source deep reinforcement learning library built on top of the Tensorflow library. The main highlight of this library is the modularized design for ease of use. It offers implementation of various RL algorithms like DQN, Policy Gradient, Actor-Critic, etc. along with support for Tensorboard for visualization.
Another good thing about Tensorforce is that you will find the support of various environments like OpenAI gym, Arcade Learning Environment, OpenAI Retro, Open Sim, PyGame Learning Environment, and Vizdoom.
Tensorforce is well documented and its GitHub repository is also quite active with close to 3K stars.
RL Coach
Reinforcement Learning Coach a.k.a RL-coach is a reinforcement learning library created by Intel AI Lab to provide implementations of various state-of-art RL algorithms. As evident from the above illustration, RL-coach supports almost all family of RL algorithms under value optimization & policy optimization types, and more. Interestingly, it also supports batch reinforcement learning where the agent can be trained using a dataset without any simulator.
It also provides Coach Board which is a visualization dashboard to track various learning parameters of the agents which can be useful for debugging purposes. Besides this, it supports various RL environments like OpenAI Gym, ViZDoom, Roboschool, GymExtensions, PyBullet, CARLA, Starcraft, Deepmind Control Suite.
RL-coach is well documented with examples to get you started and with close to 2K stars its GitHub repository still sees activity.
Chainer RL
Chainer RL is a reinforcement library built on the deep learning framework Chainer to implement various state-of-art RL algorithms. The list of implemented algorithms includes DQN, Categorical DQN, Rainbow, IQN, DDPG, A3C, ACER, NSQ, PPO, PCL, TRPO, TD3, SAC.
It also offers Chainer RL Visualizer to visualize various parameters of your agent’s behavior in a browser dashboard making it easy to debug.
With more than 900 stars, Chainer RL’s Github repository is still maintained actively by the community.
Mushroom RL
Mushroom RL a Python library for reinforcement learning that is simple yet powerful to run various RL algorithms like Q Learning, SARSA, FQI, DQN, DDPG, SAC, TD3, TRPO, PPO.
It provides a modular and common interface to let you train your agent on any library easily. You can also add your own custom algorithms with ease. Mushroom RL works on tensor computation so you can use it with both PyTorch or Tensorflow and it also supports GPU. It has support for OpenAI Gym, DeepMind Control Suite, and MuJoCo environments.
It is a relatively new RL library and its Github repository with more than 400+ stars is still actively maintained.
Acme
Acme is a reinforcement learning framework created by Deepmind that is targeted more towards researchers to help in rapid prototyping and develop novel RL algorithms. It offers a unified platform for researchers to reproduce the results of novel algorithms with ease.
As an addon, it also supports distributed agents environment for reinforcement learning. However, there is a lacking in the documentation part which can make it difficult to get started.
Acme was released in 2020 and its GitHub repository with 2K stars is quite active currently.
Dopamine
Dopamine is a Google reinforcement learning framework to help researchers with the fast prototyping of RL algorithms. It has been designed in a manner to provide a platform for easy experimentation, flexible development, and reproducibility of the results.
It supports limited agents only – C51, DQN, IQN, Quantile (JAX), Rainbow, and the training parameters of the agents can be visualized on Tensorboard.
Its GitHub repository has over 9K stars and it is still very actively developed and maintained.
RLLib
RLlib is a reinforcement learning library that provides high scalability and a unified API for a variety of RL applications. It supports both PyTorch and Tensorflow natively but most of its internal frameworks are agnostic. It supports more than 20 RL algorithms out of the box but some are exclusive either to Tensorflow or PyTorch. It also has inbuilt support for OpenAI Gym environments.
RLlib is popularly documented with various examples to help you kickstarted.
TRFL
TRFL is a Tensorflow based reinforcement learning framework that offers various building blocks for writing RL algorithms. These are not complete algorithms in themselves but functions like Loss functions for various RL algorithms. Moreover, it supports both CPU and GPU versions of Tensorflow.
TRFL is well documented and has more than 3K stars on GitHub although it has not seen activities for one year now.
ReAgent
ReAgent is a Facebook framework for reinforcement learning which was formerly also known as Horizon. It is built on top of PyTorch for creating models and uses TorchScript for serving them. Its offerings include various RL algorithms like Discrete Action DQN, Parametric-Action DQN, C5, TD, SAC, etc. Apart from this it also supports data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, and optimized serving.
It is well documented and its GitHub repository with 2.8K stars is still actively maintained.
MAgent
MAgent is quite different from other Reinforcement libraries in the sense that it not only supports multiple agents but can let you scale hundred to million agents. The idea is quite interesting but unfortunately, although its GitHub repository has 1.3K stars it is no longer maintained. However, there is another forked version of MAgent now maintained by Petting Zoo.
SLM Lab
SLM Lab is a reinforcement learning framework that offers modular components for flexible experimentations and reproducibility. It offers various RL algorithms like SARSA, DQN (and its variations), A2C, PPO, SAC, SIL, etc. All the algorithms have benchmark results and support hyperparameter search and result analysis.
SLM Labs at the moment supports RL environments of OpenAI Gym, OpenAI School, VizDoom, Unity Environments. Its documentation is quite robust with examples.
Its GitHub repository has 900+ stars and is somewhat still active.
DeeR
DeeR is a deep reinforcement learning library that provides several RL algorithm implementations using Keras. Some of the implementations include  Double Q-learning, prioritized Experience Replay, Deep deterministic policy gradient (DDPG), Combined Reinforcement via Abstract Representations (CRAR), etc. It also provides flexibility to modify the learning process or write your own algorithm using any other Python framework.
Its GitHub repository has 450+ stars and is still actively maintained, also it has good documentation to get started.
Garage
Garage is a reinforcement learning toolkit that lets you build your own reinforcement learning algorithms and also comes with implementations of state-of-art implementations of RL algorithms. Some of the algorithms include DQN, REINFORCE, SAC, TD3, DDPG, BG, CEM, ERWR, MAML, etc. These algorithms are mostly implemented in Tensorflow and PyTorch
It provides inbuilt support for RL environments such as OpenAI Gym, DeepMind, Metaworld, PyBullet. Garage also has support for Tensorboard to help you visualize the training logs of the agent.
Its GitHub repository has 1.1K stars with active development and its documentation is also quite good.
SURREAL
Surreal is a framework by Standford Vision and Learning Lab which enables distributed reinforcement learning. Even the simpler RL tasks like Atar games may take take a good amount of processing time to learn. Using Surreal the processing can take be scaled to thousands of CPUs and hundreds of GPUs with ease. It can be operated in two modes local mode and Kubernetes mode on Google cloud.
Surreal currently supports DDPG ad PPO RL algorithms. It also comes with support for OpenAI Gym, Deepmind Control Suite, and its own Surreal Robotics Suite environments.
Its GitHub repository has got 440 stars but has not seen much activity in the last couple of years and even its documentation could have been more elaborate.
RLGraph
Â
RLGraph is a reinforcement learning library to help you quickly prototype RL algorithms for research purposes or otherwise. It supports both TensorFlow and PyTorch with a common interface. Currently, RLGraph offers various RL algorithms such DQN, Double-DQN, Dueling-DQN, Prioritized experience replay, DQFD, SAC, Actor-Critic, etc.
It also provides integration with RL environments of OpenAI Gym, Deepmind Lab, and Simple Grid World. A good thing about RLGraph is that it is well documented but its GitHub repository with 272 stars has not seen activities since the last couple of years.
Â
- Also Read – 16 Reinforcement Learning Environments and Platforms You Did Not Know Exist
- Also Read – 7 Reinforcement Learning GitHub Repositories To Give You Project Ideas
Â
