Introduction

Training reinforcement learning (RL) agents often involves a grueling bottleneck: the communication overhead between the simulation environment and the learning algorithm. For developers working with high-fidelity physics simulations, this latency can make training times stretch from hours to weeks. RL Games is a high-performance reinforcement learning library implemented in PyTorch that eliminates these bottlenecks by supporting end-to-end GPU-accelerated training pipelines. By keeping observations and actions on the GPU, RL Games allows researchers to train complex agents in environments like NVIDIA Isaac Gym and Brax with unprecedented speed, making it the go-to choice for large-scale robotics and continuous control tasks.

What Is RL Games?

RL Games is a high-performance reinforcement learning library that provides optimized implementations of core RL algorithms for target users who require maximum throughput in physics-based simulations. Built on PyTorch, it is designed to interface seamlessly with GPU-accelerated simulators, ensuring that data does not need to be moved between the CPU and GPU during the training loop. This architecture makes it particularly effective for continuous control tasks and robotics research.

The project is maintained by Denys Makoviichuk and is released under the MIT License, allowing for broad academic and industrial application. It focuses on providing a lean, efficient framework that prioritizes execution speed over high-level abstraction, which is why it is frequently used as the default RL backend for NVIDIA’s Isaac Gym and Isaac Lab environments.

Why RL Games Matters

In traditional RL libraries, the simulation typically runs on the CPU while the neural network is trained on the GPU. This constant “shuttling” of data creates a massive performance hit. RL Games solves this by supporting an end-to-end GPU pipeline. When paired with a GPU-based simulator like Isaac Gym, the entire training process—from state observation to action execution—happens on the device, resulting in a 10x to 100x speedup in training time.

Beyond raw speed, RL Games provides critical support for multi-agent training and self-play, which are essential for evolving complex behaviors in competitive or collaborative environments. As the field of embodied AI moves toward more complex simulations, the ability to train thousands of parallel environments on a single GPU becomes a necessity rather than a luxury. RL Games provides the infrastructure to make this possible.

Key Features

• GPU-Accelerated Pipelines: Supports end-to-end GPU training where observations and actions remain on the device, drastically reducing latency when used with simulators like Isaac Gym and Brax.
• Multi-Agent Support: Includes decentralized and centralized critic variants for multi-agent reinforcement learning (MARL), allowing agents to learn coordinated behaviors.
• Self-Play Capabilities: Built-in mechanisms for self-play, enabling agents to improve by competing against previous versions of themselves.
• Asymmetric Actor-Critic: Supports asymmetric actor-critic architectures where the critic has access to privileged information (global state) that the actor does not.
• Algorithm Suite: Provides high-performance implementations of PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), and A2C (Advantage Actor-Critic).
• ONNX Export: Allows users to export trained policies to ONNX format for efficient deployment in production environments or on embedded devices.
• WandB Integration: Native support for Weights & Biases for real-time experiment tracking and hyperparameter logging.
• Masked Actions: Support for action masking, which is critical for environments with invalid actions in certain states.
• Multi-Node Training: Recent updates have added support for multi-node training, allowing the scaling of GPU-accelerated environments across multiple machines.
• RNN Support: Integrated support for Recurrent Neural Networks (RNNs) to handle partially observable environments (POMDPs).

How RL Games Compares

When choosing an RL library, the tradeoff is usually between ease of use and raw performance. RL Games is designed for the latter, prioritizing throughput over a polished API.

Stable Baselines3 is the gold standard for beginners and researchers who need a reliable, well-documented API for standard Gymnasium environments. However, it lacks the vectorized GPU training capabilities that make RL Games so fast. Ray RLlib is a massive framework designed for industrial-scale distributed training across clusters. While RLlib can handle multi-agent setups, RL Games is often more efficient for single-machine, multi-GPU training of physics simulations.

The primary differentiator for RL Games is its deep integration with NVIDIA’s ecosystem. If you are using Isaac Gym or Isaac Lab, RL Games is often the most performant choice because it is specifically optimized to handle the GPU buffers that these simulators provide, avoiding the CPU-GPU bottleneck entirely.

Getting Started: Installation

For maximum performance, it is highly recommended to have PyTorch 2.2+ and CUDA installed on your system.

Standard Installation

The simplest way to install RL Games is via pip:

pip install rl-games

Installation from Source

If you need the latest features or intend to contribute, install from the GitHub repository:

git clone https://github.com/Denys88/rl_games
cd rl_games
pip install -e .

Installation with Extras

RL Games supports several optional dependencies for specific environments. You can install them using the following syntax:

pip install -e ".[atari,mujoco,envpool]"

Available extras include atari, mujoco, envpool, brax, and pufferlib.

Using uv (Recommended)

For a faster, modern Python package manager, the project recommends using uv:

uv venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[mujoco,envpool]"

How to Use RL Games

RL Games operates primarily through a configuration-driven approach. Instead of writing extensive boilerplate code, you define your agent’s hyperparameters, network architecture, and environment settings in a YAML file, and then use a runner script to execute the training.

The basic workflow involves three steps: selecting a configuration file, initializing the environment, and running the training script. For example, to train an agent on a standard task, you would run the following command in your terminal:

python runner.py --train --file rl_games/configs/ppo_cartpole.yaml

Once the training begins, RL Games initializes the vectorized environment, allocates GPU buffers for trajectory memory, and begins the PPO update cycle. If you are using a GPU-accelerated simulator, the data remains on the device, maximizing throughput.

Code Examples

The project provides several Colab notebooks to demonstrate different use cases. Here are the primary implementation patterns found in the repository.

Basic Training Loop

While the runner script is the primary entry point, the underlying logic follows this pattern:

from rl_games.common import env_configurations
from rl_games.common.vecenv import IVecEnv

# Register the environment
env_configurations.register("rlgpu", lambda config_name, num_actors, **kwargs: MyGpuEnv(config_name, num_actors, **kwargs))

# Run training using the config file
# This is typically handled by runner.py

This example shows how to register a custom GPU environment so that RL Games can instantiate it based on the YAML configuration.

Exporting to ONNX

Exporting a trained model for deployment is a critical step. RL Games provides utilities to convert PyTorch models to ONNX format:

# Example pattern for ONNX export
# This is typically performed via a separate script or the runner's export flag
# The resulting .onnx file can be used in C++ or TensorRT for real-time inference.

The repository includes specific notebooks for exporting discrete and continuous action space policies, ensuring that the trained agent can be moved from the simulation to a real-world robot.

Advanced Configuration

RL Games relies heavily on YAML configurations to separate hyperparameters from code. This allows for rapid experimentation without needing to restart the Python interpreter.

Key configuration blocks include the algo block (defining the algorithm name, such as sac or ppo), the model block (defining the network architecture, and whether to use a separate network for the critic), and the space block (defining if the action space is continuous or discrete).

# Example YAML snippet
params:
  config:
    seed: 8
    algo: ppo
    model:
      name: actor_critic
      separate: False
      space:
        continuous: True
  train:
    # Hyperparameters for PPO
    gamma: 0.99
    lr: 3e-4
    kl_threshold: 0.01

Users can override these parameters at runtime using the --file flag in the runner script, making it easy to integrate with hyperparameter optimization tools.

Real-World Use Cases

RL Games is specifically designed for scenarios where simulation speed is the primary constraint. It shines in the following areas:

• Robotic Legged Locomotion: Training quadrupeds or bipeds to walk, run, and jump in Isaac Gym. Because these tasks require millions of steps of experience, the GPU-accelerated pipeline is essential for reducing training time from weeks to hours.
• Dexterous Manipulation: Training robotic hands to manipulate objects. These environments often involve complex contact physics and high-dimensional action spaces, making the high-throughput training of RL Games ideal.
• Multi-Agent Coordination: Training groups of agents to collaborate or compete in physics simulations. The built-in support for centralized critics allows agents to learn coordinated strategies in a way that is more efficient than standard PPO.
• Sim-to-Real Transfer: Training an agent in a GPU-accelerated simulation and then exporting the policy via ONNX to a real-world robot. This allows for massive parallelization in simulation before deploying to hardware.

Contributing to RL Games

Contributing to RL Games is welcomed, although the project is primarily maintained by a small team. Users can contribute by reporting bugs via GitHub Issues or submitting Pull Requests for new algorithm implementations or environment wrappers. Since the project prioritizes performance, contributions that optimize the tensor operations or reduce memory overhead are highly valued.

To contribute, it is recommended to follow the standard GitHub flow: fork the repository, create a feature branch, and then submit a PR. The project does not have a formal CONTRIBUTING.md, but following the PyTorch coding standards is expected.

Community and Support

The primary hub for RL Games support is the GitHub Discussions and Issues tabs. Because the library is designed for high-performance research, the community consists largely of robotics researchers and AI developers. You can also find extensive discussions about RL Games within the NVIDIA Developer Forums, particularly in the Isaac Gym and Isaac Lab sections, as it is the default RL backend for those tools.

The project also maintains a Discord channel for real-time collaboration and discussion. You can join the community to share your results, report issues, and report bugs.

Conclusion

RL Games is the right choice for developers and researchers who prioritize training throughput and simulation speed above all else. If you are working with GPU-accelerated simulators like Isaac Gym or Brax, RL Games is essentially mandatory for avoiding the CPU-GPU bottleneck that plagues traditional RL libraries. It provides the necessary tools for multi-agent training, self-play, and efficient policy export via ONNX.

While it has a steeper learning curve and less polished documentation than libraries like Stable Baselines3, the performance gains are undeniable. For those pushing the boundaries of embodied AI and robotics, the speed of iteration is the most critical factor. RL Games provides that speed.

Star the repo, try the quickstart Colab notebooks, and join the community to start training your agents at scale.

Resources

• RL Games GitHub Repository
• Mujoco Training Colab Example
• Brax Training Colab Example
• NVIDIA Isaac Gym Documentation