Introduction

Training reinforcement learning (RL) agents often hits a critical bottleneck: the speed of environment simulation. While GPUs accelerate neural network updates, the CPU-bound nature of environment stepping is frequently the slowest part of the training pipeline. EnvPool is a C++-based high-performance parallel environment execution engine designed to eliminate this bottleneck, enabling researchers to achieve millions of frames per second (FPS) on standard hardware. With its ability to scale from a laptop to high-end workstations like the NVIDIA DGX-A100, EnvPool transforms the iteration speed of RL experiments.

What Is EnvPool?

EnvPool is a C++-based batched environment pool that leverages pybind11 and a custom thread pool to provide high-performance parallel execution for general reinforcement learning environments. It is designed to be a drop-in replacement for standard vectorized environments, providing a Python frontend for ease of use while maintaining a high-speed C++ backend. The project is licensed under the Apache License 2.0 and is maintained by SAIL at Singapore.

By managing a pool of environments and interacting with them via batched APIs, EnvPool allows for the simultaneous execution of thousands of environments on a single machine. It supports both synchronous and asynchronous execution modes, making it compatible with a wide range of RL algorithms and training frameworks.

Why EnvPool Matters

In traditional RL setups, environment execution is often handled via Python subprocesses (e.g., gym.vector_env), which introduces significant overhead due to inter-process communication and Python’s Global Interpreter Lock (GIL). This overhead often leaves CPU cores underutilized and limits the total throughput of the training system.

EnvPool solves this by moving the environment logic into C++, effectively bypassing the GIL and utilizing a highly efficient thread pool. This results in a massive increase in throughput. For example, on a high-end machine, EnvPool can achieve 1 million frames per second for Atari games and 3 million steps per second for MuJoCo environments. Even on a laptop with 12 CPU cores, it typically provides a 3x throughput increase over Python subprocess-based environments.

This speedup allows researchers to iterate on ideas much faster. Tasks that previously took hours to train can now be completed in minutes. For instance, training an agent to play Atari Pong or MuJoCo Ant on a laptop can be reduced to just five minutes using EnvPool.

Key Features

• Extreme High Throughput: Achieves up to 1M Atari FPS and 3M MuJoCo FPS on high-end hardware, representing a ~20x increase over Python subprocess-based vectorized environments.
• C++ Backend with Python Frontend: Combines the execution speed of C++ with the flexibility of Python, allowing users to interact with the engine via a familiar Gymnasium-like API.
• Synchronous and Asynchronous Execution: Supports both sync and async modes, allowing developers to choose the execution model that best fits their algorithm’s requirements.
• Broad Environment Support: Provides native, high-performance implementations for Atari and MuJoCo, while offering a C++ API for developers to integrate custom environments.
• Wide Framework Compatibility: Seamlessly integrates with popular RL libraries such as Stable-Baselines3, Tianshou, ACME, CleanRL, and rl_games.
• XLA and JAX Support: Includes an XLA interface allowing for JAX jit-compiled functions to further optimize the interaction between the environment and the agent.
• Batched API Interaction: Interacts with environments in batches by default, reducing the number of calls between Python and C++ and further minimizing overhead.
• Multi-Platform Support: Provides pre-compiled wheels for Linux, macOS, and Windows for Python versions 3.11 through 3.14.

How EnvPool Compares

EnvPool is designed to solve the CPU bottleneck in RL. While GPU-based simulators like Brax or Isaac Gym provide even higher throughput by simulating everything on the GPU, they require the environment to be written in a specific GPU-compatible language (like JAX or PhysX). EnvPool is a general-purpose solution that speeds up traditional CPU-based environments.

The primary tradeoff is the development effort. While gym.vector_env is the easiest to implement, it is the slowest. EnvPool provides a massive speedup by moving the environment logic to C++, which is a higher barrier to entry for many ML researchers who are primarily Python-based. However, for those using standard environments like Atari or MuJoCo, EnvPool is a laught-out winner in terms of performance without requiring any C++ knowledge.

Getting Started: Installation

EnvPool is available on PyPI and supports Python 3.11 through 3.14 on Linux, macOS, and Windows.

PyPI Installation

The simplest way to install EnvPool is via pip:

pip install envpool

Build From Source

For developers who wish to customize the engine or add new environments, building from source is required. This process uses Bazel as the build system.

git clone https://github.com/sail-sg/envpool.git
cd envpool
# Follow the platform-specific build guidelines in the repository's Build From Source documentation

Prerequisites: On Linux, you may need to install the system Qt 5 runtime (e.g., apt install qtbase5-dev) if you are using Procgen environments.

How to Use EnvPool

EnvPool is designed to be compatible with the Gymnasium API, making it easy to integrate into existing RL workflows. The basic workflow involves creating a pool of environments using envpool.make and then interacting with them using the standard reset and step methods.

By default, EnvPool interacts with environments in batches. When you call step, you provide a batch of actions for all environments in the pool, and the engine returns a batch of observations, rewards, and dones.

If you are using a high-end machine with multiple NUMA nodes, you can further optimize performance by running multi-processing on each NUMA node to reduce thread contention on the ActionBufferQueue.

Code Examples

Below are examples of how to use EnvPool in synchronous mode with the Gymnasium API.

Basic Synchronous Execution

This example shows how to create a pool of 100 Atari Pong environments and step through them.

import envpool
import numpy as np

# Create a pool of 100 Pong environments
env = envpool.make("Pong-v5", env_type="gym", num_envs=100)

# Reset all environments to get initial observations
obs = env.reset()

# Take a step with random actions
# EnvPool expects a batch of actions
actions = np.random.randint(0, env.action_space.n, size=100)
obs, reward, done, info = env.step(actions)

print(f"Observations shape: {obs.shape}")
print(f"Rewards: {reward[:5]}")

Integration with Stable-Baselines3

Since EnvPool provides a vectorized environment, it can be wrapped to be compatible with Stable-Baselines3 (SB3). Researchers often use a VecAdapter to bridge the gap between EnvPool’s output and SB3’s expected input format.

from stable_baselines3 import PPO
import envpool

# Create EnvPool environment
venv = envpool.make("Ant-v4", num_envs=32)

# Wrap the EnvPool object to make it SB3 compatible
# Note: The VecAdapter is typically a custom wrapper provided in the repo's examples
model = PPO("MlpPolicy", venv, verbose=1)
model.learn(total_timesteps=100_000)

Real-World Use Cases

EnvPool shines in scenarios where the environment simulation is the primary bottleneck of the training process.

• Rapid Prototyping of RL Algorithms: Researchers can test new PPO or DQN variants on Atari games in minutes rather than hours, significantly accelerating the research cycle.
• Large-Scale Atari/MuJoCo Benchmarking: When comparing multiple seeds or hyperparameters, EnvPool allows for running hundreds of parallel environments on a single workstation, maximizing CPU utilization.
• High-Throughput Data Collection: For offline RL, EnvPool can be used to generate massive amounts of transition data quickly by running a variety of agents in parallel across thousands of environments.
• CPU-Bound Simulation: For projects that require traditional CPU-based physics engines (like MuJoCo) and and cannot be transitioned to GPU-based simulators, EnvPool is the most efficient way to scale.

Contributing to EnvPool

EnvPool is an open-source project that welcomes contributions. Because the core engine is written in C++, contributions typically involve adding new environments or optimizing the thread pool logic.

To contribute, you should first fork the project and follow the CONTRIBUTING.md guidelines. If you are adding a new environment, you should implement the environment logic in C++ and register it via a registration.py file to make it available to the Python API. The project also welcomes bug reports and documentation improvements via GitHub Issues.

Community and Support

The primary hub for EnvPool is its GitHub repository, where developers can report issues, request features, and discuss implementation details. Documentation is available at envpool.readthedocs.io, which includes detailed guides on the Python interface, C++ API reference, and XLA interface.

The project is maintained by SAIL at Singapore, and its activity level is high, with regular updates to support newer Python versions and provide pre-compiled wheels for multiple platforms.

Conclusion

EnvPool is a critical tool for any RL researcher who finds their training process stalled by CPU-bound environment simulation. By moving the environment execution to a high-performance C++ backend, it provides a massive increase in throughput without sacrificing the ease of use provided by the Python frontend. It is the ideal choice for those working with Atari and MuJoCo environments who want to maximize their hardware utilization.

While the barrier to adding custom environments is higher than in pure Python, the performance gains are undeniable. If you are using standard benchmarks, EnvPool is a no-brainer for your training pipeline. Star the repo, try the quickstart, and join the community to accelerate your RL experiments.

Resources

• EnvPool GitHub Repository
• Official EnvPool Documentation
• EnvPool Research Paper (arXiv)