Real-World Reinforcement Learning (RWRL) Challenge Framework
The "Challenges of Real-World RL" paper
identifies and describes a set of nine challenges that are currently preventing
Reinforcement Learning (RL) agents from being utilized on real-world
applications and products. It also describes an evaluation framework and a set
of environments that can provide an evaluation of an RL algorithmโs potential
applicability to real-world systems. It has since then been followed up with
"An Empirical Investigation of the challenges of real-world reinforcement
learning" which implements eight of the
nine described challenges (excluding explainability) and analyses their effects
on various state-of-the-art RL algorithms. This is the codebase used to perform
this analysis, and is also intended as a common platform for easily reproducible
experimentation around these challenges, it is referred to as the
realworldrl-suite (Real-World Reinforcement Learning (RWRL) Suite).
Currently the suite is to comprised of five environments:
- Cartpole
- Walker
- Quadriped
- Manipulator (less tested)
- Humanoid
The codebase is currently structured as:
- environments/ -- the extended environments
- utils/ -- wrapper classes for logging and standardized evaluations
- analysis/ -- Notebook for training an agent and generating plots
- examples/ -- Random policy and PPO agent example implementations
- docs/ -- Documentation
Questions can be directed to the Real-World RL group e-mail [[email protected]].
โน๏ธ If you wish to test your agent in a principled fashion on related challenges in low-dimensional domains, we highly recommend using bsuite.
Documentation
We overview the challenges here, but more thorough documentation on how to configure each challenge can be found here.
Starter examples are presented in the examples section.
Challenges
Safety
Adds a set of constraints on the task. Returns an additional entry in the observations ('constraints') in the length of the number of the contraints, where each entry is True if the constraint is satisfied and False otherwise.
Delays
Action, observation and reward delays.
- Action delay is the number of steps between passing the action to the environment to when it is actually performed.
- Observation delay is the offset of freshness of the returned observation after performing a step.
- Reward delay indicates the number of steps before receiving a reward after taking an action.
Noise
Action and observation noise. Different noise include:
- White Gaussian action/observation noise
- Dropped actions/observations
- Stuck actions/observations
- Repetitive actions
The noise specifications can be parameterized in the noise_spec dictionary.
Perturbations
Perturbs physical quantities of the environment. These perturbations are non-stationary and are governed by a scheduler.
Dimensionality
Adds extra dummy features to observations to increase dimensionality of the state space.
Multi-Objective Rewards:
Adds additional objectives and specifies objectives interaction (e.g., sum).
Offline Learning
We provide our offline datasets through the RL Unplugged library. There is an example and an associated colab.
RWRL Combined Challenge Benchmarks:
Combines multiple challenges into the same environment. The challenges are divided into 'Easy', 'Medium' and 'Hard' which depend on the magnitude of the challenge effects applied along each challenge dimension.
Installation
-
Install pip:
-
Run the following commands:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python get-pip.py
-
Make sure pip is up to date.
pip3 install --upgrade pip
-
(Optional) You may wish to create a Python virtual environment to manage your dependencies, so as not to clobber your system installation:
sudo pip3 install virtualenv /usr/local/bin/virtualenv realworldrl_suite source ./realworldrl/bin/activate -
Install MuJoCo (see dm_control - https://github.com/deepmind/dm_control).
-
To install
realworldrl_suite:- Clone the repository by running:
git clone https://github.com/google-research/realworldrl_suite.git
- Ensure you are in the parent directory of realworldrl_suite
- Run the command:
pip3 install realworldrl_suite/
Running examples
We provide three example agents: a random agent, a PPO agent, and an ACME-based DMPO agent.
-
For PPO, running the examples requires installing the following packages:
pip3 install tensorflow==1.15.0 dm2gym pip3 install git+git://github.com/openai/baselines.git
-
The PPO example can then be run with
cd realworldrl_suite/examples mkdir /tmp/rwrl/ python3 run_ppo.py -
For DMPO, one can run the example by installing the following packages:
pip install dm-acme pip install dm-acme[reverb] pip install dm-acme[tf]
You may also have to install the following:
pip install gym pip install jax pip install dm-sonnet
-
The examples look for the MuJoCo licence key in
~/.mujoco/mjkey.txtby default.
RWRL Combined Challenge Benchmark Instantiation:
As mentioned above, these benchmark challenges are divided into 'Easy', 'Medium' and 'Hard' difficulty levels. For the current state-of-the-art performance on these benchmarks, please see this paper.
Instantiating a combined challenge environment with 'Easy' difficulty is done as follows:
import realworldrl_suite.environments as rwrl
env = rwrl.load(
domain_name='cartpole',
task_name='realworld_swingup',
combined_challenge='easy',
log_output='/tmp/path/to/results.npz',
environment_kwargs=dict(log_safety_vars=True, flat_observation=True))Acknowledgements
If you use realworldrl_suite in your work, please cite:
@article{dulacarnold2020realworldrlempirical,
title={An empirical investigation of the challenges of real-world reinforcement learning},
author={Dulac-Arnold, Gabriel and
Levine, Nir and
Mankowitz, Daniel J. and
Li, Jerry and
Paduraru, Cosmin and
Gowal, Sven and
Hester, Todd
},
year={2020},
}