• Stars
    star
    1,185
  • Rank 38,034 (Top 0.8 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created about 4 years ago
  • Updated 6 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

RLax

CI status docs pypi

RLax (pronounced "relax") is a library built on top of JAX that exposes useful building blocks for implementing reinforcement learning agents. Full documentation can be found at rlax.readthedocs.io.

Installation

You can install the latest released version of RLax from PyPI via:

pip install rlax

or you can install the latest development version from GitHub:

pip install git+https://github.com/deepmind/rlax.git

All RLax code may then be just in time compiled for different hardware (e.g. CPU, GPU, TPU) using jax.jit.

In order to run the examples/ you will also need to clone the repo and install the additional requirements: optax, haiku, and bsuite.

Content

The operations and functions provided are not complete algorithms, but implementations of reinforcement learning specific mathematical operations that are needed when building fully-functional agents capable of learning:

  • Values, including both state and action-values;
  • Values for Non-linear generalizations of the Bellman equations.
  • Return Distributions, aka distributional value functions;
  • General Value Functions, for cumulants other than the main reward;
  • Policies, via policy-gradients in both continuous and discrete action spaces.

The library supports both on-policy and off-policy learning (i.e. learning from data sampled from a policy different from the agent's policy).

See file-level and function-level doc-strings for the documentation of these functions and for references to the papers that introduced and/or used them.

Usage

See examples/ for examples of using some of the functions in RLax to implement a few simple reinforcement learning agents, and demonstrate learning on BSuite's version of the Catch environment (a common unit-test for agent development in the reinforcement learning literature):

Other examples of JAX reinforcement learning agents using rlax can be found in bsuite.

Background

Reinforcement learning studies the problem of a learning system (the agent), which must learn to interact with the universe it is embedded in (the environment).

Agent and environment interact on discrete steps. On each step the agent selects an action, and is provided in return a (partial) snapshot of the state of the environment (the observation), and a scalar feedback signal (the reward).

The behaviour of the agent is characterized by a probability distribution over actions, conditioned on past observations of the environment (the policy). The agents seeks a policy that, from any given step, maximises the discounted cumulative reward that will be collected from that point onwards (the return).

Often the agent policy or the environment dynamics itself are stochastic. In this case the return is a random variable, and the optimal agent's policy is typically more precisely specified as a policy that maximises the expectation of the return (the value), under the agent's and environment's stochasticity.

Reinforcement Learning Algorithms

There are three prototypical families of reinforcement learning algorithms:

  1. those that estimate the value of states and actions, and infer a policy by inspection (e.g. by selecting the action with highest estimated value)
  2. those that learn a model of the environment (capable of predicting the observations and rewards) and infer a policy via planning.
  3. those that parameterize a policy that can be directly executed,

In any case, policies, values or models are just functions. In deep reinforcement learning such functions are represented by a neural network. In this setting, it is common to formulate reinforcement learning updates as differentiable pseudo-loss functions (analogously to (un-)supervised learning). Under automatic differentiation, the original update rule is recovered.

Note however, that in particular, the updates are only valid if the input data is sampled in the correct manner. For example, a policy gradient loss is only valid if the input trajectory is an unbiased sample from the current policy; i.e. the data are on-policy. The library cannot check or enforce such constraints. Links to papers describing how each operation is used are however provided in the functions' doc-strings.

Naming Conventions and Developer Guidelines

We define functions and operations for agents interacting with a single stream of experience. The JAX construct vmap can be used to apply these same functions to batches (e.g. to support replay and parallel data generation).

Many functions consider policies, actions, rewards, values, in consecutive timesteps in order to compute their outputs. In this case the suffix _t and tm1 is often to clarify on which step each input was generated, e.g:

  • q_tm1: the action value in the source state of a transition.
  • a_tm1: the action that was selected in the source state.
  • r_t: the resulting rewards collected in the destination state.
  • discount_t: the discount associated with a transition.
  • q_t: the action values in the destination state.

Extensive testing is provided for each function. All tests should also verify the output of rlax functions when compiled to XLA using jax.jit and when performing batch operations using jax.vmap.

Citing RLax

RLax is part of the DeepMind JAX Ecosystem, to cite RLax please use the DeepMind JAX Ecosystem citation.

More Repositories

1

deepmind-research

This repository contains implementations and illustrative code to accompany DeepMind publications
Jupyter Notebook
12,817
star
2

alphafold

Open source code for AlphaFold.
Python
11,700
star
3

sonnet

TensorFlow-based neural network library
Python
9,691
star
4

pysc2

StarCraft II Learning Environment
Python
7,904
star
5

mujoco

Multi-Joint dynamics with Contact. A general purpose physics simulator.
Jupyter Notebook
7,202
star
6

lab

A customisable 3D platform for agent-based AI research
C
7,012
star
7

graph_nets

Build Graph Nets in Tensorflow
Python
5,325
star
8

graphcast

Python
4,242
star
9

learning-to-learn

Learning to Learn in TensorFlow
Python
4,063
star
10

open_spiel

OpenSpiel is a collection of environments and algorithms for research in general reinforcement learning and search/planning in games.
C++
4,019
star
11

alphageometry

Python
3,580
star
12

dm_control

Google DeepMind's software stack for physics-based simulation and Reinforcement Learning environments, using MuJoCo.
Python
3,473
star
13

acme

A library of reinforcement learning components and agents
Python
3,372
star
14

trfl

TensorFlow Reinforcement Learning
Python
3,139
star
15

dm-haiku

JAX-based neural network library
Python
2,697
star
16

alphatensor

Python
2,616
star
17

dnc

A TensorFlow implementation of the Differentiable Neural Computer.
Python
2,478
star
18

mctx

Monte Carlo tree search in JAX
Python
2,209
star
19

gemma

Open weights LLM from Google DeepMind.
Jupyter Notebook
2,061
star
20

code_contests

C++
2,010
star
21

kinetics-i3d

Convolutional neural network model for video classification trained on the Kinetics dataset.
Python
1,639
star
22

mathematics_dataset

This dataset code generates mathematical question and answer pairs, from a range of question types at roughly school-level difficulty.
Python
1,621
star
23

optax

Optax is a gradient processing and optimization library for JAX.
Python
1,492
star
24

bsuite

bsuite is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent
Python
1,465
star
25

penzai

A JAX research toolkit for building, editing, and visualizing neural networks.
Python
1,405
star
26

educational

Jupyter Notebook
1,382
star
27

jraph

A Graph Neural Network Library in Jax
Python
1,306
star
28

rc-data

Question answering dataset featured in "Teaching Machines to Read and Comprehend
Python
1,285
star
29

tapnet

Tracking Any Point (TAP)
Python
1,033
star
30

scalable_agent

A TensorFlow implementation of Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures.
Python
972
star
31

neural-processes

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).
Jupyter Notebook
966
star
32

android_env

RL research on Android devices.
Python
946
star
33

mujoco_menagerie

A collection of high-quality models for the MuJoCo physics engine, curated by Google DeepMind.
Jupyter Notebook
926
star
34

dramatron

Dramatron uses large language models to generate coherent scripts and screenplays.
Jupyter Notebook
904
star
35

tree

tree is a library for working with nested data structures
Python
891
star
36

xmanager

A platform for managing machine learning experiments
Python
796
star
37

mujoco_mpc

Real-time behaviour synthesis with MuJoCo, using Predictive Control
C++
771
star
38

materials_discovery

Python
770
star
39

chex

Python
716
star
40

reverb

Reverb is an efficient and easy-to-use data storage and transport system designed for machine learning research
C++
692
star
41

alphadev

Python
662
star
42

pycolab

A highly-customisable gridworld game engine with some batteries included. Make your own gridworld games to test reinforcement learning agents!
Python
654
star
43

ferminet

An implementation of the Fermionic Neural Network for ab-initio electronic structure calculations
Python
643
star
44

hanabi-learning-environment

hanabi_learning_environment is a research platform for Hanabi experiments.
Python
614
star
45

funsearch

Jupyter Notebook
611
star
46

ai-safety-gridworlds

This is a suite of reinforcement learning environments illustrating various safety properties of intelligent agents.
Python
577
star
47

dqn

Lua/Torch implementation of DQN (Nature, 2015)
Lua
546
star
48

ithaca

Restoring and attributing ancient texts using deep neural networks
Jupyter Notebook
540
star
49

meltingpot

A suite of test scenarios for multi-agent reinforcement learning.
Python
516
star
50

distrax

Python
509
star
51

recurrentgemma

Open weights language model from Google DeepMind, based on Griffin.
Python
505
star
52

surface-distance

Library to compute surface distance based performance metrics for segmentation tasks.
Python
493
star
53

tracr

Python
467
star
54

dsprites-dataset

Dataset to assess the disentanglement properties of unsupervised learning methods
Jupyter Notebook
463
star
55

alphamissense

Python
455
star
56

narrativeqa

This repository contains the NarrativeQA dataset. It includes the list of documents with Wikipedia summaries, links to full stories, and questions and answers.
Shell
432
star
57

lab2d

A customisable 2D platform for agent-based AI research
C++
415
star
58

open_x_embodiment

Jupyter Notebook
409
star
59

dqn_zoo

DQN Zoo is a collection of reference implementations of reinforcement learning agents developed at DeepMind based on the Deep Q-Network (DQN) agent.
Python
406
star
60

clrs

Python
376
star
61

spriteworld

Spriteworld: a flexible, configurable python-based reinforcement learning environment
Python
366
star
62

dm_pix

PIX is an image processing library in JAX, for JAX.
Python
363
star
63

concordia

A library for generative social simulation
Python
351
star
64

mathematics_conjectures

Jupyter Notebook
348
star
65

alphastar

Python
346
star
66

spiral

We provide a pre-trained model for unconditional 19-step generation of CelebA-HQ images
C++
327
star
67

dm_env

A Python interface for reinforcement learning environments
Python
326
star
68

dm_robotics

Libraries, tools and tasks created and used at DeepMind Robotics.
Python
315
star
69

uncertain_ground_truth

Dermatology ddx dataset, Jax implementations of Monte Carlo conformal prediction, plausibility regions and statistical annotation aggregation from our recent work on uncertain ground truth (TMLR'23 and ArXiv pre-print).
Python
315
star
70

long-form-factuality

Benchmarking long-form factuality in large language models. Original code for our paper "Long-form factuality in large language models".
Python
314
star
71

launchpad

Python
305
star
72

leo

Implementation of Meta-Learning with Latent Embedding Optimization
Python
302
star
73

streetlearn

A C++/Python implementation of the StreetLearn environment based on images from Street View, as well as a TensorFlow implementation of goal-driven navigation agents solving the task published in “Learning to Navigate in Cities Without a Map”, NeurIPS 2018
C++
279
star
74

gqn-datasets

Datasets used to train Generative Query Networks (GQNs) in the ‘Neural Scene Representation and Rendering’ paper.
Python
269
star
75

enn

Python
265
star
76

multi_object_datasets

Multi-object image datasets with ground-truth segmentation masks and generative factors.
Python
247
star
77

AQuA

A algebraic word problem dataset, with multiple choice questions annotated with rationales.
238
star
78

card2code

A code generation dataset for generating the code that implements Hearthstone and Magic The Gathering card effects.
236
star
79

grid-cells

Implementation of the supervised learning experiments in Vector-based navigation using grid-like representations in artificial agents, as published at https://www.nature.com/articles/s41586-018-0102-6
Python
236
star
80

arnheim

Jupyter Notebook
235
star
81

synjax

Python
233
star
82

torch-hdf5

Torch interface to HDF5 library
Lua
231
star
83

dm_memorytasks

A set of 13 diverse machine-learning tasks that require memory to solve.
Python
220
star
84

Temporal-3D-Pose-Kinetics

Exploiting temporal context for 3D human pose estimation in the wild: 3D poses for the Kinetics dataset
Python
214
star
85

opro

official code for "Large Language Models as Optimizers"
Python
199
star
86

dm_alchemy

DeepMind Alchemy task environment: a meta-reinforcement learning benchmark
Python
197
star
87

neural_testbed

Jupyter Notebook
187
star
88

kfac-jax

Second Order Optimization and Curvature Estimation with K-FAC in JAX.
Python
187
star
89

pg19

179
star
90

xquad

173
star
91

jmp

JMP is a Mixed Precision library for JAX.
Python
171
star
92

spectral_inference_networks

Implementation of Spectral Inference Networks, ICLR 2019
Python
165
star
93

abstract-reasoning-matrices

Progressive matrices dataset, as described in: Measuring abstract reasoning in neural networks (Barrett*, Hill*, Santoro*, Morcos, Lillicrap), ICML2018
162
star
94

neural_networks_chomsky_hierarchy

Neural Networks and the Chomsky Hierarchy
Python
162
star
95

xitari

This is the 0.4 release of the Arcade Learning Environment (ALE), a platform designed for AI research. ALE is based on Stella, an Atari 2600 VCS emulator.
C++
159
star
96

tensor_annotations

Annotating tensor shapes using Python types
Python
158
star
97

symplectic-gradient-adjustment

A colab that implements the Symplectic Gradient Adjustment optimizer from "The mechanics of n-player differentiable games"
Jupyter Notebook
150
star
98

mc_gradients

Jupyter Notebook
149
star
99

interval-bound-propagation

This repository contains a simple implementation of Interval Bound Propagation (IBP) using TensorFlow: https://arxiv.org/abs/1810.12715
Python
148
star
100

s6

C++
146
star