Dyna, an integrated architecture for learning, planning, and reacting

• Richard S. Sutton. ACM 1991
• Key: dyna architecture
• ExpEnv: None

PILCO: A Model-Based and Data-Efficient Approach to Policy Search

• Marc Peter Deisenroth, Carl Edward Rasmussen. ICML 2011
• Key: probabilistic dynamics model
• ExpEnv: cart-pole system, robotic unicycle

Learning Complex Neural Network Policies with Trajectory Optimization

• Sergey Levine, Vladlen Koltun. ICML 2014
• Key: guided policy search
• ExpEnv: mujoco

Learning Continuous Control Policies by Stochastic Value Gradients

• Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, Tom Erez. NIPS 2015
• Key: backpropagation through paths, gradient on real trajectory
• ExpEnv: mujoco

Value Prediction Network

• Junhyuk Oh, Satinder Singh, Honglak Lee. NIPS 2017
• Key: value-prediction model
• ExpEnv: collect domain, atari

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

• Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee. NIPS 2018
• Key: ensemble model and Qnet, value expansion
• ExpEnv: mujoco, roboschool

Recurrent World Models Facilitate Policy Evolution

• David Ha, Jürgen Schmidhuber. NIPS 2018
• Key: vae(representation), rnn(predictive model)
• ExpEnv: car racing, vizdoom

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

• Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine. NIPS 2018
• Key: probabilistic ensembles with trajectory sampling
• ExpEnv: cartpole, mujoco

When to Trust Your Model: Model-Based Policy Optimization

• Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine. NeurIPS 2019
• Key: ensemble model, sac, k-branched rollout
• ExpEnv: mujoco

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

• Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, Tengyu Ma. ICLR 2019
• Key: Discrepancy Bounds Design, ME-TRPO with multi-step, Entropy regularization
• ExpEnv: mujoco

Model-Ensemble Trust-Region Policy Optimization

• Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel. ICLR 2018
• Key: ensemble model, TRPO

• ExpEnv: mujoco

Dream to Control: Learning Behaviors by Latent Imagination

• Danijar Hafner, Timothy Lillicrap, Jimmy Ba, Mohammad Norouzi. ICLR 2019
• Key: DreamerV1, latent space imagination
• ExpEnv: deepmind control suite, atari, deepmind lab

Exploring Model-based Planning with Policy Networks

• Tingwu Wang, Jimmy Ba. ICLR 2020
• Key: model-based policy planning in action space and parameter space
• ExpEnv: mujoco

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

• Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver. Nature 2020
• Key: MCTS, value equivalence
• ExpEnv: chess, shogi, go, atari

Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels

• Sai Rajeswar Mudumba, Pietro Mazzaglia, Tim Verbelen, Alexandre Piche, Bart Dhoedt, Aaron Courville, Alexandre Lacoste
• Key: unsupervised pretrain, task-aware finetune, dyna-mpc
• ExpEnv: URLB benchmark, RWRL suite

Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy

• Xiyao Wang, Wichayaporn Wongkamjan, Ruonan Jia, Furong Huang
• Key: policy-adapted model learning, weight design
• ExpEnv: mujoco

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

• Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Baher Abdulhai, Scott Sanner
• Key: model-based offline, bayesian posterior value estimate
• OpenReview: 8, 8, 6, 6
• ExpEnv: d4rl

User-Interactive Offline Reinforcement Learning

• Phillip Swazinna, Steffen Udluft, Thomas Runkler
• Key: let the user adapt the policy behavior after training is finished
• OpenReview: 10, 8, 6, 3
• ExpEnv: 2d-world, industrial benchmark

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning

• Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, Junshan Zhang
• Key: offline IRL, reward extrapolation error
• OpenReview: 8, 8, 6, 6
• ExpEnv: d4rl

Efficient Offline Policy Optimization with a Learned Model

• Zichen Liu, Siyi Li, Wee Sun Lee, Shuicheng YAN, Zhongwen Xu
• Key: offline rl, analysis of MuZero Unplugged, one-step look-ahead policy improvement
• OpenReview: 8, 6, 5
• ExpEnv: atari dataset

Efficient Planning in a Compact Latent Action Space

• zhengyao jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, Yuandong Tian
• Key: planning with VQ-VAE
• OpenReview: 6, 6, 6, 6
• ExpEnv: d4rl dataset

Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function

• Ruijie Zheng, Xiyao Wang, Huazhe Xu, Furong Huang
• Key: lipschitz regularization
• OpenReview: 8, 8, 6, 6
• ExpEnv: mujoco

MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations

• Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, Aravind Rajeswaran
• Key: three phases -- policy pretraining, targeted exploration, interactive learning
• OpenReview: 8, 6, 6, 6
• ExpEnv: adroit, meta-world, deepmind control suite

Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective

• Raj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach, Sergey Levine, Ruslan Salakhutdinov
• Key: Aligned Latent Models
• OpenReview: 8, 6, 6, 6, 6
• ExpEnv: mujoco

Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning

• Daniel Palenicek, Michael Lutter, Joao Carvalho, Jan Peters
• Key: longer horizons yield diminishing returns in terms of sample efficiency
• OpenReview: 8, 6, 6, 6
• ExpEnv: brax

Planning Goals for Exploration

• Edward S. Hu, Richard Chang, Oleh Rybkin, Dinesh Jayaraman
• Key: sampling-based planning, set goals for each training episode to directly optimize an intrinsic exploration reward
• OpenReview: 8, 8, 8, 8, 6
• ExpEnv: point maze, walker, ant maze, 3-block stack

Making Better Decision by Directly Planning in Continuous Control

• Jinhua Zhu, Yue Wang, Lijun Wu, Tao Qin, Wengang Zhou, Tie-Yan Liu, Houqiang Li
• Key: deep differentiable dynamic programming planner
• OpenReview: 8, 8, 8, 6
• ExpEnv: mujoco

Latent Variable Representation for Reinforcement Learning

• Tongzheng Ren, Chenjun Xiao, Tianjun Zhang, Na Li, Zhaoran Wang, sujay sanghavi, Dale Schuurmans, Bo Dai
• Key: variational learning, representation learning
• OpenReview: 8, 6, 6, 3
• ExpEnv: mujoco, deepmind control suite

SpeedyZero: Mastering Atari with Limited Data and Time

• Yixuan Mei, Jiaxuan Gao, Weirui Ye, Shaohuai Liu, Yang Gao, Yi Wu
• Key: distributed model-based rl, speed up EfficientZero
• OpenReview: 6, 6, 5
• ExpEnv: atari 100k

Transformer-based World Models Are Happy With 100k Interactions

• Jan Robine, Marc Höftmann, Tobias Uelwer, Stefan Harmeling
• Key: autoregressive world model, Transformer-XL, balanced cross-entropy loss, balanced dataset sampling
• OpenReview: 8, 6, 6, 6
• ExpEnv: atari 100k

On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning

• Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chieh Chan, Hao Su, Zhuowen Tu
• Key: offline multi-task pretraining, online finetuning
• OpenReview: 6, 6, 6, 6
• ExpEnv: atari 100k

Become a Proficient Player with Limited Data through Watching Pure Videos

• Weirui Ye, Yunsheng Zhang, Pieter Abbeel, Yang Gao
• Key: unsupervised pre-training, finetune with down-stream tasks
• OpenReview: 8, 6, 6, 5
• ExpEnv: atari 100k

EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model

• Yifu Yuan, Jianye HAO, Fei Ni, Yao Mu, YAN ZHENG, Yujing Hu, Jinyi Liu, Yingfeng Chen, Changjie Fan
• Key: jointly pretrain the multi-headed dynamics model and unsupervised exploration policy, finetune to downstream tasks
• OpenReview: 6, 6, 6, 6
• ExpEnv: URLB benchmark

Choreographer: Learning and Adapting Skills in Imagination

• Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, Sai Rajeswar
• Key: world model, skill discovery, skill learning, Skill adaptation
• OpenReview: 8, 8, 6, 6
• ExpEnv: deepmind control suite, Meta-World

Bidirectional Learning for Offline Infinite-width Model-based Optimization

• Can Chen, Yingxue Zhang, Jie Fu, Xue Liu, Mark Coates
• Key: model-based, offline
• OpenReview: 7, 6, 5
• ExpEnv: design-bench

A Unified Framework for Alternating Offline Model Training and Policy Learning

• Shentao Yang, Shujian Zhang, Yihao Feng, Mingyuan Zhou
• Key: model-based, offline, marginal importance weight
• OpenReview: 7, 6, 6, 5
• ExpEnv: d4rl dataset

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

• Kaiyang Guo, Shao Yunfeng, Yanhui Geng
• Key: model-based, offline
• OpenReview: 8, 8, 7, 7
• ExpEnv: d4rl dataset

Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination

• Jiafei Lyu, Xiu Li, Zongqing Lu
• Key: double check mechanism, bidirectional modeling, offline RL
• OpenReview: 7, 6, 6
• ExpEnv: d4rl dataset

Model-Based Opponent Modeling

• XiaoPeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, Zongqing Lu
• Key: multi-agent, model-based
• OpenReview: 7, 6, 4, 3
• ExpEnv: mpe, google research football

Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning

• Zhiwei Xu, Dapeng Li, Bin Zhang, Yuan Zhan, Yunpeng Bai, Guoliang Fan
• Key: multi-agent, model-based
• OpenReview: 6, 5
• ExpEnv: StarCraft II, Google Research Football, Multi-Agent Discrete MuJoCo

MoCoDA: Model-based Counterfactual Data Augmentation

• Silviu Pitis, Elliot Creager, Ajay Mandlekar, Animesh Garg
• Key: data augmentation framework, offline RL
• OpenReview: 7, 7, 7, 6
• ExpEnv: 2D Navigation, Hook-Sweep

When to Update Your Model: Constrained Model-based Reinforcement Learning

• Tianying Ji, Yu Luo, Fuchun Sun, Mingxuan Jing, Fengxiang He, Wenbing Huang
• Key: event-triggered mechanism, constrained model-shift lower-bound optimization
• OpenReview: 6, 6, 5, 5
• ExpEnv: mujoco

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

• Ashish Jayant, Shalabh Bhatnagar
• Key: constrained RL, model-based
• OpenReview: 7, 6, 5, 5
• ExpEnv: safety gym

Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework

• Henger Li, Xiaolin Sun, Zizhan Zheng
• Key: attack & defense, federated learning, model-based
• OpenReview: 6, 6, 6, 5
• ExpEnv: MNIST, FashionMNIST, EMNIST, CIFAR-10 and synthetic dataset

Model-Based Imitation Learning for Urban Driving

• Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, Jamie Shotton
• Key: model-based, imitation learning, autonomous driving
• OpenReview: 7, 6, 6
• ExpEnv: CARLA

Data-Driven Model-Based Optimization via Invariant Representation Learning

• Han Qi, Yi Su, Aviral Kumar, Sergey Levine
• Key: domain adaptation, invariant objective models, representation learning (no about model-based RL)
• OpenReview: 7, 6, 6, 5, 5
• ExpEnv: design-bench

Model-based Lifelong Reinforcement Learning with Bayesian Exploration

• Haotian Fu, Shangqun Yu, Michael Littman, George Konidaris
• Key: lifelong RL, variational bayesian
• OpenReview: 7, 6, 6
• ExpEnv: mujoco, meta-world

Plan To Predict: Learning an Uncertainty-Foreseeing Model For Model-Based Reinforcement Learning

• Zifan Wu, Chao Yu, Chen Chen, Jianye Hao, Hankz Hankui Zhuo
• Key: treat the model rollout process as a sequential decision making problem
• OpenReview: 7, 7, 6, 6
• ExpEnv: mujoco, d4rl

Joint Model-Policy Optimization of a Lower Bound for Model-Based RL

• Benjamin Eysenbach, Alexander Khazatsky, Sergey Levine, Russ Salakhutdinov
• Key: unified objective for model-based RL
• OpenReview: 8, 8, 7, 6
• ExpEnv: gridworld, mujoco, ROBEL manipulation

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning

• Marc Rigter, Bruno Lacerda, Nick Hawes
• Key: offline rl, model-based rl, two-player game, adversarial model training
• OpenReview: 6, 6, 6, 4
• ExpEnv: d4rl

Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning

• Shenao Zhang
• Key: posterior sampling RL, referential update, constrained conservative update
• OpenReview: 7, 7, 5, 5
• ExpEnv: mujoco, N-Chain MDPs

Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning

• Chenyang Wu, Tianci Li, Zongzhang Zhang, Yang Yu
• Key: optimism in the face of uncertainty(OFU), BOO Regret
• OpenReview: 6, 6, 5
• ExpEnv: RiverSwim, Chain, Random MDPs

Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexity

• Alekh Agarwal, Tong Zhang
• Key: posterior sampling RL, Bellman error decoupling framework
• OpenReview: 7, 7, 7, 6
• ExpEnv: None

Exponential Family Model-Based Reinforcement Learning via Score Matching

• Gene Li, Junbo Li, Nathan Srebro, Zhaoran Wang, Zhuoran Yang
• Key: optimistic model-based, score matching
• OpenReview: 7, 7, 6
• ExpEnv: None

Deep Hierarchical Planning from Pixels

• Danijar Hafner, Kuang-Huei Lee, Ian Fischer, Pieter Abbeel
• Key: hierarchical RL, long-horizon and sparse reward tasks
• OpenReview: 6, 6, 5
• ExpEnv: atari, deepmind control suite, deepmind lab, crafter

DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations

• Fei Deng, Ingook Jang, Sungjin Ahn
• Key: dreamer, prototypes
• ExpEnv: deepmind control suite

Denoised MDPs: Learning World Models Better Than the World Itself

• Tongzhou Wang, Simon Du, Antonio Torralba, Phillip Isola, Amy Zhang, Yuandong Tian
• Key: representation learning, denoised model
• ExpEnv: deepmind control suite, RoboDesk

Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models and Amortized Policy Search

• Qi Wang, Herke van Hoof
• Key: graph structured surrogate model, meta training
• ExpEnv: atari, mujoco

Towards Adaptive Model-Based Reinforcement Learning

• Yi Wan, Ali Rahimi-Kalahroudi, Janarthanan Rajendran, Ida Momennejad, Sarath Chandar, Harm van Seijen
• Key: local change adaptation
• ExpEnv: GridWorldLoCA, ReacherLoCA, MountaincarLoCA

Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation

• Pier Giuseppe Sessa, Maryam Kamgarpour, Andreas Krause
• Key: model-based multi-agent, confidence bound
• ExpEnv: SMART

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning

• Shentao Yang, Yihao Feng, Shujian Zhang, Mingyuan Zhou
• Key: offline rl, model-based rl, stationary distribution regularization
• ExpEnv: d4rl

Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization

• Brandon Trabucco, Xinyang Geng, Aviral Kumar, Sergey Levine
• Key: benchmark, offline MBO
• ExpEnv: Design-Bench Benchmark Tasks

Temporal Difference Learning for Model Predictive Control

• Nicklas Hansen, Hao Su, Xiaolong Wang
• Key: td-learning, MPC
• ExpEnv: deepmind control suite, Meta-World

Revisiting Design Choices in Offline Model Based Reinforcement Learning

• Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, Stephen J. Roberts
• Key: model-based offline, uncertainty quantification
• OpenReview: 8, 8, 6, 6, 6
• ExpEnv: d4rl dataset

Value Gradient weighted Model-Based Reinforcement Learning

• Claas A Voelcker, Victor Liao, Animesh Garg, Amir-massoud Farahmand
• Key: Value-Gradient weighted Model loss
• OpenReview: 8, 8, 6, 6
• ExpEnv: mujoco

Planning in Stochastic Environments with a Learned Model

• Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K Hubert, David Silver
• Key: MCTS, stochastic MuZero
• OpenReview: 10, 8, 8, 5
• ExpEnv: 2048 game, Backgammon, Go

Policy improvement by planning with Gumbel

• Ivo Danihelka, Arthur Guez, Julian Schrittwieser, David Silver
• Key: Gumbel AlphaZero, Gumbel MuZero
• OpenReview: 8, 8, 8, 6
• ExpEnv: go, chess, atari

Model-Based Offline Meta-Reinforcement Learning with Regularization

• Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang, Junshan Zhang
• Key: model-based offline Meta-RL
• OpenReview: 8, 6, 6, 6
• ExpEnv: d4rl dataset

On-Policy Model Errors in Reinforcement Learning

• Lukas Froehlich, Maksym Lefarov, Melanie Zeilinger, Felix Berkenkamp
• Key: model errors, on-policy corrections
• OpenReview: 8, 6, 6, 5
• ExpEnv: mujoco, pybullet

A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning

• Jiaxian Guo, Mingming Gong, Dacheng Tao
• Key: relational intervention, dynamics generalization
• OpenReview: 8, 8, 6, 6
• ExpEnv: Pendulum, mujoco

Information Prioritization through Empowerment in Visual Model-based RL

• Homanga Bharadhwaj, Mohammad Babaeizadeh, Dumitru Erhan, Sergey Levine
• Key: mutual information, visual model-based RL
• OpenReview: 8, 8, 8, 6
• ExpEnv: deepmind control suite, Kinetics dataset

Transfer RL across Observation Feature Spaces via Model-Based Regularization

• Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew E Cohen, Furong Huang
• Key: latent dynamics model, transfer RL
• OpenReview: 8, 6, 5, 5
• ExpEnv: CartPole, Acrobot and Cheetah-Run, mujoco, 3DBall

Learning State Representations via Retracing in Reinforcement Learning

• Changmin Yu, Dong Li, Jianye HAO, Jun Wang, Neil Burgess
• Key: representation learning, learning via retracing
• OpenReview: 8, 6, 5, 3
• ExpEnv: deepmind control suite

Model-augmented Prioritized Experience Replay

• Youngmin Oh, Jinwoo Shin, Eunho Yang, Sung Ju Hwang
• Key: prioritized experience replay, mbrl
• OpenReview: 8, 8, 6, 5
• ExpEnv: pybullet

Evaluating Model-Based Planning and Planner Amortization for Continuous Control

• Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim, Mehdi Mirza, Alessandro Davide Ialongo, Yuval Tassa, Jost Tobias Springenberg, Abbas Abdolmaleki, Nicolas Heess, Josh Merel, Martin Riedmiller
• Key: model predictive control
• OpenReview: 8, 6, 6, 6
• ExpEnv: mujoco

Gradient Information Matters in Policy Optimization by Back-propagating through Model

• Chongchong Li, Yue Wang, Wei Chen, Yuting Liu, Zhi-Ming Ma, Tie-Yan Liu
• Key: two-model-based method, analyze model error and policy gradient
• OpenReview: 8, 8, 6, 6
• ExpEnv: mujoco

Pareto Policy Pool for Model-based Offline Reinforcement Learning

• Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, Yuhui Shi
• Key: model-based offline, model return-uncertainty trade-off
• OpenReview: 8, 8, 6, 5
• ExpEnv: d4rl dataset

Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

• Masatoshi Uehara, Wen Sun
• Key: model-based offline theory, PAC bounds
• OpenReview: 8, 6, 6, 5
• ExpEnv: None

Know Thyself: Transferable Visual Control Policies Through Robot-Awareness

• Edward S. Hu, Kun Huang, Oleh Rybkin, Dinesh Jayaraman
• Key: world models that transfer to new robots
• OpenReview: 8, 6, 6, 5
• ExpEnv: mujoco, WidowX and Franka Panda robot

On Effective Scheduling of Model-based Reinforcement Learning

• Hang Lai, Jian Shen, Weinan Zhang, Yimin Huang, Xing Zhang, Ruiming Tang, Yong Yu, Zhenguo Li
• Key: extension of mbpo, hyper-controller learning
• OpenReview: 8, 6, 6
• ExpEnv: mujoco, pybullet

Safe Reinforcement Learning by Imagining the Near Future

• Garrett Thomas, Yuping Luo, Tengyu Ma
• Key: safe rl, reward penalty, theory about model-based rollouts
• OpenReview: 8, 6, 6
• ExpEnv: mujoco

Model-Based Reinforcement Learning via Imagination with Derived Memory

• Yao Mu, Yuzheng Zhuang, Bin Wang, Guangxiang Zhu, Wulong Liu, Jianyu Chen, Ping Luo, Shengbo Eben Li, Chongjie Zhang, Jianye HAO
• Key: extension of dreamer, prediction-reliability weight
• OpenReview: 6, 6, 6, 6
• ExpEnv: deepmind control suite

MobILE: Model-Based Imitation Learning From Observation Alone

• Rahul Kidambi, Jonathan Chang, Wen Sun
• Key: imitation learning from observations alone, mbrl
• OpenReview: 6, 6, 6, 4
• ExpEnv: cartpole, mujoco

Model-Based Episodic Memory Induces Dynamic Hybrid Controls

• Hung Le, Thommen Karimpanal George, Majid Abdolshah, Truyen Tran, Svetha Venkatesh
• Key: model-based, episodic control
• OpenReview: 7, 7, 6, 6
• ExpEnv: 2D maze navigation, cartpole, mountainCar and lunarlander, atari, 3D navigation: gym-miniworld

A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning

• Mingde Zhao, Zhen Liu, Sitao Luan, Shuyuan Zhang, Doina Precup, Yoshua Bengio
• Key: mbrl, set representation
• OpenReview: 7, 7, 7, 6
• ExpEnv: MiniGrid-BabyAI framework

Mastering Atari Games with Limited Data

• Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, Yang Gao
• Key: muzero, self-supervised consistency loss
• OpenReview: 7, 7, 7, 5
• ExpEnv: atrai 100k, deepmind control suite

Online and Offline Reinforcement Learning by Planning with a Learned Model

• Julian Schrittwieser, Thomas K Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, David Silver
• Key: muzero, reanalyse, offline
• OpenReview: 8, 8, 7, 6
• ExpEnv: atrai dataset, deepmind control suite dataset

Self-Consistent Models and Values

• Gregory Farquhar, Kate Baumli, Zita Marinho, Angelos Filos, Matteo Hessel, Hado van Hasselt, David Silver
• Key: new model learning way
• OpenReview: 7, 7, 7, 6
• ExpEnv: tabular MDP, Sokoban, atari

Proper Value Equivalence

• Christopher Grimm, Andre Barreto, Gregory Farquhar, David Silver, Satinder Singh
• Key: value equivalence, value-based planning, muzero
• OpenReview: 8, 7, 7, 6
• ExpEnv: four rooms, atari

MOPO: Model-based Offline Policy Optimization

• Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma
• Key: model-based, offline
• OpenReview: None
• ExpEnv: d4rl dataset, halfcheetah-jump and ant-angle

RoMA: Robust Model Adaptation for Offline Model-based Optimization

• Sihyun Yu, Sungsoo Ahn, Le Song, Jinwoo Shin
• Key: model-based, offline
• OpenReview: 7, 6, 6
• ExpEnv: design-bench

Offline Reinforcement Learning with Reverse Model-based Imagination

• Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, Chongjie Zhang
• Key: model-based, offline
• OpenReview: 7, 6, 6, 5
• ExpEnv: d4rl dataset

Offline Model-based Adaptable Policy Learning

• Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Tony Qin, Shang Wenjie, Jieping Ye
• Key: model-based, offline
• OpenReview: 6, 6, 6, 4
• ExpEnv: d4rl dataset

Weighted model estimation for offline model-based reinforcement learning

• Toru Hishinuma, Kei Senda
• Key: model-based, offline, off-policy evaluation
• OpenReview: 7, 6, 6, 6
• ExpEnv: pendulum, d4rl dataset

Reward-Free Model-Based Reinforcement Learning with Linear Function Approximation

• Weitong Zhang, Dongruo Zhou, Quanquan Gu
• Key: learning theory, model-based reward-free RL, linear function approximation
• OpenReview: 6, 6, 5, 5
• ExpEnv: None

Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature

• Kefan Dong, Jiaqi Yang, Tengyu Ma
• Key: learning theory, model-based bandit RL, nonlinear function approximation
• OpenReview: 7, 7, 7, 6
• ExpEnv: None

Discovering and Achieving Goals via World Models

• Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, Deepak Pathak
• Key: unsupervised goal reaching, goal-conditioned RL
• OpenReview: 6, 6, 6, 6, 6
• ExpEnv: walker, quadruped, bins, kitchen

Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

• Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, Shixiang Gu
• Key: model-based, behavior cloning (warmup), trpo
• OpenReview: 8, 7, 7, 5
• ExpEnv: d4rl dataset

Control-Aware Representations for Model-based Reinforcement Learning

• Brandon Cui, Yinlam Chow, Mohammad Ghavamzadeh
• Key: representation learning, model-based soft actor-critic
• OpenReview: 6, 6, 6
• ExpEnv: planar system, inverted pendulum – swingup, cartpole, 3-link manipulator — swingUp & balance

Mastering Atari with Discrete World Models

• Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba
• Key: DreamerV2, many tricks(multiple categorical variables, KL balancing, etc)
• OpenReview: 9, 8, 5, 4
• ExpEnv: atari

Model-Based Visual Planning with Self-Supervised Functional Distances

• Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, Sergey Levine
• Key: goal-reaching task, dynamics learning, distance learning (goal-conditioned Q-function)
• OpenReview: 7, 7, 7, 7
• ExpEnv: sawyer, door sliding

Model-Based Offline Planning

• Arthur Argenson, Gabriel Dulac-Arnold
• Key: model-based, offline
• OpenReview: 8, 7, 5, 5
• ExpEnv: RL Unplugged(RLU), d4rl dataset

Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation

• Justin Fu, Sergey Levine
• Key: model-based, offline
• OpenReview: 8, 6, 6
• ExpEnv: design-bench

On the role of planning in model-based deep reinforcement learning

• Jessica B. Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing, Petar Veličković, Théophane Weber
• Key: discussion about planning in MuZero
• OpenReview: 7, 7, 6, 5
• ExpEnv: atari, go, deepmind control suite

Representation Balancing Offline Model-based Reinforcement Learning

• Byung-Jun Lee, Jongmin Lee, Kee-Eung Kim
• Key: Representation Balancing MDP, model-based, offline
• OpenReview: 7, 7, 7, 6
• ExpEnv: d4rl dataset

Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

• Balázs Kégl, Gabriel Hurtado, Albert Thomas
• Key: mixture density nets, heteroscedasticity
• OpenReview: 7, 7, 7, 6, 5
• ExpEnv: acrobot system

Conservative Objective Models for Effective Offline Model-Based Optimization

• Brandon Trabucco, Aviral Kumar, Xinyang Geng, Sergey Levine
• Key: conservative objective model, offline mbrl
• ExpEnv: design-bench

Continuous-Time Model-Based Reinforcement Learning

• Çağatay Yıldız, Markus Heinonen, Harri Lähdesmäki
• Key: continuous-time
• ExpEnv: pendulum, cartPole and acrobot

Model-Based Reinforcement Learning via Latent-Space Collocation

• Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, Sergey Levine
• Key: latent space collocation
• ExpEnv: sparse metaworld tasks

Model-Free and Model-Based Policy Evaluation when Causality is Uncertain

• David A Bruns-Smith
• Key: worst-case bounds
• ExpEnv: ope-tools

Muesli: Combining Improvements in Policy Optimization

• Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, Hado van Hasselt
• Key: value equivalence
• ExpEnv: atari

Vector Quantized Models for Planning

• Sherjil Ozair, Yazhe Li, Ali Razavi, Ioannis Antonoglou, Aäron van den Oord, Oriol Vinyals
• Key: VQVAE, MCTS
• ExpEnv: chess datasets, DeepMind Lab

PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration

• Yuda Song, Wen Sun
• Key: sample complexity, kernelized nonlinear regulators, linear MDPs
• ExpEnv: mountain car, antmaze, mujoco

Temporal Predictive Coding For Model-Based Planning In Latent Space

• Tung Nguyen, Rui Shu, Tuan Pham, Hung Bui, Stefano Ermon
• Key: temporal predictive coding with a RSSM, latent space
• ExpEnv: deepmind control suite

Model-based Reinforcement Learning for Continuous Control with Posterior Sampling

• Ying Fan, Yifei Ming
• Key: regret bound of psrl, mpc
• ExpEnv: continuous cartpole, pendulum swingup, mujoco

A Sharp Analysis of Model-based Reinforcement Learning with Self-Play

• Qinghua Liu, Tiancheng Yu, Yu Bai, Chi Jin
• Key: learning theory, multi-agent, model-based self play, two-player zero-sum Markov games
• ExpEnv: None