• Below are the benchmark results of AlphaZero and MuZero on two board games: TicTacToe, Gomoku.

• Below are the benchmark results of MuZero, MuZero w/ SSL , EfficientZero and Sampled EfficientZero on three discrete action space games in Atari.

• Below are the benchmark results of Sampled EfficientZero with Factored/Gaussian policy representation on three classic continuous action space games: Pendulum-v1, LunarLanderContinuous-v2, BipedalWalker-v3 and two MuJoCo continuous action space games: Hopper-v3, Walker2d-v3.

"Factored Policy" indicates that the agent learns a policy network that outputs a categorical distribution. After manual discretization, the dimensions of the action space for the five environments are 11, 49 (7^2), 256 (4^4), 64 (4^3), and 4096 (4^6), respectively. On the other hand, "Gaussian Policy" refers to the agent learning a policy network that directly outputs parameters (mu and sigma) for a Gaussian distribution.

• Below are the benchmark results of GumbelMuZero and MuZero (under different simulation cost) on four environments: PongNoFrameskip-v4, MsPacmanNoFrameskip-v4, Gomoku, and LunarLanderContinuous-v2.

• AlphaZero
• MuZero
• EfficientZero
• SampledMuZero
• GumbelMuZero
• NotationTable

• MCTS
• AlphaZero
• MuZero
• EfficientZero
• SampledMuZero
• GumbelMuZero

LightZero Implemented series

• 2018 Science AlphaZero: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
• 2019 MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• 2021 EfficientZero: Mastering Atari Games with Limited Data
• 2021 Sampled MuZero: Learning and Planning in Complex Action Spaces

AlphaGo series

• 2015 Nature AlphaGo Mastering the game of Go with deep neural networks and tree search
• 2017 Nature AlphaGo Zero Mastering the game of Go without human knowledge
• 2019 ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero
  • Code

MuZero series

• 2022 Online and Offline Reinforcement Learning by Planning with a Learned Model
• 2021 Vector Quantized Models for Planning
• 2022 Stochastic MuZero: Plannig in Stochastic Environments with A Learned Model
• 2021 Muesli: Combining Improvements in Policy Optimization.

MCTS Analysis

• 2020 Monte-Carlo Tree Search as Regularized Policy Optimization
• 2021 Self-Consistent Models and Values
• 2022 Adversarial Policies Beat Professional-Level Go AIs
• 2022 PNAS Acquisition of Chess Knowledge in AlphaZero.

MCTS Application

• 2022 Nature Discovering faster matrix multiplication algorithms with reinforcement learning
  • Code
• 2022 MuZero with Self-competition for Rate Control in VP9 Video Compression
• 2021 DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning
• 2019 Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving

ICML

• Efficient Learning for AlphaZero via Path Consistency 2022
  • Dengwei Zhao, Shikui Tu, Lei Xu
  • Key: limited amount of self-plays, path consistency (PC) optimality
  • ExpEnv: Go, Othello, Gomoku
• Convex Regularization in Monte-Carlo Tree Search 2021
  • Tuan Dam, Carlo D'Eramo, Jan Peters, Joni Pajarinen
  • Key: entropy-regularization backup operators, regret analysis, Tsallis etropy,
  • ExpEnv: synthetic tree, Atari
• Information Particle Filter Tree: An Online Algorithm for POMDPs with Belief-Based Rewards on Continuous Domains 2020
  • Johannes Fischer, Ömer Sahin Tas
  • Key: Continuous POMDP, Particle Filter Tree, information-based reward shaping, Information Gathering.
  • ExpEnv: POMDPs.jl framework
  • Code
• Retro*: Learning Retrosynthetic Planning with Neural Guided A* Search 2020
  • Binghong Chen, Chengtao Li, Hanjun Dai, Le Song
  • Key: chemical retrosynthetic planning, neural-based A*-like algorithm, ANDOR tree
  • ExpEnv: USPTO datasets
  • Code

ICLR

• SpeedyZero: Mastering Atari with Limited Data and Time 2023
  • Yixuan Mei, Jiaxuan Gao, Weirui Ye, Shaohuai Liu, Yang Gao, Yi Wu
  • Key: distributed RL system, Priority Refresh, Clipped LARS
  • ExpEnv: Atari
• Efficient Offline Policy Optimization with a Learned Model 2023
  • Zichen Liu, Siyi Li, Wee Sun Lee, Shuicheng YAN, Zhongwen Xu
  • Key: Regularized One-Step Model-based algorithm for Offline-RL
  • ExpEnv: Atari，BSuite
  • Code
• Enabling Arbitrary Translation Objectives with Adaptive Tree Search 2022
  • Wang Ling, Wojciech Stokowiec, Domenic Donato, Chris Dyer, Lei Yu, Laurent Sartran, Austin Matthews
  • Key: adaptive tree search, translation models, autoregressive models,
  • ExpEnv: Chinese–English and Pashto–English tasks from WMT2020, German–English from WMT2014
• What's Wrong with Deep Learning in Tree Search for Combinatorial Optimization 2022
  • Maximili1an Böther, Otto Kißig, Martin Taraz, Sarel Cohen, Karen Seidel, Tobias Friedrich
  • Key: Combinatorial optimization, open-source benchmark suite for the NP-hard MAXIMUM INDEPENDENT SET problem, an in-depth analysis of the popular guided tree search algorithm, compare the tree search implementations to other solvers
  • ExpEnv: NP-hard MAXIMUM INDEPENDENT SET.
  • Code
• Monte-Carlo Planning and Learning with Language Action Value Estimates 2021
  • Youngsoo Jang, Seokin Seo, Jongmin Lee, Kee-Eung Kim
  • Key: Monte-Carlo tree search with language-driven exploration, locally optimistic language value estimates,
  • ExpEnv: Interactive Fiction (IF) games
• Practical Massively Parallel Monte-Carlo Tree Search Applied to Molecular Design 2021
  • Xiufeng Yang, Tanuj Kr Aasawat, Kazuki Yoshizoe
  • Key: massively parallel Monte-Carlo Tree Search, molecular design, Hash-driven parallel search,
  • ExpEnv: octanol-water partition coefficient (logP) penalized by the synthetic accessibility (SA) and large Ring Penalty score.
• Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search 2020
  • Anji Liu, Jianshu Chen, Mingze Yu, Yu Zhai, Xuewen Zhou, Ji Liu
  • Key: parallel Monte-Carlo Tree Search, partition the tree into sub-trees efficiently, compare the observation ratio of each processor
  • ExpEnv: speedup and performance comparison on JOY-CITY game, average episode return on atari game
  • Code
• Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees 2020
  • Binghong Chen,  Bo Dai, Qinjie Lin, Guo Ye, Han Liu, Le Song
  • Key: meta path planning algorithm, exploits a novel neural architecture which can learn promising search directions from problem structures.
  • ExpEnv: a 2d workspace with a 2 DoF (degrees of freedom) point robot, a 3 DoF stick robot and a 5 DoF snake robot

NeurIPS

• Planning for Sample Efficient Imitation Learning 2022
  • Zhao-Heng Yin, Weirui Ye, Qifeng Chen, Yang Gao
  • Key: Behavioral Cloning，Adversarial Imitation Learning (AIL)，MCTS-based RL，
  • ExpEnv: DeepMind Control Suite
  • Code
• Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex 2022
  • Charles Lovering, Jessica Zosa Forde, George Konidaris, Ellie Pavlick, Michael L. Littman
  • Key: AlphaZero’s internal representations, model probing and behavioral tests, how these concepts are captured in the network.
  • ExpEnv: Hex
• Are AlphaZero-like Agents Robust to Adversarial Perturbations? 2022
  • Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng-Yu Tsai, I-Chen Wu, 4 Cho-Jui Hsieh
  • Key: adversarial states, first adversarial attack on Go AIs
  • ExpEnv: Go
• Monte Carlo Tree Descent for Black-Box Optimization 2022
  • Yaoguang Zhai, Sicun Gao
  • Key: Black-Box Optimization, how to further integrate samplebased descent for faster optimization.
  • ExpEnv: synthetic functions for nonlinear optimization, reinforcement learning problems in MuJoCo locomotion environments, and optimization problems in Neural Architecture Search (NAS).
• Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization 2022
  • Lei Song∗ , Ke Xue∗ , Xiaobin Huang, Chao Qian
  • Key: a low-dimensional subspace via MCTS, optimizes in the subspace with any Bayesian optimization algorithm.
  • ExpEnv: NAS-bench problems and MuJoCo locomotion
• Monte Carlo Tree Search With Iteratively Refining State Abstractions 2021
  • Samuel Sokota, Caleb Ho, Zaheen Ahmad, J. Zico Kolter
  • Key: stochastic environments, Progressive widening, abstraction refining,
  • ExpEnv: Blackjack, Trap, five by five Go.
• Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess 2021
  • Gregory Clark
  • Key: imperfect information, belief state with an unweighted particle filter, a novel stochastic abstraction of information states.
  • ExpEnv: reconnaissance blind chess
• POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysis 2020
  • Weichao Mao, Kaiqing Zhang, Qiaomin Xie, Tamer Ba¸sar
  • Key: continuous state-action spaces, Hierarchical Optimistic Optimization,
  • ExpEnv: CartPole, Inverted Pendulum, Swing-up, and LunarLander.
• Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search 2020
  • Linnan Wang, Rodrigo Fonseca, Yuandong Tian
  • Key: learns the partition of the search space using a few samples, a nonlinear decision boundary and learns a local model to pick good candidates.
  • ExpEnv: MuJoCo locomotion tasks, Small-scale Benchmarks,
• Mix and Match: An Optimistic Tree-Search Approach for Learning Models from Mixture Distributions 2020
  • Matthew Faw, Rajat Sen, Karthikeyan Shanmugam, Constantine Caramanis, Sanjay Shakkottai
  • Key: covariate shift problem, Mix&Match combines stochastic gradient descent (SGD) with optimistic tree search and model re-use (evolving partially trained models with samples from different mixture distributions)
  • Code

Other Conference or Journal

• On Monte Carlo Tree Search and Reinforcement Learning Journal of Artificial Intelligence Research 2017.
• Sample-Efficient Neural Architecture Search by Learning Actions for Monte Carlo Tree Search IEEE Transactions on Pattern Analysis and Machine Intelligence 2022.