SPIRAL in TensorFlow (in progress)
TensorFlow implementation of Synthesizing Programs for Images using Reinforced Adversarial Learning (SPIRAL).
SPIRAL is an adversarially trained agent that generates a program which is executed by a graphics engine to interpret and sample images. This agent is trained to fool a discriminator with a distributed reinforcement learning without any supervision.
In short, Distributed RL + GAN + Program synthesis.
Prerequisites
- Python 2.7
- MyPaint 1.2.x
- TensorFlow 1.6.0
Usage
Install prerequisites with:
./install.sh
pip install -r requirements.txt
To debug a SPIARL model:
python run.py --num_workers 8 --env simple --episode_length=1 \
--location_size=8 --conditional=True \
--loss=l2 --policy_batch_size=1
To train a SPIARL model:
python run.py --num_workers 16 --env simple_mnist --episode_length=3 \
--color_channel=1 --location_size=32 --loss=gan --num_gpu=1 \
--disc_dim=8 --conditional=False \
--mnist_nums=1,7 --jump=False --curve=False
python run.py --num_workers 24 --env simple_mnist --episode_length=6 \
--color_channel=1 --location_size=32 --loss=gan --num_gpu=2 \
--disc_dim=64 --conditional=False \
--mnist_nums=0,1,2,3,4,5,6,7,8,9 --jump=True
python run.py --num_workers 12 --env simple_mnist --episode_length=2 \
--color_channel=1 --location_size=32 --conditional=True \
--mnist_nums=1 --loss=gan
python run.py --num_workers 24 --env simple_mnist --episode_length=3 \
--color_channel=1 --location_size=32 --conditional=True \
--mnist_nums=1,2,7 --loss=l2
python run.py --num_workers 24 --env simple_mnist --episode_length=3 \
--color_channel=1 --location_size=32 --conditional=True \
--mnist_nums=1,2,7 --loss=gan --num_gpu=2
python run.py --num_workers 24 --env simple_mnist --episode_length=5 \
--color_channel=1 --location_size=32 --conditional=True \
--mnist_nums=0,1,2,7 --loss=gan --num_gpu=2
Results
(in progress)
Random generated samples at early stage:
Incorrectly converged samples at early stage:
Tensorboard:
To-do
- IMPALA A2C
- IMPALA V-trace
- Simple environment (debugging)
- Find a correct libmypaint setting
- MNIST environment
- ReplayThread (
--loss=gan) -
--num_gpu=2test -
--conditional=True(need more details) - Replay memory needs more detailed information
- Population Based Training (to be honest, I don't have any plan for this)
References
This code is heavily based on openai/universe-starter-agent.
- Population Based Training of Neural Networks
- Asynchronous Methods for Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Author
Taehoon Kim / @carpedm20