Recycle-GAN :Unsupervised Video Retargeting
This repository provides the code for our work on unsupervised video retargeting.
@inproceedings{Recycle-GAN,
author = {Aayush Bansal and
Shugao Ma and
Deva Ramanan and
Yaser Sheikh},
title = {Recycle-GAN: Unsupervised Video Retargeting},
booktitle = {ECCV},
year = {2018},
}Acknowledgements: This code borrows heavily from the PyTorch implementation of Cycle-GAN and Pix2Pix. A huge thanks to them!
John Oliver to Stephen Colbert
Click above to see video!
Video by CMU Media folks
Click above to see video!
Introduction
We use this formulation in our ECCV'18 paper on unsupervised video retargeting for various domains where space and time information matters such as face retargeting. Without any manual annotation, our approach could learn retargeting from one domain to another.
Using the Code
The repository contains the code for training a network for retargeting from one domain to another, and use a trained module for this task. Following are the things to consider with this code:
Requirements
- Linux or MacOS
- Python 3
- Pytorch 0.4
- NVIDIA GPU + CUDA CuDNN
Python Dependencies
- numpy 1.15.0
- torch 0.4.1.post2
- torchvision 0.2.2
- visdom
- dominate
Run the following command to install automatically:
pip install requirements.txt
Data pre-processing
For each task, create a new folder in "dataset/" directory. The images from two domains are placed respectively in "trainA/" and "trainB/". Each image file consists of horizontally concatenated images, "{t, t+1, t+2}" frames from the video. The test images are placed in "testA/" and "testB/". Since we do not use temporal information at test time, the test data consists of single image "{t}".
Training
There are two training modules in "scripts/" directory: (1). Recycle-GAN, (2). ReCycle-GAN
Recycle-GAN
Recycle-GAN is the model described in the paper and is used for most examples in the paper, specifically face to face, flower to flower, clouds and wind synthesis, sunrise and sunset.
ReCycle-GAN
ReCycle-GAN is mostly similar to Recycle-GAN. Additionally, we also use vanilla cycle-losses from CycleGAN between corresponding source and target frames. We found this module useful for tasks such as unpaired image to labels, and labels to image on VIPER dataset, image to normals, and normals to image on NYU-v2 depth dataset.
Prediction Model
There are two prediction model used in this work: (1). simple U-Net, (2). higher-capacity prediction.
unet-128, unet-256
If you want to use this prediction module, please set the flag "--which_model_netP" to "unet_128" and "unet_256" respectively.
prediction
An advanced version of prediction module is a higher capacity module by setting the flag "--which_model_netP" to "prediction".
Observation about training:
We observed that model converges in 20-40 epochs when sufficiently large data is used. For smaller datasets (ones having 1000 images or less), it is suitable to let it train for longer.
Test
At test time, we do inference per image (as mentioned previously). The test code is based on cycle-gan.
Data & Trained Models:
Please use following links to download Face, Flowers, and Viper data:
Please contact Aayush Bansal for any specific data or trained models, or for any other information.