Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs

Figure: Recovered 3D shape and rotation&relighting effects using GAN2Shape.

Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, Ping Luo
ICLR2021 (Oral)

[Paper] [Project Page]

In this repository, we present GAN2Shape, which reconstructs the 3D shape of an image using off-the-shelf 2D image GANs in an unsupervised manner. Our method does not rely on mannual annotations or external 3D models, yet it achieves high-quality 3D reconstruction, object rotation, and relighting effects.

Requirements

• python>=3.6
• pytorch=1.1 or 1.2
• neural_renderer

  pip install neural_renderer_pytorch  # or follow the guidance at https://github.com/elliottwu/unsup3d

• mmcv

  pip install mmcv

• other dependencies

  conda install -c conda-forge scikit-image matplotlib opencv pyyaml tensorboardX

Dataset and pre-trained weights

To download dataset and pre-trained weights, simply run:

sh scripts/download.sh

Training

Before training, you may optionally compile StyleGAN2 operations, which would be faster:

cd gan2shape/stylegan/stylegan2-pytorch/op
python setup.py install
cd ../../../..

Example1: training on car images:

sh scripts/run_car.sh

This would run on 4 GPUs by default. You can view the results at results/car/images or Tensorboard.

Example2: training on Celeba images:

sh scripts/run_celeba.sh

This by default uses our provided pre-trained weights. You can also perform joint pre-training via:

sh scripts/run_celeba-pre.sh

Example3: evaluating on synface (BFM) dataset:

sh scripts/run_synface.sh

This by default uses our provided pre-trained weights. You can also perform joint pre-training via:

sh scripts/run_synface-pre.sh

If you want to train on new StyleGAN2 samples, simply run the following script to generate new samples:

sh scripts/run_sample.sh

Note:
- For human and cat faces, we perform joint training before instance-specific training, which produces better results.
- For car and church, the quality of StyleGAN2 samples vary a lot, thus our approach may not produce good result on every sample. The downloaded dataset contains examples of good samples.

Acknowledgement

Part of the code is borrowed from Unsup3d and StyleGAN2.
Colab demo reproduced by ucalyptus: Link

BibTeX

@inproceedings{pan2020gan2shape,
  title   = {Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs},
  author  =  {Pan, Xingang and Dai, Bo and Liu, Ziwei and Loy, Chen Change and Luo, Ping},
  booktitle = {International Conference on Learning Representations},
  year    = {2021}
}