ADGAN

PyTorch | project page | paper

PyTorch implementation for controllable person image synthesis.

ADGAN: Controllable Person Image Synthesis with Attribute-Decomposed GAN,
Yifang Men, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: Proc. Computer Vision and Pattern Recognition (CVPR Oral), 2020 arXiv preprint (arXiv 2003.12267)

ADGAN++: Controllable Image Synthesis with Attribute-Decomposed GAN,
Guo Pu*, Yifang Men*, Yiming Mao, Yuning Jiang, Wei-ying Ma, Zhouhui Lian
In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. arXiv preprint (comming soon) Code

Updates

-(03/22/2022)ADGAN++, an extension vision with improved methods and more applications will be released soon.

Demo

Component Attribute Transfer

Pose Transfer

Requirement

• python 3
• pytorch(>=1.0)
• torchvision
• numpy
• scipy
• scikit-image
• pillow
• pandas
• tqdm
• dominate

Getting Started

You can directly download our generated images (in Deepfashion) from Google Drive.

Installation

• Clone this repo:

git clone https://github.com/menyifang/ADGAN.git
cd ADGAN

Data Preperation

We use DeepFashion dataset and provide our dataset split files, extracted keypoints files and extracted segmentation files for convience.

The dataset structure is recommended as:

+—deepfashion
|   +—fashion_resize
|       +--train (files in 'train.lst')
|          +-- e.g. fashionMENDenimid0000008001_1front.jpg
|       +--test (files in 'test.lst')
|          +-- e.g. fashionMENDenimid0000056501_1front.jpg
|       +--trainK(keypoints of person images)
|          +-- e.g. fashionMENDenimid0000008001_1front.jpg.npy
|       +--testK
|          +-- e.g. fashionMENDenimid0000056501_1front.jpg.npy
|   +—semantic_merge
|   +—fashion-resize-pairs-train.csv
|   +—fashion-resize-pairs-test.csv
|   +—fashion-resize-annotation-pairs-train.csv
|   +—fashion-resize-annotation-pairs-test.csv
|   +—train.lst
|   +—test.lst
|   +—vgg19-dcbb9e9d.pth
|   +—vgg_conv.pth
...

1. Person images

• Download person images from deep fasion dataset in-shop clothes retrival benchmark and download dataset split from Google Drive.
• Crop the images. Split the raw images into the train split (fashion_resize/train) and the test split (fashion_resize/test). Launch

python tool/generate_fashion_datasets.py

Note: In our settings, we crop the images of DeepFashion into the resolution of 176x256 in a center-crop manner.

2. Keypoints files

• Download train/test pairs and train/test key points annotations from Google Drive, including fashion-resize-pairs-train.csv, fashion-resize-pairs-test.csv, fashion-resize-annotation-train.csv, fashion-resize-annotation-train.csv. Put these four files under the deepfashion directory.
• Generate the pose heatmaps. Launch

python tool/generate_pose_map_fashion.py

3. Segmentation files

• Extract human segmentation results from existing human parser (e.g. Look into Person) and merge into 8 categories. Our segmentation results are provided in Google Drive, including ‘semantic_merge2’ and ‘semantic_merge3’ in different merge manner. Put one of them under the deepfashion directory.

Optionally, you can also generate these files by yourself.

1. Keypoints files

We use OpenPose to generate keypoints.

• Download pose estimator from Google Drive. Put it under the root folder ADGAN.
• Change the paths input_folder and output_path in tool/compute_coordinates.py. And then launch

python2 compute_coordinates.py

2. Dataset split files

python2 tool/create_pairs_dataset.py

Train a model

bash ./scripts/train.sh 

Test a model

Download our pretrained model from Google Drive. Modify your data path and launch

bash ./scripts/test.sh 

Evaluation

We adopt SSIM, IS, DS, CX for evaluation. This part is finished by Yiming Mao.

1) SSIM

For evaluation, Tensorflow 1.4.1(python3) is required.

python tool/getMetrics_market.py

2) DS Score

Download pretrained on VOC 300x300 model and install propper caffe version SSD. Put it in the ssd_score forlder.

python compute_ssd_score_fashion.py --input_dir path/to/generated/images

3) CX (Contextual Score)

Refer to folder ‘cx’ to compute contextual score.

Citation

If you use this code for your research, please cite our paper:

@inproceedings{men2020controllable,
  title={Controllable Person Image Synthesis with Attribute-Decomposed GAN},
  author={Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Computer Vision and Pattern Recognition (CVPR), 2020 IEEE Conference on},
  year={2020}
}

@inproceedings{pu2022controllable,
  title={Controllable Image Synthesis with Attribute-Decomposed GAN},
  author={Pu, Guo and Men, Yifang and Mao, Yiming and Jiang, Yuning and Ma, Wei-Ying and Lian, Zhouhui},
  booktitle={Pattern Analysis and Machine Intelligence (TPAMI), 2022 IEEE Transactions on},
  year={2022}
}

Acknowledgments

Our code is based on PATN and thanks for their great work.