clovaai/stargan-v2 clovaai/stargan-v2

  • Stars
    star
    3,478
  • Rank 12,808 (Top 0.3 %)
  • Language
    Python
  • License
    Other
  • Created almost 5 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
clovaai/stargan-v2
github

clovaai

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

StarGAN v2 - Official PyTorch Implementation (CVPR 2020)

StarGAN v2 - Official PyTorch Implementation

StarGAN v2: Diverse Image Synthesis for Multiple Domains
Yunjey Choi*, Youngjung Uh*, Jaejun Yoo*, Jung-Woo Ha
In CVPR 2020. (* indicates equal contribution)

Paper: https://arxiv.org/abs/1912.01865
Video: https://youtu.be/0EVh5Ki4dIY

Abstract: A good image-to-image translation model should learn a mapping between different visual domains while satisfying the following properties: 1) diversity of generated images and 2) scalability over multiple domains. Existing methods address either of the issues, having limited diversity or multiple models for all domains. We propose StarGAN v2, a single framework that tackles both and shows significantly improved results over the baselines. Experiments on CelebA-HQ and a new animal faces dataset (AFHQ) validate our superiority in terms of visual quality, diversity, and scalability. To better assess image-to-image translation models, we release AFHQ, high-quality animal faces with large inter- and intra-domain variations. The code, pre-trained models, and dataset are available at clovaai/stargan-v2.

Teaser video

Click the figure to watch the teaser video.

IMAGE ALT TEXT HERE

TensorFlow implementation

The TensorFlow implementation of StarGAN v2 by our team member junho can be found at clovaai/stargan-v2-tensorflow.

Software installation

Clone this repository:

git clone https://github.com/clovaai/stargan-v2.git
cd stargan-v2/

Install the dependencies:

conda create -n stargan-v2 python=3.6.7
conda activate stargan-v2
conda install -y pytorch=1.4.0 torchvision=0.5.0 cudatoolkit=10.0 -c pytorch
conda install x264=='1!152.20180717' ffmpeg=4.0.2 -c conda-forge
pip install opencv-python==4.1.2.30 ffmpeg-python==0.2.0 scikit-image==0.16.2
pip install pillow==7.0.0 scipy==1.2.1 tqdm==4.43.0 munch==2.5.0

Datasets and pre-trained networks

We provide a script to download datasets used in StarGAN v2 and the corresponding pre-trained networks. The datasets and network checkpoints will be downloaded and stored in the data and expr/checkpoints directories, respectively.

CelebA-HQ. To download the CelebA-HQ dataset and the pre-trained network, run the following commands:

bash download.sh celeba-hq-dataset
bash download.sh pretrained-network-celeba-hq
bash download.sh wing

AFHQ. To download the AFHQ dataset and the pre-trained network, run the following commands:

bash download.sh afhq-dataset
bash download.sh pretrained-network-afhq

Generating interpolation videos

After downloading the pre-trained networks, you can synthesize output images reflecting diverse styles (e.g., hairstyle) of reference images. The following commands will save generated images and interpolation videos to the expr/results directory.

CelebA-HQ. To generate images and interpolation videos, run the following command:

python main.py --mode sample --num_domains 2 --resume_iter 100000 --w_hpf 1 \
               --checkpoint_dir expr/checkpoints/celeba_hq \
               --result_dir expr/results/celeba_hq \
               --src_dir assets/representative/celeba_hq/src \
               --ref_dir assets/representative/celeba_hq/ref

To transform a custom image, first crop the image manually so that the proportion of face occupied in the whole is similar to that of CelebA-HQ. Then, run the following command for additional fine rotation and cropping. All custom images in the inp_dir directory will be aligned and stored in the out_dir directory.

python main.py --mode align \
               --inp_dir assets/representative/custom/female \
               --out_dir assets/representative/celeba_hq/src/female

AFHQ. To generate images and interpolation videos, run the following command:

python main.py --mode sample --num_domains 3 --resume_iter 100000 --w_hpf 0 \
               --checkpoint_dir expr/checkpoints/afhq \
               --result_dir expr/results/afhq \
               --src_dir assets/representative/afhq/src \
               --ref_dir assets/representative/afhq/ref

Evaluation metrics

To evaluate StarGAN v2 using Frรฉchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS), run the following commands:

# celeba-hq
python main.py --mode eval --num_domains 2 --w_hpf 1 \
               --resume_iter 100000 \
               --train_img_dir data/celeba_hq/train \
               --val_img_dir data/celeba_hq/val \
               --checkpoint_dir expr/checkpoints/celeba_hq \
               --eval_dir expr/eval/celeba_hq

# afhq
python main.py --mode eval --num_domains 3 --w_hpf 0 \
               --resume_iter 100000 \
               --train_img_dir data/afhq/train \
               --val_img_dir data/afhq/val \
               --checkpoint_dir expr/checkpoints/afhq \
               --eval_dir expr/eval/afhq

Note that the evaluation metrics are calculated using random latent vectors or reference images, both of which are selected by the seed number. In the paper, we reported the average of values from 10 measurements using different seed numbers. The following table shows the calculated values for both latent-guided and reference-guided synthesis.

Dataset FID (latent) LPIPS (latent) FID (reference) LPIPS (reference) Elapsed time
celeba-hq 13.73 ยฑ 0.06 0.4515 ยฑ 0.0006 23.84 ยฑ 0.03 0.3880 ยฑ 0.0001 49min 51s
afhq 16.18 ยฑ 0.15 0.4501 ยฑ 0.0007 19.78 ยฑ 0.01 0.4315 ยฑ 0.0002 64min 49s

Training networks

To train StarGAN v2 from scratch, run the following commands. Generated images and network checkpoints will be stored in the expr/samples and expr/checkpoints directories, respectively. Training takes about three days on a single Tesla V100 GPU. Please see here for training arguments and a description of them.

# celeba-hq
python main.py --mode train --num_domains 2 --w_hpf 1 \
               --lambda_reg 1 --lambda_sty 1 --lambda_ds 1 --lambda_cyc 1 \
               --train_img_dir data/celeba_hq/train \
               --val_img_dir data/celeba_hq/val

# afhq
python main.py --mode train --num_domains 3 --w_hpf 0 \
               --lambda_reg 1 --lambda_sty 1 --lambda_ds 2 --lambda_cyc 1 \
               --train_img_dir data/afhq/train \
               --val_img_dir data/afhq/val

Animal Faces-HQ dataset (AFHQ)

We release a new dataset of animal faces, Animal Faces-HQ (AFHQ), consisting of 15,000 high-quality images at 512ร—512 resolution. The figure above shows example images of the AFHQ dataset. The dataset includes three domains of cat, dog, and wildlife, each providing about 5000 images. By having multiple (three) domains and diverse images of various breeds per each domain, AFHQ sets a challenging image-to-image translation problem. For each domain, we select 500 images as a test set and provide all remaining images as a training set. To download the dataset, run the following command:

bash download.sh afhq-dataset

[Update: 2021.07.01] We rebuild the original AFHQ dataset by using high-quality resize filtering (i.e., Lanczos resampling). Please see the clean FID paper that brings attention to the unfortunate software library situation for downsampling. We thank to Alias-Free GAN authors for their suggestion and contribution to the updated AFHQ dataset. If you use the updated dataset, we recommend to cite not only our paper but also their paper.

The differences from the original dataset are as follows:

  • We resize the images using Lanczos resampling instead of nearest neighbor downsampling.
  • About 2% of the original images had been removed. So the set is now has 15803 images, whereas the original had 16130.
  • Images are saved as PNG format to avoid compression artifacts. This makes the files bigger than the original, but it's worth it.

To download the updated dataset, run the following command:

bash download.sh afhq-v2-dataset

License

The source code, pre-trained models, and dataset are available under Creative Commons BY-NC 4.0 license by NAVER Corporation. You can use, copy, tranform and build upon the material for non-commercial purposes as long as you give appropriate credit by citing our paper, and indicate if changes were made.

For business inquiries, please contact [email protected].
For technical and other inquires, please contact [email protected].

Citation

If you find this work useful for your research, please cite our paper:

@inproceedings{choi2020starganv2,
  title={StarGAN v2: Diverse Image Synthesis for Multiple Domains},
  author={Yunjey Choi and Youngjung Uh and Jaejun Yoo and Jung-Woo Ha},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2020}
}

Acknowledgements

We would like to thank the full-time and visiting Clova AI Research (now NAVER AI Lab) members for their valuable feedback and an early review: especially Seongjoon Oh, Junsuk Choe, Muhammad Ferjad Naeem, and Kyungjune Baek. We also thank Alias-Free GAN authors for their contribution to the updated AFHQ dataset.

More Repositories

1

donut

Official Implementation of OCR-free Document Understanding Transformer (Donut) and Synthetic Document Generator (SynthDoG), ECCV 2022
Python
5,573
star
2

deep-text-recognition-benchmark

Text recognition (optical character recognition) with deep learning methods, ICCV 2019
Jupyter Notebook
3,692
star
3

CRAFT-pytorch

Official implementation of Character Region Awareness for Text Detection (CRAFT)
Python
3,024
star
4

CutMix-PyTorch

Official Pytorch implementation of CutMix regularizer
Python
1,211
star
5

voxceleb_trainer

In defence of metric learning for speaker recognition
Python
1,029
star
6

WCT2

Software that can perform photorealistic style transfer without the need of any post-processing steps.
Python
869
star
7

synthtiger

Official Implementation of SynthTIGER (Synthetic Text Image Generator), ICDAR 2021
Python
482
star
8

tunit

Rethinking the Truly Unsupervised Image-to-Image Translation - Official PyTorch Implementation (ICCV 2021)
Python
452
star
9

rexnet

Official Pytorch implementation of ReXNet (Rank eXpansion Network) with pretrained models
Python
451
star
10

AdamP

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)
Python
411
star
11

overhaul-distillation

Official PyTorch implementation of "A Comprehensive Overhaul of Feature Distillation" (ICCV 2019)
Python
409
star
12

cord

CORD: A Consolidated Receipt Dataset for Post-OCR Parsing
384
star
13

cutblur

Rethinking Data Augmentation for Image Super-resolution (CVPR 2020)
Jupyter Notebook
379
star
14

wsolevaluation

Evaluating Weakly Supervised Object Localization Methods Right (CVPR 2020)
Python
331
star
15

assembled-cnn

Tensorflow implementation of "Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network"
Python
329
star
16

generative-evaluation-prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.
Python
239
star
17

ext_portrait_segmentation

Python
238
star
18

ClovaCall

ClovaCall dataset and Pytorch LAS baseline code (Interspeech 2020)
Python
218
star
19

fewshot-font-generation

The unified repository for few-shot font generation methods. This repository includes FUNIT (ICCV'19), DM-Font (ECCV'20), LF-Font (AAAI'21) and MX-Font (ICCV'21).
Python
203
star
20

stargan-v2-tensorflow

StarGAN v2 - Official Tensorflow Implementation (CVPR 2020)
Python
187
star
21

EXTD_Pytorch

Official EXTD Pytorch code
Python
187
star
22

CLEval

CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks
Python
185
star
23

TedEval

TedEval: A Fair Evaluation Metric for Scene Text Detectors
Python
176
star
24

rebias

Official Pytorch implementation of ReBias (Learning De-biased Representations with Biased Representations), ICML 2020
Python
168
star
25

aasist

Official PyTorch implementation of "AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks"
Python
167
star
26

SATRN

Official Tensorflow Implementation of SATRN (CVPR Workshop WTDDLE 2020)
Python
162
star
27

lffont

Official PyTorch implementation of LF-Font (Few-shot Font Generation with Localized Style Representations and Factorization) AAAI 2021
Python
156
star
28

bros

Python
156
star
29

som-dst

SOM-DST: Efficient Dialogue State Tracking by Selectively Overwriting Memory (ACL 2020)
Python
150
star
30

mxfont

Official PyTorch implementation of MX-Font (Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts) ICCV 2021
Python
148
star
31

dmfont

Official PyTorch implementation of DM-Font (ECCV 2020)
Python
133
star
32

rainbow-memory

Official pytorch implementation of Rainbow Memory (CVPR 2021)
Python
119
star
33

FocusSeq2Seq

[EMNLP 2019] Mixture Content Selection for Diverse Sequence Generation (Question Generation / Abstractive Summarization)
Python
113
star
34

attention-feature-distillation

Official implementation for (Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching, AAAI-2021)
Python
111
star
35

frostnet

FrostNet: Towards Quantization-Aware Network Architecture Search
Python
106
star
36

webvicob

Official Implementation of Web-based Visual Corpus Builder (Webvicob), ICDAR 2023
Python
101
star
37

length-adaptive-transformer

Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)
Python
99
star
38

spade

Python
81
star
39

embedding-expansion

Official MXNet implementation of "Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning" (CVPR 2020)
Python
76
star
40

symmetrical-synthesis

Official Tensorflow implementation of "Symmetrical Synthesis for Deep Metric Learning" (AAAI 2020)
Python
71
star
41

units

Python
70
star
42

lookwhostalking

Look Whoโ€™s Talking: Active Speaker Detection in the Wild
Python
70
star
43

subword-qac

Subword Language Model for Query Auto-Completion
Python
67
star
44

ssmix

Official PyTorch Implementation of SSMix (Findings of ACL 2021)
Python
60
star
45

SSUL

[NeurIPS 2021] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
Python
59
star
46

BESTIE

[CVPR 2022] Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement
Python
55
star
47

PointWSSIS

[CVPR2023] The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation
Python
55
star
48

c3_sinet

Python
52
star
49

puridiver

Official PyTorch Implementation of PuriDivER CVPR 2022.
Python
45
star
50

EResFD

Lightweight Face Detector from CLOVA
Python
44
star
51

minimal-rnr-qa

[NAACL 2021] Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
Python
36
star
52

ECLIPSE

(CVPR 2024) ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning
Python
34
star
53

group-transformer

Official code for Group-Transformer (Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model, COLING-2020).
Python
25
star
54

ProxyDet

Official implementation of the paper "ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection"
Python
22
star
55

GeNAS

Official pytorch implementation for GeNAS: Neural Architecture Search with Better Generalization
Python
15
star
56

meev

Python
12
star
57

pkm-transformers

Official implementation of PKM-augmented language models (Findings of EMNLP 2020)
9
star
58

DCutMix

DCutMix official repo
Python
8
star
59

TVQ-VAE

Official pytorch implementation for TVQ-VAE
Jupyter Notebook
8
star
60

textual-kd-slu

Official Implementation of Textual KD SLU (ICASSP 2021)
Python
6
star
61

vat-d

Official Implementation of VAT-D
Python
5
star
62

ActiveASR_AugCR

Repositoty for Efficient Active Learning for Automatic Speech Recognition via Augmented Consistency Regularization
3
star
63

WSSS-BED

Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation
Python
1
star