clovaai/tunit clovaai/tunit

  • Stars
    star
    452
  • Rank 96,126 (Top 2 %)
  • Language
    Python
  • License
    Other
  • Created over 4 years ago
  • Updated about 2 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
clovaai/tunit
github

clovaai

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Rethinking the Truly Unsupervised Image-to-Image Translation - Official PyTorch Implementation (ICCV 2021)

Rethinking the Truly Unsupervised Image-to-Image Translation
(ICCV 2021)

teaser

Each image is generated with the source image in the left and the average style vector of each cluster. The network is trained under fully unsupervised manner.

Official pytorch implementation of "Rethinking the Truly Unsupervised Image-to-Image Translation"

Rethinking the Truly Unsupervised Image-to-Image Translation
Kyungjune Baek1*, Yunjey Choi2, Youngjung Uh1, Jaejun Yoo3, Hyunjung Shim1
* Work done during his internship at Clova AI Research
1 Yonsei University
2 NAVER AI Lab.
3 UNIST

Absract Every recent image-to-image translation model inherently requires either image-level (i.e. input-output pairs) or set-level (i.e. domain labels) supervision. However, even set-level supervision can be a severe bottleneck for data collection in practice. In this paper, we tackle image-to-image translation in a fully unsupervised setting, i.e., neither paired images nor domain labels. To this end, we propose a truly unsupervised image-to-image translation model (TUNIT) that simultaneously learns to separate image domains and translates input images into the estimated domains. Experimental results show that our model achieves comparable or even better performance than the set-level supervised model trained with full labels, generalizes well on various datasets, and is robust against the choice of hyperparameters (e.g. the preset number of pseudo domains). Furthermore, TUNIT can be easily extended to semi-supervised learning with a few labeled data.

Requirement

Library

pip install -r requirements.txt

* pytorch==1.1.0 or 1.2.0  
* tqdm  
* opencv-python  
* scipy  
* sklearn
* matplotlib  
* pillow  
* tensorboardX

Dataset

Project
|--- tunit
|          |--- main.py
|          |--- train
|                 |--- train_unsupervised.py
|                 |--- ...
|
|--- data
       |--- afhq
             |--- train
             |--- test
       |--- animal_faces
             |--- n02085620
             |--- n02085782
             |--- ...
       |--- ffhq
             |--- images
                    |--- 000001.jpg
                    |--- ...
       |--- lsun_car
             |--- images
                    |--- 000001.jpg
                    |--- ...

Then, call --data_path='../data'

Hardware

  • This source code is mainly tested on V100 and P40.

How to Run (Quick Start)

After setting the dataset directory, the code can be easily run by the scripts below.

Train on local

Supervised
python main.py --gpu $GPU_TO_USE --p_semi 1.0 --dataset animal_faces --data_path='../data'

Semi-supervised
python main.py --gpu $GPU_TO_USE --p_semi 0.5 --dataset animal_faces --data_path='../data'

Unsupervised
python main.py --gpu $GPU_TO_USE --p_semi 0.0 --dataset animal_faces --data_path='../data'

Test on local

python main.py --gpu $GPU_TO_USE --validation --load_model $DIR_TO_LOAD --dataset animal_faces

Monitoring

tensorboard --logdir=$DIR/events --port=$PORT

Actual example

Train
python main.py --gpu 0 --dataset animal_faces --output_k 10 --data_path '../data' --p_semi 0.0
python main.py --gpu 0 --dataset animal_faces --output_k 10 --data_path '../data' --p_semi 0.2
python main.py --gpu 0 --dataset afhq_cat --output_k 10 --data_path '../data' --p_semi 0.0
python main.py --gpu 1 --dataset animal_faces --data_path '../data' --p_semi 1.0
python main.py --gpu 0,1 --dataset summer2winter --output_k 2 --data_path '../data' --p_semi 0.0 --img_size 256 --batch_size 16 --ddp

Test
python main.py --gpu 0 --dataset animal_faces --output_k 10 --data_path '../data' --validation --load_model GAN_20190101_101010
python main.py --gpu 1 --dataset afhq_cat --output_k 10 --data_path '../data' --validation --load_model GAN_20190101_101010
python main.py --gpu 2 --dataset summer2winter --output_k 2 --data_path '../data' --validation --load_model GAN_20190101_101010

Monitoring - open terminal at ./tunit/logs
tensorboard --logdir=./GAN_20200101_101010/events

Pretrained Model

Download

One Drive

  • Download folders to load, then place the folder under 'logs'.
  • Links of google drive are deprecated.
Project
|--- tunit
|          |--- main.py
|          |--- logs
|                 |--- animalFaces10_0_00
|                               |--- checkpoint.txt
|                               |--- model_4568.ckpt
|          |--- train
|                 |--- train_unsupervised.py
|                 |--- ...

Then, RUN
python main.py --gpu 0 --dataset animal_faces --output_k 10 --img_size 128 --data_path $DATAPATH --validation --load_model animalFaces10_0_00 --p_semi 0.0

How to run

AFHQ Cat
python main.py --gpu 0 --dataset afhq_cat --output_k 10 --img_size 128 --data_path $DATAPATH --validation --load_model afhq_cat_128
python main.py --gpu 0 --dataset afhq_cat --output_k 10 --img_size 256 --data_path $DATAPATH --validation --load_model afhq_cat_256

AFHQ Dog
python main.py --gpu 0 --dataset afhq_dog --output_k 10 --img_size 128 --data_path $DATAPATH --validation --load_model afhq_dog_128
python main.py --gpu 0 --dataset afhq_dog --output_k 10 --img_size 256 --data_path $DATAPATH --validation --load_model afhq_dog_256


AFHQ Wild
python main.py --gpu 0 --dataset afhq_wild --output_k 10 --img_size 128 --data_path $DATAPATH --validation --load_model afhq_wild_128
python main.py --gpu 0 --dataset afhq_wild --output_k 10 --img_size 256 --data_path $DATAPATH --validation --load_model afhq_wild_256

AnimalFaces-10
python main.py --gpu 0 --dataset animal_faces --output_k 10 --img_size 128 --data_path $DATAPATH --validation --load_model animalFaces10_0_00 --p_semi 0.0
python main.py --gpu 0 --dataset animal_faces --output_k 10 --img_size 128 --data_path $DATAPATH --validation --load_model animalFaces10_0_20 --p_semi 0.2

Explanation for codes

The validation generates 200 images per args.iters iterations. To reduce the number of images, please adjust the validation frequency. The checkpoint file is saved per ((args.epochs//10) * args.iters) iterations. Or comment out validation.py#L81 to validation.py#L162.

  • For more classes on AnimalFaces, change the list at main.py#L227 then, set args.output_k to len(args.att_to_use)
    • ex) args.att_to_use = [i for i in range(100)] then, run: python main.py --output_k 100 ...

Arguments

  • batch_size, img_size, data_path and p_semi are frequently speified.
  • Please refer "help" of the arguments in main.py.

Code Structure

  • main.py
    • Execute main.py to run the codes.
    • The script builds networks, optimizers and data loaders, and manages the checkpoint files.
  • datasets
    • custom_dataset.py
      • Basically, it is the same as ImageFolder but contains remap procedure of class numbers.
    • datasetgetter.py
      • Returns dataset instance of the dataset specified by args.dataset.
      • The instance returns original image, transformed image and its ground truth label.
  • models
    • blocks.py
      • Blocks for building networks.
      • This code is based on FUNIT repos.
    • guidingNet.py
      • Definition of guiding network.
    • discriminator.py
      • Definition of discriminator.
      • The architecture is based on StarGANv2, but it contains two residual blocks for each resolution.
    • generator.py
      • Definition of generator.
      • It consists of decoder, content encoder and MLP for AdaIN.
      • This code is from FUNIT repos.
  • train
    • train_unsupervised.py
      • It is called by setting --p_semi to 0.0
      • This mode does not utilize the labels at all.
    • train_supervised.py
      • It is called by setting --p_semi to 1.0
      • This mode fully utilizes the labels.
    • train_semisupervised.py
      • It is called by setting --p_semi between 0.0 to 1.0.
      • This mode utilizes (--p_semi * 100)% labels.
  • validation
    • cluster_eval.py
    • eval_metrics.py
      • These two scripts contain the functions for evaluating the classification performance.
      • These are from IIC repos.
    • plot_tsne.py (can be removed)
      • For plotting t-SNE.
    • validation.py
      • Generate fake samples and calculate FID.
  • tools
    • utils.py
      • Functions and class for logger, make folders, averageMeter and add logs.
    • ops.py
      • Queue operation and loss functions.
  • resrc
    • For image files of README.md

You can change the adversarial loss by modifying calc_adv_loss in ops.py. For the different strategy of training, please refer the files in train.

Results

afhq_cat afhq_dog afhq_wild ffhq lsun

Each image is generated with the source image in left and the average vector of reference images. The network is trained under fully unsupervised manner.

License

TUNIT is distributed under MIT unless the header specifies another license.

Copyright (c) 2020-present NAVER Corp.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORTd OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

The pretrained models is covered by Creative Commons BY-NC 4.0 license by NAVER Corporation. You can use, copy, tranform and build upon the material for non-commercial purposes as long as you give appropriate credit by citing our paper, and indicate if changes were made.

Citation

If you find this work useful for your research, please cite our paper:

@InProceedings{Baek_2021_ICCV,
    author    = {Baek, Kyungjune and Choi, Yunjey and Uh, Youngjung and Yoo, Jaejun and Shim, Hyunjung},
    title     = {Rethinking the Truly Unsupervised Image-to-Image Translation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {14154-14163}
}

More Repositories

1

donut

Official Implementation of OCR-free Document Understanding Transformer (Donut) and Synthetic Document Generator (SynthDoG), ECCV 2022
Python
5,573
star
2

deep-text-recognition-benchmark

Text recognition (optical character recognition) with deep learning methods, ICCV 2019
Jupyter Notebook
3,692
star
3

stargan-v2

StarGAN v2 - Official PyTorch Implementation (CVPR 2020)
Python
3,459
star
4

CRAFT-pytorch

Official implementation of Character Region Awareness for Text Detection (CRAFT)
Python
3,024
star
5

CutMix-PyTorch

Official Pytorch implementation of CutMix regularizer
Python
1,211
star
6

voxceleb_trainer

In defence of metric learning for speaker recognition
Python
1,012
star
7

WCT2

Software that can perform photorealistic style transfer without the need of any post-processing steps.
Python
867
star
8

synthtiger

Official Implementation of SynthTIGER (Synthetic Text Image Generator), ICDAR 2021
Python
457
star
9

rexnet

Official Pytorch implementation of ReXNet (Rank eXpansion Network) with pretrained models
Python
451
star
10

AdamP

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)
Python
411
star
11

overhaul-distillation

Official PyTorch implementation of "A Comprehensive Overhaul of Feature Distillation" (ICCV 2019)
Python
409
star
12

cord

CORD: A Consolidated Receipt Dataset for Post-OCR Parsing
384
star
13

cutblur

Rethinking Data Augmentation for Image Super-resolution (CVPR 2020)
Jupyter Notebook
379
star
14

wsolevaluation

Evaluating Weakly Supervised Object Localization Methods Right (CVPR 2020)
Python
331
star
15

assembled-cnn

Tensorflow implementation of "Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network"
Python
329
star
16

generative-evaluation-prdc

Code base for the precision, recall, density, and coverage metrics for generative models. ICML 2020.
Python
239
star
17

ext_portrait_segmentation

Python
238
star
18

ClovaCall

ClovaCall dataset and Pytorch LAS baseline code (Interspeech 2020)
Python
218
star
19

fewshot-font-generation

The unified repository for few-shot font generation methods. This repository includes FUNIT (ICCV'19), DM-Font (ECCV'20), LF-Font (AAAI'21) and MX-Font (ICCV'21).
Python
199
star
20

EXTD_Pytorch

Official EXTD Pytorch code
Python
187
star
21

stargan-v2-tensorflow

StarGAN v2 - Official Tensorflow Implementation (CVPR 2020)
Python
187
star
22

CLEval

CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks
Python
184
star
23

TedEval

TedEval: A Fair Evaluation Metric for Scene Text Detectors
Python
176
star
24

rebias

Official Pytorch implementation of ReBias (Learning De-biased Representations with Biased Representations), ICML 2020
Python
168
star
25

SATRN

Official Tensorflow Implementation of SATRN (CVPR Workshop WTDDLE 2020)
Python
162
star
26

bros

Python
156
star
27

lffont

Official PyTorch implementation of LF-Font (Few-shot Font Generation with Localized Style Representations and Factorization) AAAI 2021
Python
153
star
28

aasist

Official PyTorch implementation of "AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks"
Python
151
star
29

som-dst

SOM-DST: Efficient Dialogue State Tracking by Selectively Overwriting Memory (ACL 2020)
Python
150
star
30

mxfont

Official PyTorch implementation of MX-Font (Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts) ICCV 2021
Python
147
star
31

dmfont

Official PyTorch implementation of DM-Font (ECCV 2020)
Python
133
star
32

rainbow-memory

Official pytorch implementation of Rainbow Memory (CVPR 2021)
Python
117
star
33

FocusSeq2Seq

[EMNLP 2019] Mixture Content Selection for Diverse Sequence Generation (Question Generation / Abstractive Summarization)
Python
113
star
34

attention-feature-distillation

Official implementation for (Show, Attend and Distill: Knowledge Distillation via Attention-based Feature Matching, AAAI-2021)
Python
111
star
35

frostnet

FrostNet: Towards Quantization-Aware Network Architecture Search
Python
106
star
36

webvicob

Official Implementation of Web-based Visual Corpus Builder (Webvicob), ICDAR 2023
Python
101
star
37

length-adaptive-transformer

Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)
Python
99
star
38

spade

Python
81
star
39

embedding-expansion

Official MXNet implementation of "Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning" (CVPR 2020)
Python
76
star
40

symmetrical-synthesis

Official Tensorflow implementation of "Symmetrical Synthesis for Deep Metric Learning" (AAAI 2020)
Python
71
star
41

units

Python
70
star
42

lookwhostalking

Look Who’s Talking: Active Speaker Detection in the Wild
Python
70
star
43

subword-qac

Subword Language Model for Query Auto-Completion
Python
67
star
44

ssmix

Official PyTorch Implementation of SSMix (Findings of ACL 2021)
Python
60
star
45

SSUL

[NeurIPS 2021] SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning
Python
59
star
46

BESTIE

[CVPR 2022] Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement
Python
55
star
47

PointWSSIS

[CVPR2023] The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation
Python
55
star
48

c3_sinet

Python
52
star
49

EResFD

Lightweight Face Detector from CLOVA
Python
44
star
50

puridiver

Official PyTorch Implementation of PuriDivER CVPR 2022.
Python
44
star
51

minimal-rnr-qa

[NAACL 2021] Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
Python
36
star
52

ECLIPSE

(CVPR 2024) ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning
Python
34
star
53

group-transformer

Official code for Group-Transformer (Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model, COLING-2020).
Python
25
star
54

ProxyDet

Official implementation of the paper "ProxyDet: Synthesizing Proxy Novel Classes via Classwise Mixup for Open-Vocabulary Object Detection"
Python
22
star
55

GeNAS

Official pytorch implementation for GeNAS: Neural Architecture Search with Better Generalization
Python
15
star
56

meev

Python
12
star
57

pkm-transformers

Official implementation of PKM-augmented language models (Findings of EMNLP 2020)
9
star
58

DCutMix

DCutMix official repo
Python
8
star
59

TVQ-VAE

Official pytorch implementation for TVQ-VAE
Jupyter Notebook
8
star
60

textual-kd-slu

Official Implementation of Textual KD SLU (ICASSP 2021)
Python
6
star
61

vat-d

Official Implementation of VAT-D
Python
5
star
62

ActiveASR_AugCR

Repositoty for Efficient Active Learning for Automatic Speech Recognition via Augmented Consistency Regularization
3
star
63

WSSS-BED

Rethinking Saliency-Guided Weakly-Supervised Semantic Segmentation
Python
1
star