Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions
This repository provides the official Python implementation of Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions (Paper: https://arxiv.org/abs/2003.01826).
Common up-convolution methods are inducing heavy spectral distortions into generated images. (Left) Statistics (mean and variance) after azimuthal integration over the power-spectrum of real and GAN generated images. (Right) Results of the same experiments as above, adding our proposed spectral loss during GAN training.
Spectral Regularization
Since common generative network architectures are mostly exclusively using image-space based loss functions, it is not possible to capture and correct spectral distortions directly. Hence, we extend existing GAN architectures in two ways:
- Spectral loss term
- Replace last last up-convolution unit.
Dependencies
Tested on Python 3.6.x.
- Pytorch (1.1.0)
- NumPy (1.16.2)
- Opencv (4.0.0)
- Matplotlib (3.1.1)
Downloading Dataset
Link to download CelebA dataset.
Training Netwroks
Training vanilla models
We train different GAN models using this repo. Then, we employ our Visualization script to analyse the frequency behaviour.
Training Spectral Regularization models
Make the following changes to incorporate the regularizer. Then, train the model.
- Substitue train.py for train_spectrum.py
- Substitue module.py for module_spectrum.py
Click here to go the Regularization implementation.
Deepfake detection
Experiments and code for the deepfake detection parts of the paper can be found in the repository of our prior Arxiv pre-print.
Citation
If this work is useful for your research, please cite our paper:
@misc{durall2020upconv,
title={Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions},
author={Ricard Durall and Margret Keuper and Janis Keuper},
year={2020},
eprint={2003.01826},
archivePrefix={arXiv},
primaryClass={cs.LG}
}