Seeing What a GAN Cannot Generate
State-of-the art GANs can create increasingly realistic images, yet they are not perfect.
What is a GAN unable to generate? This repository contains the code for the ICCV 2019 paper Seeing What a GAN Cannot Generate, which introduces a framework that can be used to answer this question.
 |
 |
|---|---|
| GAN reconstruction | Real photo |
Our goal is not to benchmark how far the generated distribution is from the target. Instead, we want to visualize and understand what is different between real and fake images.
Mode-dropping and the problem of visualizing omissions
We visualize the omissions of an image generator in two ways.
- We identify omissions within the distribution of images.
- We identify omissions within individual images.
Seeing omissions in the distribution
To understand omissions in a GAN's output distribution, we compare segmentation statistics between the GAN output and the training distribution.
A Progressive GAN trained to generate LSUN outdoor church images is analyzed below.
The model does not generate enough pixels of people, cars, fences,
palm trees, or signboards compared to the training distribution.
The script run_fsd.sh and the notebook seeing_distributions.ipynb
show how we collect and visualize these segmentation statistics.
Seeing omissions in individual images
To understand omission in specific GAN-generated output, we must pair the output with a real photo that shows what the GAN should have drawn but did not. So we compare real training photos to a reconstructed image derived from the model of the GAN.
These visualizations are created by run_invert.sh.
People
As seen in the distribution statistics, thie GAN does not draw enough people. By visualizing reconstructions, we can see how: the GAN seems to avoid drawing large person figures entirely, instead synthesizing plausible scenes without people.
| GAN reconstruction | Real photo |
|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
Vehicles
A similar effect is seen for vehicles.
| GAN reconstruction | Real photo |
|---|---|
 |
 |
 |
 |
 |
 |
Signs
| GAN reconstruction | Real photo |
|---|---|
 |
 |
 |
 |
 |
 |
Monuments
| GAN reconstruction | Real photo |
|---|---|
 |
 |
 |
 |
 |
 |
Palm trees
| GAN reconstruction | Real photo |
|---|---|
 |
 |