StyleGAN2: projecting images

The goal of this Google Colab notebook is to project images to latent space with StyleGAN2.

Usage

To discover how to project a real image using the original StyleGAN2 implementation, run:

• stylegan2_projecting_images.ipynb

To process the projection of a batch of images, using either W(1,*) (original) or W(18,*) (extended), run:

• stylegan2_projecting_images_with_my_fork.ipynb

To edit latent vectors of projected images, run:

• stylegan2_editing_latent_vectors.ipynb

For more information about W(1,*) and W(18,*), please refer to the the original paper (section 5 on page 7):

Inverting the synthesis network $g$ is an interesting problem that has many applications. Manipulating a given image in the latent feature space requires finding a matching latent code $w$ for it first.

The following is about W(18,*):

Previous research suggests that instead of finding a common latent code $w$, the results improve if a separate $w$ is chosen for each layer of the generator. The same approach was used in an early encoder implementation.

The following is about W(1,*), which is the approach used in the original implementation:

While extending the latent space in this fashion finds a closer match to a given image, it also enables projecting arbitrary images that should have no latent representation. Instead, we concentrate on finding latent codes in the original, unextended latent space, as these correspond to images that the generator could have produced.

Data

Data consists of:

• 1 picture of the French president Emmanuel Macron, found on Nice Matin (archive),
• 37 individual pictures of the French government, found on Wikipedia (list),
• 5 pictures of famous paintings, found on Wikipedia (list):
  • La Joconde,
  • Le Condottière,
  • La Naissance de Vénus,
  • Ginevra de' Benci,
  • La Jeune Fille à la perle.

Pre-processing

There are two possible pre-processing methods:

• either center-cropping (to 1024x1024 resolution) as sole pre-processing,

• or the same pre-processing as for the FFHQ dataset:
  1. first, an alignment based on 68 face landmarks returned by dlib,
  2. then reproduce recreate_aligned_images(), as detailed in FFHQ pre-processing code.

Finally, the pre-processed image can be projected to the latent space of the StyleGAN2 model trained with configuration f on the Flickr-Faces-HQ (FFHQ) dataset.

Results: influence of pre-processing

NB: results are different if the code is run twice, even if the same pre-processing is used.

With center-cropping as sole pre-processing

The result below is obtained with center-cropping as sole pre-processing, hence some issues with the projection.

From left to right: the target image, the result obtained at the start of the projection, and the final result of the projection.

From left to right: the target image, the result obtained at the start of the projection, intermediate results, and the final result.

The background, the hair, the ears, and the suit are relatively well reproduced, but the face is wrong, especially the neck (in the original image) is confused with the chin (in the projected images). It is possible that the face is too small relatively to the rest of the image, compared to the FFHQ training dataset, hence the poor results of the projection.

With the same pre-processing as for the FFHQ dataset

The result below is obtained with the same pre-processing as for the FFHQ dataset, which allows to avoid the projection issues mentioned above.

From left to right: the target image, the result obtained at the start of the projection, and the final result of the projection.

From left to right: the target image, the result obtained at the start of the projection, intermediate results, and the final result.

Results: comparison with the extended projection

For the rest of the repository, the same-preprocessing as for the FFHQ dataset is used.

Shared data on Google Drive

Additional projection results are shown on the Wiki.

To make it easier to download them, they are also shared on Google Drive.

The directory structure is as follows:

stylegan2_projections/
├ aligned_images/
├ └ emmanuel-macron_01.png    # FFHQ-aligned image
├ generated_images_no_tiled/  # projections with `W(18,*)`
├ ├ emmanuel-macron_01.npy    # - latent code
├ └ emmanuel-macron_01.png    # - projected image
├ generated_images_tiled/     # projections with `W(1,*)`
├ ├ emmanuel-macron_01.npy    # - latent code
├ └ emmanuel-macron_01.png    # - projected image
├ aligned_images.tar.gz             # folder archive
├ generated_images_no_tiled.tar.gz  # folder archive
└ generated_images_tiled.tar.gz     # folder archive

Projection results

Images below allow us to compare results obtained with the original projection W(1,*) and the extended projection W(18,*).

A projected image obtained with W(18,*) is expected to be closer to the target image, at the expense of semantics.

If image fidelity is very important, W(18,*) can be run for a higher number of iterations (default is 1000 steps), but truncation might be needed for later applications.

French politicians

From top to bottom: aligned target image, projection with W(1,*), projection with W(18,*).

From top to bottom: aligned target image, projection with W(1,*), projection with W(18,*).

Art

From top to bottom: aligned target image, projection with W(1,*), projection with W(18,*).

Applications

In the following, we assume that real images have been projected, so that we have access to their latent codes, of shape (1, 512) or (18, 512) depending on the projection method.

There are three main applications:

1. morphing (linear interpolation),
2. style transfer (crossover),
3. expression transfer (adding a vector and a scaled difference vector).

Shared data on Google Drive

Results corresponding to each application are:

• shown on the Wiki,
• shared on Google Drive.

The directory structure is as follows:

stylegan2_editing/
├ expression/                   # expression transfer
| ├ no_tiled/                   # - `W(18,*)`
| | └ expression_01_age.jpg     # face n°1 ; age
| └ tiled/                      # - `W(1,*)`
|   └ expression_01_age.jpg
├ morphing/                     # morphing
| ├ no_tiled/                   # - `W(18,*)`
| | └ morphing_07_01.jpg        # face n°7 to face n°1
| └ tiled/                      # - `W(1,*)`
|   └ morphing_07_01.jpg
├ style_mixing/                 # style transfer
| ├ no_tiled/                   # - `W(18,*)`
| | └ style_mixing_42-07-10-29-41_42-07-22-39.jpg
| └ tiled/                      # - `W(1,*)`
|   └ style_mixing_42-07-10-29-41_42-07-22-39.jpg
├ video_style_mixing/           # style transfer
| ├ no_tiled/                   # - `W(18,*)`
| | └ video_style_mixing_000.000.jpg
| ├ tiled/                      # - `W(1,*)`
| | └ video_style_mixing_000.000.jpg
| ├ no_tiled_small.mp4          # with 2 reference faces
| ├ no_ tiled.mp4               # with 4 reference faces
| ├ tiled_small.mp4
| └ tiled.mp4
├ expression_transfer.tar.gz    # folder archive
├ morphing.tar.gz               # folder archive
├ style_mixing.tar.gz           # folder archive
└ video_style_mixing.tar.gz     # folder archive

1. Morphing

Morphing consists in a linear interpolation between two latent vectors (two faces).

Results are shown on the Wiki.

With the original projection W(1,*)

With the extended projection W(18,*)

2. Style transfer

Style transfer consists in a crossover of latent vectors at the layer level (cf. this piece of code).

There are 18 layers for the generator. The latent vector of the reference face is used for the first 7 layers. The latent vector of the face whose style has to be copied is used for the remaining 11 layers.

Results are shown on the Wiki.

With the original projection W(1,*)

Thanks to morphing of the faces whose style is copied, style transfer can be watched as a video.

With the extended projection W(18,*)

Thanks to morphing of the faces whose style is copied, style transfer can be watched as a video.

3. Expression transfer

Expression transfer consists in the addition of:

• a latent vector (a face),
• a scaled difference vector (an expression).

Expressions were defined, learnt, and shared on Github by a Chinese speaker:

1. age
2. angle_horizontal
3. angle_pitch
4. beauty
5. emotion_angry
6. emotion_disgust
7. emotion_easy
8. emotion_fear
9. emotion_happy
10. emotion_sad
11. emotion_surprise
12. eyes_open
13. face_shape
14. gender
15. glasses
16. height
17. race_black
18. race_white
19. race_yellow
20. smile
21. width

Results are shown on the Wiki.

With the original projection W(1,*)

• Age:

• Smile:

• Age:

• Smile:

With the extended projection W(18,*)

• Age:

• Smile:

• Age:

• Smile:

References

• StyleGAN2:
  • StyleGAN2 / StyleGAN2-ADA / StyleGAN2-ADA-PyTorch
  • Steam-StyleGAN2
  • My fork of StyleGAN2 to project a batch of images, using any projection (original or extended)
• Programming resources:
  • rolux/stylegan2encoder: align faces based on detected landmarks (FFHQ pre-processing)
  • Learnt latent directions tailored for StyleGAN2 (required for expression transfer)
  • Minimal example code for morphing and expression transfer
• Experimenting materials:
  • The website ArtBreeder by Joel Simon
  • Colab user interface for extended projection and expression transfer
  • A fast projector called encoder4editing and released in 2021
• Reading materials:
  • A blog post about editing projected images to add a cartoon effect
  • On the Wiki: GIF editing with MoviePy and Gifsicle
• Papers:
  • Karras, Tero, et al. A Style-Based Generator Architecture for Generative Adversarial Networks. CVPR 2019.
  • Abdal, Rameen, et al. Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?. ICCV 2019.
  • Karras, Tero, et al. Analyzing and Improving the Image Quality of StyleGAN. CVPR 2020.