A Neural Algorithm of Artistic Style (Keras Implementation)

An implementation of the arXiv preprint A Neural Algorithm of Artistic Style [1] & paper Image Style Transfer Using Convolutional Neural Networks [2].

Supports TensorFlow 2.4.1.

Style Transfer

style-transfer.ipynb describes the style transfer process between a white noise image x, a content image p, and a style representation a. Performing gradient descent of the content loss and style loss with respect to x impressions the content of p into x, bearing local styles, and colors from a.

Original Photograph Tubingen, Germany
Claude Monet Houses of Parliament
Pablo Picasso Seated Nude
Edvard Munch The Scream
Vincent van Gogh The Starry Night
William Turner The Shipwreck of The Minotaur
Wassily Kandinsky Composition VII

Content Reconstruction

content-reconstruction.ipynb describes the content reconstruction process from white noise. Performing gradient descent of the content loss on a white noise input x for a given content p yields a representation of the networks activation for a given layer l.

Layer	Result
block1_conv1
block2_conv1
block3_conv1
block4_conv1
block4_conv2
block5_conv1

Style Reconstruction

style-reconstruction.ipynb describes the style reconstruction process on Wassily Kandinsky's Composition VII from white noise. Performing gradient descent of the style loss on a white noise input x for a given artwork a yields a representation of the networks activation for a given set of layers L.

Layer	Result
block1_conv1
block1_conv1, block2_conv1
block1_conv1, block2_conv1, block3_conv1
block1_conv1, block2_conv1, block3_conv1, block4_conv1
block1_conv1, block2_conv1, block3_conv1, block4_conv1, block5_conv1

Content Layer

content-layer.ipynb visualizes how the style transfer is affected by using different layers for content loss.

Layer	Result
block1_conv1
block2_conv1
block3_conv1
block4_conv1
block5_conv1

Style Layers

style-layers.ipynb visualizes how the style transfer is affected by using different sets of layers for style loss.

Layers	Result
block1_conv1
block1_conv1, block2_conv1
block1_conv1, block2_conv1, block3_conv1
block1_conv1, block2_conv1, block3_conv1, block4_conv1
block1_conv1, block2_conv1, block3_conv1, block4_conv1, block5_conv1

Optimizers

optimizers.ipynb employs gradient descent, adam, and L-BFGS to understand the effect of different black-box optimizers. Gatys et. al use L-BFGS, but Adam appears to produce comparable results without as much overhead.

Gradient Descent	Adam	L-BFGS

TV Loss

tv-loss.ipynb introduces total-variation loss to reduce impulse noise in the images.

TV Loss Scale Factor	Result
0
1
10
100
1000

Photo-Realistic Style Transfer

photo-realistic-style-transfer.ipynb describes the photo-realistic style transfer process. Opposed to transferring style from an artwork, this notebook explores transferring a nighttime style from a picture of Piedmont Park at night to a daytime picture of Piedmont Park.

Content	Style	Result

References

[1] L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015.

[2] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, pages 2414–2423. IEEE, 2016.