Deep Dream 💻 + 🌊💤 = ❤️
This repo contains a PyTorch implementation of the Deep Dream algorithm (🔗 blog by Mordvintstev et al.).
It's got a full support for the command line usage and a Jupyter Notebook!
And it will give you the power to create these weird, psychedelic-looking images:
Not bad, huh?
I strongly suggest you start with the Jupyter notebook that I've created!
Note: it's pretty large, ~10 MBs, so it may take a couple of attempts to load it in the browser here on GitHub.
Table of Contents
What is DeepDream algorithm?
In a nutshell the algorithm maximizes the activations of chosen network layers by doing a gradient ascent.
So from an input image like the one on the left after "dreaming" we get the image on the right:
Who would have said that neural networks had this creativity hidden inside? 🎨
Why yet another Deep Dream repo?
Most of the original Deep Dream repos were written in Caffe and the ones written in PyTorch are usually really hard to read and understand. This repo is an attempt of making the cleanest DeepDream repo that I'm aware of + it's written in PyTorch! ❤️
Static Image Examples
Here are some examples that you can create using this code!
Optimizing shallower layers = Amplify low-level features
By using shallower layers of neural networks you'll get lower level patterns (edges, circles, colors, etc.) as the output:
Here the first 2 images came from ResNet50 and the last one came from GoogLeNet (both pretrained on ImageNet).
Optimizing deeper Layers = Amplify high-level features
By using deeper network layers you'll get higher level patterns (eyes, snouts, animal heads):
The 1st and 3rd were created using VGG 16 (ImageNet) and the middle one using ResNet50 pretrained on Places 365.
Dataset matters (ImageNet vs Places 365)
If we keep every other parameter the same but we swap the pretrained weights we get these:
Left: ResNet50-ImageNet (we can see more animal features) Right: ResNet50-Places365 (human built stuff, etc.).
Impact of increasing the pyramid size
Dreaming is performed on multiple image resolutions stacked "vertically" (we call this an image pyramid).
Going from left to right the only parameter that changed was the pyramid size (from left to right: 3, 7, 9 levels).
Impact of increasing the pyramid ratio
Playing with pyramid ratio has a similar/related effect - the basic idea is that the relative area of the image which the deeper neurons can modify and "see" (the so-called receptive field of the net) is increasing and we get increasingly bigger features like eyes popping out (from left to right: 1.1, 1.5, 1.8):
Note: you can see the exact params used to create these images encoded into the filename!
Make sure to check out the Jupyter notebook!, I've explained this thoroughly.
Ouroboros Video Examples
Here are some further examples that you can create using this code!
The idea here is that whatever the network dreams just feed that back to it's input and apply a geometric transformation.
Ouroboros: Zoom transform
If we apply only central zoom we get this:
Ouroboros: Zoom and Rotation transforms
Applying central zoom and at the same time applying a 3 degree rotation per frame yields this:
Ouroboros: Translation
Finally if we do a simple translation (5 px per frame top left to bottom right direction):
Hopefully these did not break your brain - it feels like web 1.0 early 2000s. Bear with me.
DeepDream Video Examples
Instead of feeding the output back to input we just apply the algorithm per frame and apply some linear blending:
Linear blending just combines the current frame with the last one so as to reduce the flicker (here I used 0.85)
Note: all of the deepdream images/GIFs were produced by me, credits for original image artists are given bellow.
Setup
git clone https://github.com/gordicaleksa/pytorch-deepdream- Open Anaconda Prompt and navigate into project directory
cd path_to_repo - Run
conda env createfrom project directory (this will create a brand new conda environment). - Run
activate pytorch-deepdream(for running scripts from your console or setup the interpreter in your IDE)
That's it! It should work out-of-the-box executing environment.yml file which deals with dependencies.
Note: If you wish to use video functions I have - you'll need ffmpeg in your system path.
PyTorch pip package will come bundled with some version of CUDA/cuDNN with it, but it is highly recommended that you install a system-wide CUDA beforehand, mostly because of the GPU drivers. I also recommend using Miniconda installer as a way to get conda on your system. Follow through points 1 and 2 of this setup and use the most up-to-date versions of Miniconda and CUDA/cuDNN for your system.
Usage
Option 1: Jupyter Notebook
Just run jupyter notebook from you Anaconda console and it will open up a session in your default browser.
Open The Annotated DeepDream.ipynb and you're ready to play!
Note: if you get DLL load failed while importing win32api: The specified module could not be found
Just do pip uninstall pywin32 and then either pip install pywin32 or conda install pywin32 should fix it!
Option 2: Use your IDE of choice
You just need to link the Python environment you created in the setup section.
Option 3: Command line
Navigate to/activate your env if you're using Anaconda (and I hope you do) and you can use the commands I've linked below.
Tip: Place your images/videos inside the data/input/ directory and you can then just reference
your files (images/videos) by their name instead of using absolute/relative paths.
DeepDream images
To create some static Deep Dream images run the following command:
python deepdream.py --input <img_name> --img_width 600
This will use the default settings but you'll immediately get a meaningful result saved to:
data/out-images/VGG16_EXPERIMENTAL_IMAGENET/
Note: the output directory will change depending on the model and pretrained weights you use.
Ouroboros videos
To get the out-of-the-box Ouroboros 30-frame video do the following:
python deepdream.py --input <img_name> --create_ouroboros --ouroboros_length 30
It will dump the intermediate frames to data/out-videos/VGG16_EXPERIMENTAL_IMAGENET/ and it will save the final video to data/out-videos.
DeepDream videos
To create a Deep Dream video run this command:
python deepdream.py --input <mp4 video name>
It will dump the intermediate frames to data/out-videos/tmp_out and it will save the final video to data/out-videos.
Well, enjoy playing with this project! Here are some additional, beautiful, results:
Hardware requirements
A GPU with 2+ GBs will be more than enough.
You'll be able to create DeepDream images, Ouroboros and DeepDream videos.
If you don't have a GPU, the code will automatically run on the CPU but somewhat slower (especially for videos).
Learning material
If you're having difficulties understanding DeepDream I did an overview of the algorithm in this video:
And also the Jupyter Notebook I created is the best place to start!
Acknowledgements
I found these repos useful (while developing this one):
- deepdream (Caffe, original repo)
- DeepDreamAnim (Caffe)
- AI-Art (PyTorch)
- neural-dream (PyTorch)
- DeepDream (PyTorch)
I found the images I was using here:
Other images are now already classics in the NST and DeepDream worlds.
Places 365 pretrained models came from this awesome repo.
Citation
If you find this code useful for your research, please cite the following:
@misc{Gordić2020DeepDream,
author = {Gordić, Aleksa},
title = {pytorch-deepdream},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/gordicaleksa/pytorch-deepdream}},
}
Connect with me
If you'd love to have some more AI-related content in your life 🤓, consider:
- Subscribing to my YouTube channel The AI Epiphany 🔔
- Follow me on LinkedIn and Twitter 💡
- Follow me on Medium 📚 ❤️
