Applied Deep Learning Resources

A collection of research articles, blog posts, slides and code snippets about deep learning in applied settings. Including trained models and simple methods that can be used out of the box. Mainly focusing on Convolutional Neural Networks (CNN) but Recurrent Neural Networks (RNN), deep Q-Networks (DQN) and other interesting architectures will also be listed.

CNN

Latest overview of the CNNs can be found from the paper "Deep learning for visual understanding: A review" [link, [PDF](articles/Deep learning for visual understanding A review.pdf)]

Another decent overview in Nature by LeCun, Bengio and Hinton: "Deep learning" [link, PDF]

ImageNet

ImageNet is the most important image classification and localization competition. Other data sets with results can be found from here: "Discover the current state of the art in objects classification." [link].

Prediction error of the ImageNet competition has been decreasing rapidly over the last 5 years:

Main network architectures on ImageNet

AlexNet

Original paper: "ImageNet Classification with Deep Convolutional Neural Networks" [PDF]

Properties: 8 weight layers (5 convolutional and 2 fully connected), 60 million parameters, Rectified Linear Units (ReLUs), Local Response Normalization, Dropout

VGG

Original paper: "Very Deep Convolutional Networks for Large-Scale Image Recognition" [arxiv]

Properties: 19 weight layers, 144m parameters, 3x3 convolution filters, L2 regularised, Dropout, No Local Response Normalization

GoogLeNet

Original paper: "Going deeper with convolutions" [arxiv]

Lates upgrade to the model achieves even better scores with models and import to Torch: "Rethinking the Inception Architecture for Computer Vision" [arxiv], "Torch port of Inception V3" [github]

Properties: 22 layers, 7m parameters, Inception modules, 1x1 conv layers, ReLUs, Dropout, Mid-level outputs

Inception modules:

ResNet

Original paper: "Deep Residual Learning for Image Recognition" [arxiv]

Very nice slides: "Deep Residual Learning" [PDF]

Github: [github]

Properties: 152 layers, ReLUs, Batch Normalization (See "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" [arxiv]), less hacks (no dropout), more stable (different number of layers work as well) and lower complexity than VGG.

Main building block of the network:

Features are also very good and transferable with (faster) R-CNNs (see below):

DenseNet

Original paper: "Densely Connected Convolutional Networks" [arxiv, github]

Properties: similar performance to ResNet on ImageNet dataset but two times smaller model complexity with even easier training.

A dense block with 5 layers and growth rate 4:

A deep DenseNet with three dense blocks:

Feature learning and object detection

• "CNN Features off-the-shelf: an Astounding Baseline for Recognition" [arxiv]

• First paper about R-CNN: "Rich feature hierarchies for accurate object detection and semantic segmentation" [PDF, slides]

• "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks" [arxiv, github, Slides]

• Faster and better features: "Efficient Deep Feature Learning and Extraction via StochasticNets" [arxiv]

Other fun and visual things

• Code and models for automatic captions of images: "Deep Visual-Semantic Alignments for Generating Image Descriptions" [web poster, PDF, github]

Also, a better version now open source by Google: "Show and Tell: image captioning open sourced in TensorFlow" [link]

• Google Deep Dream or neural networks on LSD: "Inceptionism: Going Deeper into Neural Networks" [link, deepdreamer.io/]

Deep dreaming from noise:

• Code and models for mixing arbitrary content and art style: "A Neural Algorithm of Artistic Style" [arxiv, deepart.io, a blog post, github.com/jcjohnson/neural-style]

• General image to image conversion (pix2pix): "Image-to-Image Translation with Conditional Adversarial Nets" [arxiv, github]

• "Automatic Colorization" and it includes a pre-trained model [Link]

• "Learning visual similarity for product design with convolutional neural networks" [PDF]

• Using images and image descriptions to improve search results: "Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank" [arxiv]

• "How Google Translate squeezes deep learning onto a phone" [post]

• "What a Deep Neural Network thinks about your #selfie" [blog]

Top selfies according to the ConvNet:

• "Recommending music on Spotify with deep learning" [github]

• "Lip Reading Sentences in the Wild" [arxiv]

• "An Empirical Evaluation of Deep Learning on Highway Driving" [arxiv]

• "Object Detectors Emerge in Deep Scene CNNs" [arxiv]

• "DeepStereo: Learning to Predict New Views from the World's Imagery" [arxiv]

• Classifying street signs: "The power of Spatial Transformer Networks" [blog] with "Spatial Transformer Networks" [arxiv]

• "Pedestrian Detection with RCNN" [PDF]

• Deep Learning for 3D shapes: "3D ShapeNets: A Deep Representation for Volumetric Shapes" [PDF]

• Code and a model for faces: "Free and open source face recognition with deep neural networks." [github]

• Fast neural networks which can perform arbitrary filters for images: "Deep Edge-Aware Filters" [PDF]

DQN

• Original paper: "Playing Atari with Deep Reinforcement Learning" [arxiv]

• My popular science article about DQN: "Artificial General Intelligence that plays Atari video games: How did DeepMind do it?" [link]

• DQN for RoboCup: "Deep Reinforcement Learning in Parameterized Action Space" [arxiv]

RNN

• Original paper of the best RNN architecture: "Long short-term memory" [PDF]

• Very good tutorial-like introduction to RNNs by Andrej Karpathy: "The Unreasonable Effectiveness of Recurrent Neural Networks" [link]

• "Visualizing and Understanding Recurrent Networks" [arxiv]

• "Composing Music With Recurrent Neural Networks" [blog]

Other random architectures

• Lot's of different models in Caffe's "Model Zoo" [github]

• HTMs by Jeff Hawkins: *"Continuous online sequence learning with an unsupervised neural network model"*​ [arxiv]

• Word2vec: "Efficient Estimation of Word Representations in Vector Space" [arxiv, Google code]

• "Feedforward Sequential Memory Networks: A New Structure to Learn Long-term Dependency" [arxiv]

Framework benchmarks

• "Comparative Study of Caffe, Neon, Theano and Torch for deep learning" [arxiv]

Their summary: From our experiments, we observe that Theano and Torch are the most easily extensible frameworks. We observe that Torch is best suited for any deep architecture on CPU, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPU for training and deployment of LSTM networks. Finally Caffe is the easiest for evaluating the performance of standard deep architectures.

• Very good qualitative analysis: zer0n/deepframeworks: [github]

• Just performance comparison: soumith/convnet-benchmarks: [github]

• "Deep Learning Libraries by Language" [link]

Other resources

• "An awesome list of (large-scale) public datasets on the Internet. (On-going collection)" [github]

• "Model Zoo" [github]

• Videos from "Deep Learning Summer School, Montreal 2015": http://videolectures.net/deeplearning2015_montreal/

• http://deeplearning.net/

Credits

Most of the snippets have come to my attention via internal mailing lists of Computational Neuroscience Lab at University of Tartu and London-based visual search company Dream It Get It. I am also reading a weekly newsletter by Data Elixir and checking research papers of the two main deep learning conferences: ICML and NIPS.