Fast Multi-threaded VGG 19 Feature Extractor

Overview

This allows you to extract deep visual features from a pre-trained VGG-19 net for collections of images in the millions. Images are loaded and preprocessed in parallel using multiple CPU threads then shipped to the GPU in minibatches for the forward pass through the net. Model weights are downloaded for you and loaded using Torch's loadcaffe library, so you don't need to compile Caffe.

The feature extractor computes a 4096 dimensional feature vector for every image that contains the activations of the hidden layer immediately before the VGG's object classifier. The activations are ReLU-ed and L2-normalized, which means they can be used as generic off-the-shelf features for tasks like classification or image similarity.

Example usage

You point it a tab separated file of (image_id, path to image on disk) e.g.

12      /home/username/images/12.jpg
342     /home/username/images/342.jpg
169     /home/username/images/169.jpg

specified by the -data flag, and it creates a tab separated file of (image_id, json encoded VGG vector) e.g.

12      [4096 dimensional vector]
342     [4096 dimensional vector]
169     [4096 dimensional vector]

specified by the -outFile flag.

-nThreads tells it how many CPU loader threads to use. -batchSize tells it how many images to put in each minibatch. The higher the batchSize, the higher the throughput, so I'd make this as large as your GPU memory will allow.

Example:

th main.lua -data [tab separated file of (image_id, path_to_image_on_disk)] -outFile out_vecs -nThreads 8 -batchSize 128

Requirements

• Install torch on a machine with CUDA GPU
• If on Mac OSX, run brew install coreutils findutils to get GNU versions of wc, find, and cut
• loadcaffe

Why should I care about pre-trained deep convnet features in the first place?

• They're powerful and transferable: Razavian et. al. show that these kinds of deep features can be used off-the-shelf to beat highly tuned state-of-the-art methods on challenging fine grained classification problems. That is, you can use the same features that distinguish a boat from a motorcycle to accurately tell two species of birds apart, even when the differences between species are extremely subtle. They show superior results to traditional feature representations (like SIFT, HOG, visual bag of words).
• They're interpretable: Zeiler and Fergus shows that the learned representations are far from a black box. They're actually quite interpretable: lower layers of the network learn filters that fire when they see color blobs, edges, lines, corners.

Middle layers see combinations of these lower level features, forming filters that respond to common textures.

Higher layers see combinations of these middle layers, forming filters that respond to object parts, and so on.

source

You can see the actual content of the image becoming increasingly explicit along the processing hierarchy.

• They're cheap: You only need to do one forward pass on a pre-trained net to get them.
• They're the go-to visual component in some pretty incredible new machine vision applications: like automatically describing images from raw pixels.

Or being able to embed images and words in a joint space then do vector arithmetic in the learned space:

Yep that's a multimodal vector describing a blue car minus the multimodal vector for the word "blue", plus the vector for "red" resulting in a vector that is near images of red cars.

When should I use these features?

Take advice from here (actually go read the entire course, it's amazing).

Thanks to

Soumith Chintala for the scalable loader starter code, Andrej Karpathy's course for teaching me about all this stuff in the first place .