Natural Language Object Retrieval

This repository contains the code for the following paper:

• R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, T. Darrell, Natural Language Object Retrieval, in Computer Vision and Pattern Recognition (CVPR), 2016 (PDF)

@article{hu2016natural,
  title={Natural Language Object Retrieval},
  author={Hu, Ronghang and Xu, Huazhe and Rohrbach, Marcus and Feng, Jiashi and Saenko, Kate and Darrell, Trevor},
  journal={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2016}
}

Project Page: http://ronghanghu.com/text_obj_retrieval

Installation

1. Download this repository or clone with Git, and then cd into the root directory of the repository.
2. Run ./external/download_caffe.sh to download the SCRC Caffe version for this experiment. It will be downloaded and unzipped into external/caffe-natural-language-object-retrieval. This version is modified from the Caffe LRCN implementation.
3. Build the SCRC Caffe version in external/caffe-natural-language-object-retrieval, following the Caffe installation instruction. Remember to also build pycaffe.

SCRC demo

1. Download the pretrained models with ./models/download_trained_models.sh.
2. Run the SCRC demo in ./demo/retrieval_demo.ipynb with Jupyter Notebook (IPython Notebook).

Train and evaluate SCRC model on ReferIt Dataset

1. Download the ReferIt dataset: ./datasets/download_referit_dataset.sh.
2. Download pre-extracted EdgeBox proposals: ./data/download_edgebox_proposals.sh.
3. You may need to add the SRCR root directory to Python's module path: export PYTHONPATH=.:$PYTHONPATH.
4. Preprocess the ReferIt dataset to generate metadata needed for training and evaluation: python ./exp-referit/preprocess_dataset.py.
5. Cache the scene-level contextual features to disk: python ./exp-referit/cache_referit_context_features.py.
6. Build training image lists and HDF5 batches: python ./exp-referit/cache_referit_training_batches.py.
7. Initialize the model parameters and train with SGD: python ./exp-referit/initialize_weights_scrc_full.py && ./exp-referit/train_scrc_full_on_referit.sh.
8. Evaluate the trained model: python ./exp-referit/test_scrc_on_referit.py.

Optionally, you may also train a SCRC version without contextual feature, using python ./exp-referit/initialize_weights_scrc_no_context.py && ./exp-referit/train_scrc_no_context_on_referit.sh.

Train and evaluate SCRC model on Kitchen Dataset

1. Download the Kitchen dataset: ./datasets/download_kitchen_dataset.sh.
2. You may need to add the SRCR root directory to Python's module path: export PYTHONPATH=.:$PYTHONPATH.
3. Build training image lists and HDF5 batches: python exp-kitchen/cache_kitchen_training_batches.py.
4. Train with SGD: ./exp-kitchen/train_scrc_kitchen.sh.
5. Evaluate the trained model: python exp-kitchen/test_scrc_on_kitchen.py.