Review Network for Caption Generation
Image Captioning on MSCOCO
You can use the code in this repo to genearte a MSCOCO evaluation server submission with CIDEr=0.96+ with just a few hours.
No fine-tuning required. No fancy tricks. Just train three end-to-end review networks and do an ensemble.
- Feature extraction: 2 hours in parallel
- Single model training: 6 hours
- Ensemble model training: 30 mins
- Beam search for caption generation: 3 hours in parallel
Below is a comparison with other state-of-the-art systems (with according published papers) on the MSCOCO evaluation server:
| Model | BLEU-4 | METEOR | ROUGE-L | CIDEr | Fine-tuned | Task specific features |
|---|---|---|---|---|---|---|
| Attention | 0.537 | 0.322 | 0.654 | 0.893 | No | No |
| MS Research | 0.567 | 0.331 | 0.662 | 0.925 | No | Yes |
| Google NIC | 0.587 | 0.346 | 0.682 | 0.946 | Yes | No |
| Semantic Attention | 0.599 | 0.335 | 0.682 | 0.958 | No | Yes |
| Review Net | 0.597 | 0.347 | 0.686 | 0.969 | No | No |
In the diretcory image_caption_online, you can use the code therein to reproduce our evaluation server results.
In the directory image_caption_offline, you can rerun experiments in our paper using offline evaluation.
Code Captioning
Predicting comments for a piece of source code is another interesting task. In the repo we also release a dataset with train/dev/test splits, along with the code of a review network.
Check out the directory code_caption.
Below is a comparison with baselines on the code captioning dataset:
| Model | LLH | CS-1 | CS-2 | CS-3 | CS-4 | CS-5 |
|---|---|---|---|---|---|---|
| LSTM Language Model | -5.34 | 0.2340 | 0.2763 | 0.3000 | 0.3153 | 0.3290 |
| Encoder-Decoder | -5.25 | 0.2535 | 0.2976 | 0.3201 | 0.3367 | 0.3507 |
| Encoder-Decoder (Bidir) | -5.19 | 0.2632 | 0.3068 | 0.3290 | 0.3442 | 0.3570 |
| Attentive Encoder-Decoder (Bidir) | -5.14 | 0.2716 | 0.3152 | 0.3364 | 0.3523 | 0.3651 |
| Review Net | -5.06 | 0.2889 | 0.3361 | 0.3579 | 0.3731 | 0.3840 |
References
This repo contains the code and data used in the following paper:
Review Networks for Caption Generation
Zhilin Yang, Ye Yuan, Yuexin Wu, Ruslan Salakhutdinov, William W. Cohen
NIPS 2016