EdinburghNLP/nematus

NEMATUS

Attention-based encoder-decoder model for neural machine translation built in Tensorflow.

Notable features include:

• support for RNN and Transformer architectures

• support for advanced RNN architectures:

  • arbitrary input features (factored neural machine translation) http://www.statmt.org/wmt16/pdf/W16-2209.pdf
  • deep models (Miceli Barone et al., 2017) https://arxiv.org/abs/1707.07631
  • dropout on all layers (Gal, 2015) http://arxiv.org/abs/1512.05287
  • tied embeddings (Press and Wolf, 2016) https://arxiv.org/abs/1608.05859
  • layer normalisation (Ba et al, 2016) https://arxiv.org/abs/1607.06450
  • mixture of softmaxes (Yang et al., 2017) https://arxiv.org/abs/1711.03953
  • lexical model (Nguyen and Chiang, 2018) https://www.aclweb.org/anthology/N18-1031

• support for advanced Transformer architectures:

  • arbitrary input features (factored neural machine translation) http://www.statmt.org/wmt16/pdf/W16-2209.pdf
  • DropHead: dropout of entire attention heads (Zhou et al., 2020) https://arxiv.org/abs/2004.13342

• training features:

  • multi-GPU support documentation
  • label smoothing
  • early stopping with user-defined stopping criterion
  • resume training (optionally with MAP-L2 regularization towards original model)
  • minimum risk training (MRT)

• scoring and decoding features:

  • batch decoding
  • n-best output
  • scripts for scoring (given parallel corpus) and rescoring (of n-best output)
  • server mode

• other usability features:

  • command line interface for training, scoring, and decoding
  • JSON-formatted storage of model hyperparameters, vocabulary files and training progress
  • pretrained models for 13 translation directions (many top-performing at WMT shared task of respective year):
    • http://data.statmt.org/rsennrich/wmt16_systems/
    • http://data.statmt.org/wmt17_systems/
  • backward compatibility: continue using publicly released models with current codebase (scripts to convert from Theano to Tensorflow-style models are provided)

SUPPORT

For general support requests, there is a Google Groups mailing list at https://groups.google.com/d/forum/nematus-support . You can also send an e-mail to [email protected] .

INSTALLATION

Nematus requires the following packages:

• Python 3 (tested on version 3.5.2)
• TensorFlow 1.15 / 2.X (tested on version 2.0)

To install tensorflow, we recommend following the steps at: ( https://www.tensorflow.org/install/ )

the following packages are optional, but highly recommended

• CUDA >= 7 (only GPU training is sufficiently fast)
• cuDNN >= 4 (speeds up training substantially)

LEGACY THEANO VERSION

Nematus originated as a fork of dl4mt-tutorial by Kyunghyun Cho et al. ( https://github.com/nyu-dl/dl4mt-tutorial ), and was implemented in Theano. See https://github.com/EdinburghNLP/nematus/tree/theano for this Theano-based version of Nematus.

To use models trained with Theano with the current Tensorflow codebase, use the script nematus/theano_tf_convert.py.

DOCKER USAGE

You can also create docker image by running following command, where you change suffix to either cpu or gpu:

docker build -t nematus-docker -f Dockerfile.suffix .

To run a CPU docker instance with the current working directory shared with the Docker container, execute:

docker run -v `pwd`:/playground -it nematus-docker

For GPU you need to have nvidia-docker installed and run:

nvidia-docker run -v `pwd`:/playground -it nematus-docker

TRAINING SPEED

Training speed depends heavily on having appropriate hardware (ideally a recent NVIDIA GPU), and having installed the appropriate software packages.

To test your setup, we provide some speed benchmarks with `test/test_train.sh', on an Intel Xeon CPU E5-2620 v4, with a Nvidia GeForce GTX Titan X (Pascal) and CUDA 9.0:

GPU, CuDNN 5.1, tensorflow 1.0.1:

CUDA_VISIBLE_DEVICES=0 ./test_train.sh

225.25 sentenses/s

USAGE INSTRUCTIONS

All of the scripts below can be run with --help flag to get usage information.

Sample commands with toy examples are available in the test directory; for training a full-scale RNN system, consider the training scripts at http://data.statmt.org/wmt17_systems/training/

An updated version of these scripts that uses the Transformer model can be found at https://github.com/EdinburghNLP/wmt17-transformer-scripts

nematus/train.py : use to train a new model

data sets; model loading and saving

network parameters (all model types)

network parameters (rnn-specific)

network parameters (transformer-specific)

training parameters

minimum risk training parameters (MRT)

validation parameters

display parameters

translate parameters

nematus/translate.py : use an existing model to translate a source text

nematus/score.py : use an existing model to score a parallel corpus

nematus/rescore.py : use an existing model to rescore an n-best list.

The n-best list is assumed to have the same format as Moses:

sentence-ID (starting from 0) ||| translation ||| scores

new scores will be appended to the end. rescore.py has the same arguments as score.py, with the exception of this additional parameter:

nematus/theano_tf_convert.py : convert an existing theano model to a tensorflow model

If you have a Theano model (model.npz) with network architecture features that are currently supported then you can convert it into a tensorflow model using nematus/theano_tf_convert.py.

PUBLICATIONS

if you use Nematus, please cite the following paper:

Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry and Maria Nadejde (2017): Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, pp. 65-68.

@InProceedings{sennrich-EtAl:2017:EACLDemo,
  author    = {Sennrich, Rico  and  Firat, Orhan  and  Cho, Kyunghyun  and  Birch, Alexandra  and  Haddow, Barry  and  Hitschler, Julian  and  Junczys-Dowmunt, Marcin  and  L\"{a}ubli, Samuel  and  Miceli Barone, Antonio Valerio  and  Mokry, Jozef  and  Nadejde, Maria},
  title     = {Nematus: a Toolkit for Neural Machine Translation},
  booktitle = {Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics},
  month     = {April},
  year      = {2017},
  address   = {Valencia, Spain},
  publisher = {Association for Computational Linguistics},
  pages     = {65--68},
  url       = {http://aclweb.org/anthology/E17-3017}
}

the code is based on the following models:

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio (2015): Neural Machine Translation by Jointly Learning to Align and Translate, Proceedings of the International Conference on Learning Representations (ICLR).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (2017): Attention is All You Need, Advances in Neural Information Processing Systems (NIPS).

please refer to the Nematus paper for a description of implementation differences to the RNN model.

ACKNOWLEDGMENTS

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements 645452 (QT21), 644333 (TraMOOC), 644402 (HimL) and 688139 (SUMMA).