OPEN NLLB effort 🦜

Roadmap

After OpenAI's ChatGPT was released the whole world got caught up in the AI frenzy (and for a good reason). While systems such as Meta's Llama and OpenAI's GPT-4 are incredibly powerful - they only focus on English language.

Similarly, for machine translation systems, Meta released a powerful NLLB ("no language left behind") MT system that supports 202 languages! - but they didn't release open-source checkpoints (they were released under non-commercial CC-NC-BY 4.0 license).

The main goal of this effort is to release truly open-source NLLB checkpoints that can be freely used even for commercial purposes.

The extended goal of this project is to scale up beyond the original 3.3B parameters dense transformers (7B+) and also support non-English LLMs.

We strongly believe that focusing on multilingual AI models can help democratize this technology, and help businesses & researchers around the world.

Getting started 🚀

Check out our getting started guide if you wish to contribute! Or just play around!

Community

👨‍👩‍👧‍👦 Join Aleksa Gordić - The AI Epiphany Discord server and the #open-nllb channel if you want to engage with the rest of the community!

📺 Daily YouTube video streams here! 👀

Language champions

Here is a list of data contributors who are owners of their respective languages (they are all native speakers). 🙏

Below is the original fairseq README:

No Language Left Behind

No Language Left Behind (NLLB) is a first-of-its-kind, AI breakthrough project that open-sources models capable of delivering high-quality translations directly between any pair of 200+ languages — including low-resource languages like Asturian, Luganda, Urdu and more. It aims to help people communicate with anyone, anywhere, regardless of their language preferences.

To enable the community to leverage and build on top of NLLB, we open source all our evaluation benchmarks(FLORES-200, NLLB-MD, Toxicity-200), LID models and training code, LASER3 encoders, data mining code, MMT training and inference code and our final NLLB-200 models and their smaller distilled versions, for easier use and adoption by the research community.

This code repository contains instructions to get the datasets, optimized training and inference code for MMT models, training code for LASER3 encoders as well as instructions for downloading and using the final large NLLB-200 model and the smaller distilled models. In addition to supporting more than 200x200 translation directions, we also provide reliable evaluations of our model on all possible translation directions on the FLORES-200 benchmark. By open-sourcing our code, models and evaluations, we hope to foster even more research in low-resource languages leading to further improvements in the quality of low-resource translation through contributions from the research community.

• Paper
• Website
• Blog
• NLLB Demo
• Request for Proposals : Translation Support for African Languages

Open Sourced Models and Community Integrations

Multilingual Translation Models

All models are licensed under CC-BY-NC 4.0 available in Model LICENSE file. We provide FLORES-200 evaluation results for all the models. For more details see the Modeling section README.

⭐ NEW ⭐ : We are releasing all the translations of NLLB-200 MoE model. Check Evaluation section README for more details.

Please use wget --trust-server-names <url> to download the provided links in proper file format.

LID Model

LID (Language IDentification) model to predict the language of the input text is available here under CC-BY-NC 4.0 license.

LASER3 Encoder Models

LASER3 models are available at LASER.

HuggingFace Integrations

Support for the dense models is available through the Hugging Face Hub under the NLLB tag. It is supported in the transformers library and the documentation for this model is available here.

Input and output languages are entirely customizable with BCP-47 codes used by the FLORES-200 dataset, here's an example usage with a translation from Romanian to German:

>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M", use_auth_token=True, src_lang="ron_Latn")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", use_auth_token=True)

>>> article = "Şeful ONU spune că nu există o soluţie militară în Siria"
>>> inputs = tokenizer(article, return_tensors="pt")

>>> translated_tokens = model.generate(
...     **inputs, forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"], max_length=30
... )
>>> tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]

Result:

UN-Chef sagt, es gibt keine militärische Lösung in Syrien

Installation

Follow installation instructions in INSTALL guide for running training/generation. For general instructions about fairseq and working with the codebase refer to fairseq README. For stopes and LASER follow their README files for installation.

Datasets

NLLB project uses data from three sources : public bitext, mined bitext and data generated using backtranslation. Details of different datasets used and open source links are provided in details here.

Primary Bitext

We provide a download script for public bitext data, and links to download NLLB-Seed data. For more details check here.

Mined Bitext

LASER3 teacher-student training code is open sourced here. LASER3 encoders and mined bitext metadata are open sourced in LASER repository. Global mining pipeline and monolingual data filtering pipelines are released and available in our stopes repository.

Backtranslated Bitext

Follow the instructions here to generate backtranslated data from a pretrained model.

Preparing Datasets for Training

We open source our dataset preparation pipeline for filtering/encoding/binarizing large scale datasets in stopes. Encoding the datasets are done using the new SPM-200 model which was trained on 200+ languages used in the NLLB project. For more details see link.

Training NLLB Models

We open source all our model training and generation code in this repo. We also share code for finetuning our models on different domains like NLLB-MD. Additionally, we also share the code for online distillation that produced our 1.3B and 600M distilled models. For more details check the Modeling section Readme.

Evaluation and Generation

NLLB project includes release of evaluation datasets like Flores-200, NLLB-MD and Toxicity-200. For instructions to run evaluation see instructions here and for instructions to produce generations from the models follow instructions here.

Flores200 | NLLB-MD | Toxicity-200

Human Evaluations - (XSTS)

(Added Jan - 2023) We open-source additional guidelines and training materials for conducting the human evaluation protocol we utilized (XSTS), as well the calibration utilized and the entire human translation evaluation set for NLLB-200 and it's published baseline here.

Citation

If you use NLLB in your work or any models/datasets/artifacts published in NLLB, please cite :

@article{nllb2022,
  title={No Language Left Behind: Scaling Human-Centered Machine Translation},
  author={{NLLB Team} and Costa-jussà, Marta R. and Cross, James and Çelebi, Onur and Elbayad, Maha and Heafield, Kenneth and Heffernan, Kevin and Kalbassi, Elahe and Lam, Janice and Licht, Daniel and Maillard, Jean and Sun, Anna and Wang, Skyler and Wenzek, Guillaume and Youngblood, Al and Akula, Bapi and Barrault, Loic and Mejia-Gonzalez, Gabriel and Hansanti, Prangthip and Hoffman, John and Jarrett, Semarley and Sadagopan, Kaushik Ram and Rowe, Dirk and Spruit, Shannon and Tran, Chau and Andrews, Pierre and Ayan, Necip Fazil and Bhosale, Shruti and Edunov, Sergey and Fan, Angela and Gao, Cynthia and Goswami, Vedanuj and Guzmán, Francisco and Koehn, Philipp and Mourachko, Alexandre and Ropers, Christophe and Saleem, Safiyyah and Schwenk, Holger and Wang, Jeff},
  year={2022}
}

License

NLLB code and fairseq(-py) is MIT-licensed available in LICENSE file.