rust-bert

Rust-native state-of-the-art Natural Language Processing models and pipelines. Port of Hugging Face's Transformers library, using tch-rs or onnxruntime bindings and pre-processing from rust-tokenizers. Supports multi-threaded tokenization and GPU inference. This repository exposes the model base architecture, task-specific heads (see below) and ready-to-use pipelines. Benchmarks are available at the end of this document.

Get started with tasks including question answering, named entity recognition, translation, summarization, text generation, conversational agents and more in just a few lines of code:

    let qa_model = QuestionAnsweringModel::new(Default::default())?;
                                                        
    let question = String::from("Where does Amy live ?");
    let context = String::from("Amy lives in Amsterdam");

    let answers = qa_model.predict(&[QaInput { question, context }], 1, 32);

Output:

[Answer { score: 0.9976, start: 13, end: 21, answer: "Amsterdam" }]

The tasks currently supported include:

• Translation
• Summarization
• Multi-turn dialogue
• Zero-shot classification
• Sentiment Analysis
• Named Entity Recognition
• Part of Speech tagging
• Question-Answering
• Language Generation
• Masked Language Model
• Sentence Embeddings
• Keywords extraction

Getting started

This library relies on the tch crate for bindings to the C++ Libtorch API. The libtorch library is required can be downloaded either automatically or manually. The following provides a reference on how to set-up your environment to use these bindings, please refer to the tch for detailed information or support.

Furthermore, this library relies on a cache folder for downloading pre-trained models. This cache location defaults to ~/.cache/.rustbert, but can be changed by setting the RUSTBERT_CACHE environment variable. Note that the language models used by this library are in the order of the 100s of MBs to GBs.

Manual installation (recommended)

1. Download libtorch from https://pytorch.org/get-started/locally/. This package requires v2.0.0: if this version is no longer available on the "get started" page, the file should be accessible by modifying the target link, for example https://download.pytorch.org/libtorch/cu118/libtorch-cxx11-abi-shared-with-deps-2.0.0%2Bcu118.zip for a Linux version with CUDA11. NOTE: When using rust-bert as dependency from crates.io, please check the required LIBTORCH on the published package readme as it may differ from the version documented here (applying to the current repository version).
2. Extract the library to a location of your choice
3. Set the following environment variables

Linux:

export LIBTORCH=/path/to/libtorch
export LD_LIBRARY_PATH=${LIBTORCH}/lib:$LD_LIBRARY_PATH

Windows

$Env:LIBTORCH = "X:\path\to\libtorch"
$Env:Path += ";X:\path\to\libtorch\lib"

macOS + Homebrew

brew install pytorch jq
export LIBTORCH=$(brew --cellar pytorch)/$(brew info --json pytorch | jq -r '.[0].installed[0].version')
export LD_LIBRARY_PATH=${LIBTORCH}/lib:$LD_LIBRARY_PATH

Automatic installation

Alternatively, you can let the build script automatically download the libtorch library for you. The download-libtorch feature flag needs to be enabled. The CPU version of libtorch will be downloaded by default. To download a CUDA version, please set the environment variable TORCH_CUDA_VERSION to cu118. Note that the libtorch library is large (order of several GBs for the CUDA-enabled version) and the first build may therefore take several minutes to complete.

Verifying installation

Verify your installation (and linking with libtorch) by adding the rust-bert dependency to your Cargo.toml or by cloning the rust-bert source and running an example:

git clone [email protected]:guillaume-be/rust-bert.git
cd rust-bert
cargo run --example sentence_embeddings

ONNX Support (Optional)

ONNX support can be enabled via the optional onnx feature. This crate then leverages the ort crate with bindings to the onnxruntime C++ library. We refer the user to this page project for further installation instructions/support.

1. Enable the optional onnx feature. The rust-bert crate does not include any optional dependencies for ort, the end user should select the set of features that would be adequate for pulling the required onnxruntime C++ library.
2. The current recommended installation is to use dynamic linking by pointing to an existing library location. Use the load-dynamic cargo feature for ort.
3. set the ORT_DYLIB_PATH to point to the location of downloaded onnxruntime library (onnxruntime.dll/libonnxruntime.so/libonnxruntime.dylib depending on the operating system). These can be downloaded from the release page of the onnxruntime project

Most architectures (including encoders, decoders and encoder-decoders) are supported. the library aims at keeping compatibility with models exported using the optimum library. A detailed guide on how to export a Transformer model to ONNX using optimum is available at https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model The resources used to create ONNX models are similar to those based on Pytorch, replacing the pytorch by the ONNX model. Since ONNX models are less flexible than their Pytorch counterparts in the handling of optional arguments, exporting a decoder or encoder-decoder model to ONNX will usually result in multiple files. These files are expected (but not all are necessary) for use in this library as per the table below:

Note that the computational efficiency will drop when the decoder with past file is optional but not provided since the model will not used cached past keys and values for the attention mechanism, leading to a high number of redundant computations. The Optimum library offers export options to ensure such a decoder with past model file is created. he base encoder and decoder model architecture are available (and exposed for convenience) in the encoder and decoder modules, respectively.

Generation models (pure decoder or encoder/decoder architectures) are available in the models module. ost pipelines are available for ONNX model checkpoints, including sequence classification, zero-shot classification, token classification (including named entity recognition and part-of-speech tagging), question answering, text generation, summarization and translation. These models use the same configuration and tokenizer files as their Pytorch counterparts when used in a pipeline. Examples leveraging ONNX models are given in the ./examples directory

Ready-to-use pipelines

Based on Hugging Face's pipelines, ready to use end-to-end NLP pipelines are available as part of this crate. The following capabilities are currently available:

Disclaimer The contributors of this repository are not responsible for any generation from the 3rd party utilization of the pretrained systems proposed herein.

    let qa_model = QuestionAnsweringModel::new(Default::default())?;
                                                        
    let question = String::from("Where does Amy live ?");
    let context = String::from("Amy lives in Amsterdam");

    let answers = qa_model.predict(&[QaInput { question, context }], 1, 32);

[Answer { score: 0.9976, start: 13, end: 21, answer: "Amsterdam" }]

use rust_bert::pipelines::translation::{Language, TranslationModelBuilder};
fn main() -> anyhow::Result<()> {
let model = TranslationModelBuilder::new()
        .with_source_languages(vec![Language::English])
        .with_target_languages(vec![Language::Spanish, Language::French, Language::Italian])
        .create_model()?;
    let input_text = "This is a sentence to be translated";
    let output = model.translate(&[input_text], None, Language::Spanish)?;
    for sentence in output {
        println!("{}", sentence);
    }
    Ok(())
}

Il s'agit d'une phrase à traduire

    let summarization_model = SummarizationModel::new(Default::default())?;
                                                        
    let input = ["In findings published Tuesday in Cornell University's arXiv by a team of scientists \
from the University of Montreal and a separate report published Wednesday in Nature Astronomy by a team \
from University College London (UCL), the presence of water vapour was confirmed in the atmosphere of K2-18b, \
a planet circling a star in the constellation Leo. This is the first such discovery in a planet in its star's \
habitable zone — not too hot and not too cold for liquid water to exist. The Montreal team, led by Björn Benneke, \
used data from the NASA's Hubble telescope to assess changes in the light coming from K2-18b's star as the planet \
passed between it and Earth. They found that certain wavelengths of light, which are usually absorbed by water, \
weakened when the planet was in the way, indicating not only does K2-18b have an atmosphere, but the atmosphere \
contains water in vapour form. The team from UCL then analyzed the Montreal team's data using their own software \
and confirmed their conclusion. This was not the first time scientists have found signs of water on an exoplanet, \
but previous discoveries were made on planets with high temperatures or other pronounced differences from Earth. \
\"This is the first potentially habitable planet where the temperature is right and where we now know there is water,\" \
said UCL astronomer Angelos Tsiaras. \"It's the best candidate for habitability right now.\" \"It's a good sign\", \
said Ryan Cloutier of the Harvard–Smithsonian Center for Astrophysics, who was not one of either study's authors. \
\"Overall,\" he continued, \"the presence of water in its atmosphere certainly improves the prospect of K2-18b being \
a potentially habitable planet, but further observations will be required to say for sure. \"
K2-18b was first identified in 2015 by the Kepler space telescope. It is about 110 light-years from Earth and larger \
but less dense. Its star, a red dwarf, is cooler than the Sun, but the planet's orbit is much closer, such that a year \
on K2-18b lasts 33 Earth days. According to The Guardian, astronomers were optimistic that NASA's James Webb space \
telescope — scheduled for launch in 2021 — and the European Space Agency's 2028 ARIEL program, could reveal more \
about exoplanets like K2-18b."];

    let output = summarization_model.summarize(&input);

"Scientists have found water vapour on K2-18b, a planet 110 light-years from Earth. 
This is the first such discovery in a planet in its star's habitable zone. 
The planet is not too hot and not too cold for liquid water to exist."

use rust_bert::pipelines::conversation::{ConversationModel, ConversationManager};

let conversation_model = ConversationModel::new(Default::default());
let mut conversation_manager = ConversationManager::new();

let conversation_id = conversation_manager.create("Going to the movies tonight - any suggestions?");
let output = conversation_model.generate_responses(&mut conversation_manager);

"The Big Lebowski."

    let model = GPT2Generator::new(Default::default())?;
                                                        
    let input_context_1 = "The dog";
    let input_context_2 = "The cat was";

    let generate_options = GenerateOptions {
        max_length: 30,
        ..Default::default()
    };

    let output = model.generate(Some(&[input_context_1, input_context_2]), generate_options);

[
    "The dog's owners, however, did not want to be named. According to the lawsuit, the animal's owner, a 29-year"
    "The dog has always been part of the family. \"He was always going to be my dog and he was always looking out for me"
    "The dog has been able to stay in the home for more than three months now. \"It's a very good dog. She's"
    "The cat was discovered earlier this month in the home of a relative of the deceased. The cat\'s owner, who wished to remain anonymous,"
    "The cat was pulled from the street by two-year-old Jazmine.\"I didn't know what to do,\" she said"
    "The cat was attacked by two stray dogs and was taken to a hospital. Two other cats were also injured in the attack and are being treated."
]

    let sequence_classification_model = ZeroShotClassificationModel::new(Default::default())?;

    let input_sentence = "Who are you voting for in 2020?";
    let input_sequence_2 = "The prime minister has announced a stimulus package which was widely criticized by the opposition.";
    let candidate_labels = &["politics", "public health", "economics", "sports"];

    let output = sequence_classification_model.predict_multilabel(
        &[input_sentence, input_sequence_2],
        candidate_labels,
        None,
        128,
    );

[
  [ Label { "politics", score: 0.972 }, Label { "public health", score: 0.032 }, Label {"economics", score: 0.006 }, Label {"sports", score: 0.004 } ],
  [ Label { "politics", score: 0.975 }, Label { "public health", score: 0.0818 }, Label {"economics", score: 0.852 }, Label {"sports", score: 0.001 } ],
]

    let sentiment_classifier = SentimentModel::new(Default::default())?;
                                                        
    let input = [
        "Probably my all-time favorite movie, a story of selflessness, sacrifice and dedication to a noble cause, but it's not preachy or boring.",
        "This film tried to be too many things all at once: stinging political satire, Hollywood blockbuster, sappy romantic comedy, family values promo...",
        "If you like original gut wrenching laughter you will like this movie. If you are young or old then you will love this movie, hell even my mom liked it.",
    ];

    let output = sentiment_classifier.predict(&input);

[
    Sentiment { polarity: Positive, score: 0.9981985493795946 },
    Sentiment { polarity: Negative, score: 0.9927982091903687 },
    Sentiment { polarity: Positive, score: 0.9997248985164333 }
]

    let ner_model = NERModel::new(default::default())?;

    let input = [
        "My name is Amy. I live in Paris.",
        "Paris is a city in France."
    ];
    
    let output = ner_model.predict(&input);

[
  [
    Entity { word: "Amy", score: 0.9986, label: "I-PER" }
    Entity { word: "Paris", score: 0.9985, label: "I-LOC" }
  ],
  [
    Entity { word: "Paris", score: 0.9988, label: "I-LOC" }
    Entity { word: "France", score: 0.9993, label: "I-LOC" }
  ]
]

fn main() -> anyhow::Result<()> {
    let keyword_extraction_model = KeywordExtractionModel::new(Default::default())?;
    
    let input = "Rust is a multi-paradigm, general-purpose programming language. \
       Rust emphasizes performance, type safety, and concurrency. Rust enforces memory safety—that is, \
       that all references point to valid memory—without requiring the use of a garbage collector or \
       reference counting present in other memory-safe languages. To simultaneously enforce \
       memory safety and prevent concurrent data races, Rust's borrow checker tracks the object lifetime \
       and variable scope of all references in a program during compilation. Rust is popular for \
       systems programming but also offers high-level features including functional programming constructs.";

    let output = keyword_extraction_model.predict(&[input])?;
}

"rust" - 0.50910604
"programming" - 0.35731024
"concurrency" - 0.33825397
"concurrent" - 0.31229728
"program" - 0.29115444

    let pos_model = POSModel::new(default::default())?;

    let input = ["My name is Bob"];
    
    let output = pos_model.predict(&input);

[
    Entity { word: "My", score: 0.1560, label: "PRP" }
    Entity { word: "name", score: 0.6565, label: "NN" }
    Entity { word: "is", score: 0.3697, label: "VBZ" }
    Entity { word: "Bob", score: 0.7460, label: "NNP" }
]

    let model = SentenceEmbeddingsBuilder::remote(
            SentenceEmbeddingsModelType::AllMiniLmL12V2
        ).create_model()?;

    let sentences = [
        "this is an example sentence", 
        "each sentence is converted"
    ];
    
    let output = model.encode(&sentences)?;

[
    [-0.000202666, 0.08148022, 0.03136178, 0.002920636 ...],
    [0.064757116, 0.048519745, -0.01786038, -0.0479775 ...]
]

    let model = MaskedLanguageModel::new(Default::default())?;

    let sentences = [
        "Hello I am a <mask> student",
        "Paris is the <mask> of France. It is <mask> in Europe.",
    ];
    
    let output = model.predict(&sentences);

[
    [MaskedToken { text: "college", id: 2267, score: 8.091}],
    [
        MaskedToken { text: "capital", id: 3007, score: 16.7249}, 
        MaskedToken { text: "located", id: 2284, score: 9.0452}
    ]
]

Benchmarks

For simple pipelines (sequence classification, tokens classification, question answering) the performance between Python and Rust is expected to be comparable. This is because the most expensive part of these pipeline is the language model itself, sharing a common implementation in the Torch backend. The End-to-end NLP Pipelines in Rust provides a benchmarks section covering all pipelines.

For text generation tasks (summarization, translation, conversation, free text generation), significant benefits can be expected (up to 2 to 4 times faster processing depending on the input and application). The article Accelerating text generation with Rust focuses on these text generation applications and provides more details on the performance comparison to Python.

Loading pretrained and custom model weights

The base model and task-specific heads are also available for users looking to expose their own transformer based models. Examples on how to prepare the date using a native tokenizers Rust library are available in ./examples for BERT, DistilBERT, RoBERTa, GPT, GPT2 and BART. Note that when importing models from Pytorch, the convention for parameters naming needs to be aligned with the Rust schema. Loading of the pre-trained weights will fail if any of the model parameters weights cannot be found in the weight files. If this quality check is to be skipped, an alternative method load_partial can be invoked from the variables store.

Pretrained models are available on Hugging face's model hub and can be loaded using RemoteResources defined in this library. A conversion utility script is included in ./utils to convert Pytorch weights to a set of weights compatible with this library. This script requires Python and torch to be set-up, and can be used as follows: python ./utils/convert_model.py path/to/pytorch_model.bin where path/to/pytorch_model.bin is the location of the original Pytorch weights.

Citation

If you use rust-bert for your work, please cite End-to-end NLP Pipelines in Rust:

@inproceedings{becquin-2020-end,
    title = "End-to-end {NLP} Pipelines in Rust",
    author = "Becquin, Guillaume",
    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.nlposs-1.4",
    pages = "20--25",
}

Acknowledgements

Thank you to Hugging Face for hosting a set of weights compatible with this Rust library. The list of ready-to-use pretrained models is listed at https://huggingface.co/models?filter=rust.