guillaume-be/SentencePiece-Rust-example

Stars
18
Rank 1,208,065 (Top 24 %)
Language
Rust
Created over 4 years ago
Updated over 4 years ago

guillaume-be/SentencePiece-Rust-example

guillaume-be

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Supporting example for "A Rust SentencePiece implementation"

rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

rust-tokenizers

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Financial-Causality-Extraction

Fincausal 2020 shared task

bpe-example

Supporting code for post on Rust implementation of Byte pair Encoding