There are no reviews yet. Be the first to send feedback to the community and the maintainers!
Repository Details
Byte-Pair Encoding tokenizer for training large language models on huge datasets. I don't know C, so most of the code comes from AI :D I hope to learn by rewriting it and making changes, fixes etc
