gautierdag/bpeasy gautierdag/bpeasy

  • Stars
    star
    141
  • Rank 259,971 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created about 1 year ago
  • Updated about 1 month ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
gautierdag/bpeasy
github

gautierdag

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Fast bare-bones BPE for modern tokenizer training

More Repositories

1

pytorch-attentive-lm

Pytorch Implementation of an A-RNN-LM
Python
13
star
2

tokenizer-bench

Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"
Python
10
star
3

cultural-evolution-engine

This is a first take at a task and model agnostic CEE
Jupyter Notebook
4
star
4

pcfg-attention

Using an Attention Seq-to-Seq model on pcfg
Python
3
star
5

pytorch-research-template

Opinionated Research Template
Python
3
star
6

batche

Batch cache decorator
Python
2
star
7

cast

Python
2
star
8

riiid

Repo for RIIID Kaggle competition
Jupyter Notebook
2
star
9

rsa-a

A deeper dive into Representational Similarity Analysis in Referential Games
Jupyter Notebook
2
star
10

learnability-communication-tradeoff

learnability-communication-tradeoff
Jupyter Notebook
2
star
11

plancraft

Plancraft is a minecraft environment and agent suite to test optimal planning capabilities in LLMs
Jupyter Notebook
2
star
12

pythondatastruc

Python Data Structures - simple implementations
Python
1
star
13

serverless-plugin-ts-validation

Serverless plugin to generate validation JSON schemas from Typescript interfaces
TypeScript
1
star
14

awesome-constrained-decoding

List of open-source constrained decoding projects
1
star