shjwudp/c4-dataset-script shjwudp/c4-dataset-script

  • Stars
    star
    114
  • Rank 306,280 (Top 7 %)
  • Language
    Python
  • License
    MIT License
  • Created over 2 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
shjwudp/c4-dataset-script
github

shjwudp

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

More Repositories

1

shu

中文书籍收录整理, Collection of Chinese Books
Python
161
star
2

mamba-jax

Python
5
star
3

ACM-ICPC-api-service

Go
2
star
4

megabyte

A PyTorch implementation of MEGABYTE. This multi-scale transformer architecture has the excellent features of tokenization-free and sub-quadratic attention. The paper link: https://arxiv.org/abs/2305.07185
Python
2
star
5

ACM-ICPC-frontend

TypeScript
1
star
6

ocr_game

Python
1
star
7

blueprint-trainer

Scaffolding for sequence model training research.
Python
1
star