This repository contains the processing scripts to scrape/process the code-pile dataset.

• Project Description
• How to use the Code-Pile (todo)
• How to Contribute
• Additional Resources

Check out The code pile proposal

The Code-Pile will be released similar to "the pile" as a folder of .jsonl.zst files, see lm-dataformat

It's not finished, ask on discord

Think about the most usefull Code-data for the next generation of textual Code Models.

The most valuable dataset properties (use your own judgment) are:

1. Open License
2. Data quality
3. Dataset size
4. Data variance/variety/nicheness
5. Ease of obtaining/processing

To add a new dataset, open a Issue under given dataset-request template. Gather all the related informations appropriate to it. Use the issue to track.

Check if there is existing Code or someone already working on it: See Additional Resources

1. Eleuthers Pile V1 Repos
2. Ask on Carper #code-pile
3. Ask on Eleuther
4. Consult the linked Spreadsheets below

Then implement it through the following steps:

1. Fork this repo
2. Use the working branch
3. Read the shared classes in datasets.py and codepile.py
4. Create mvp/example for your dataset
5. Create a pull request
6. Keep building the data-domain specific classes and repeat

Citation Placeholder:

@misc{Code-Pile,
  author = {},
  doi = {},
  month = {},
  title = {},
  url = {https://github.com/CarperAI/Code-Pile},
  version = {},
  year = {2022}
}

• Preliminary spreadsheet of useful resources

Closely related projects:

• The Pile V2 spreadsheet
• Pile V1 stackexchange-dataset repo
• Collins stackexchange gist

Previous work:

• [Codeparrot] (https://github.com/huggingface/transformers/tree/main/examples/research_projects/codeparrot)
• ...