NaturalCC

NaturalCC is a sequence modeling toolkit that allows researchers and developers to train custom models for many software engineering tasks, e.g., code summarization, code retrieval, code completion, code clone detection and type inference. Our vision is to bridge the gap between programming language and natural language through machine learning techniques.

🔖 News

• [May 10] We have merged the source code of "What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code" into NaturalCC.

⭐ Features

• A collection of code corpus with data preprocessing
• Performance benchmark
• Mixed precision training
  • Nvidia APEX
  • Automatic Mixed Precision
• Multi-GPU training
• Better logging output
• Various Implementations:
  • tensorflow gradient clipping
  • optimizers or learning schedulers
  • baseline models
  • binary data formats

🚀 Installation

Requirements

• PyTorch version >= 1.6.0
• Python version >= 3.6
• GCC/G++ > 5.0
• For training new models, you'll also need an NVIDIA GPU, NCCL and Cuda Toolkit installed.
• (optional) For faster training, you need to install NVIDIA's apex library.

1. Install prerequisite libraries

git clone https://github.com/CGCL-codes/naturalcc && cd naturalcc
pip install -r requirements.txt

Once you installed prerequisite libraries, you can check them via python -m env_test

2. Build or install NaturalCC

Export your NaturalCC cache directory (data and models will be saved in this directory) to user variables(~/.bashrc or ~/.zshrc in Linux, ~/.zsh_profile or ~/.bash_profile in macOS).

# Linux
echo "export NCC=<path_to_store ncc_data>" >> ~/.bashrc
# macOS
echo "export NCC=<path_to_store ncc_data>" >> ~/.bash_profile

Note: PyCharm cannot get environment variables and, therefore, we recommend you to register your NCC variable at ncc/__init__.py.

Compile Cython files to accelerate programs and register NaturalCC into your pip list

# compile for debug
# python setup.py build_ext --inplace
# install 
pip install --editable ./

3. Half precision computation (optional)

NaturalCC supports half precision training.

• If your Pytorch.__version__ < 1.6.0 and nvcc -V is runnable, please install apex.
• Otherwise, use Automatic Mixed Precision (AMP). Available Now (set amp: 1 in yaml file, An example).

4. Install GCC/G++ with conda (if you do not have permission)

Since NCC is build via Cython, your GCC/G++ version should be greater than 4.9. If you have the root permission, update GCC/G++; otherwise, install GCC/G++ with conda.

# install GCC/G++ with conda
conda install -c anaconda gxx_linux-64
conda install -c conda-forge gcc_linux-64
cd ~/anaconda/envs/XXX/bin
ln -s x86_64-conda_cos6-linux-gnu-gcc gcc
ln -s x86_64-conda_cos6-linux-gnu-g++ g++
# check
conda deactivate
conda activate XXX
>> type "gcc/g++ -v" in terminals

📚 Dataset

Currently, we have processed the following datasets:

• Python (Wan et al.)
• CodeSearchNet (Husain et al.)
• CodeXGlue (Feng et al.)
• Py150 (official processed) (raw)
• OpenCL (Grewe et al.)
• Java (Hu et, al.)
• Stack Overflow
• DeepCS (Gu et al.)
• AVATAR (Ahmad et al.)
• StackOverflow (Iyer et al.)

🤖 Implementations

Code retrieval (search)

• NBOW
• BiRNN
• 1D-CNN
• SelfAttn
• Deepcs

Code completion

• SeqRNN
• GPT2

Heterogeneous mapping

• static mapping
• decision tree
• deeptune
• inst2vec

Code summarization

• Naive Copy
• CodeNN
• DeepCom
• Seq2Seeq + Attention
• Nary-/ChildSum-Tree2Seq
• Code2Seq
• Transformer + (Sinusoidal/Relative/Learned Position Encoding)
• CodeBERT
• GraphCodeBERT
• PLBART

Structural Analysis of Pre-Trained Language Models for Source Code

• ICSE 2022

📋 Experiments

Code Summarization

Dataset: Python (Wan et al.)

Code Retrieval

Dataset: CodeSearchNet (Husain et al.)

Code Completion

Dataset: Py150 (official processed) (raw)

Type Inference

Dataset: CodeSearchNet-Java (Husain et al.)

Heterogeneous Mapping

Dataset: OpenCL (Grewe et al.)

🏫 Examples & Tutorials

All the running commands here should be executed in the root of project folder (the path of your naturalcc). For example, in my environment I will stay at /data/wanyao/Dropbox/ghproj-v100/naturalcc.

We also have more detailed READMEs to start your tutorial of NaturalCC.

Step 1: Download and process a dataset from datasets, and follow the instructions from the README.md file.

# ref: dataset/python_wan/README.md
# download dataset
bash dataset/python_wan/download.sh
# clean data
python -m dataset.python_wan.clean
# cast data attributes into different files
python -m dataset.python_wan.attributes_cast

# ref: dataset/python_wan/summarization/README.md
# save code tokens and docstirng tokens into MMAP format
python -m dataset.python_wan.summarization.preprocess

Step 2 (optional): Register your self-defined models

• If you want to create a new model, please add your model at ncc/models and ncc/modules.

• If your training policy are more complex than we thought, you should update your criterions and training procedure at ncc/criterions and ncc/trainers, respectively.
  

  Do not forget to update your self defined module at ncc/XX/__init__.py.

Step 3: Training and inference.

• Select a task and a model from task list and follow the instructions in its README.md to start your learning.

# ref: run/summarization/transformer/README.md
# train
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup python -m run.summarization.transformer.train -f config/python_wan/python > run/summarization/transformer/config/python_wan/python.log 2>&1 &
# inference
CUDA_VISIBLE_DEVICES=0 python -m run.summarization.transformer.eval -f config/python_wan/python -o run/summarization/transformer/config/python_wan/python.txt

❓ FAQ

Please fell free to contact me if you have any troubles.

😘 License and Acknowledgement

NaturalCC is MIT-licensed. The license applies to the pre-trained models as well. This project is also highly inspired by Fairseq and AllenNLP.

🔗 Related Links

Paper
NaturalCC-demo
About us: XCodeMind

❤️ Citation

Please cite as:

@inproceedings{wan2022naturalcc,
              author    = {Yao Wan and
                           Yang He and
                           Zhangqian Bi and
                           Jianguo Zhang and
                           Yulei Sui and
                           Hongyu Zhang and
                           Kazuma Hashimoto and
                           Hai Jin and
                           Guandong Xu and
                           Caiming Xiong and
                           Philip S. Yu},
              title     = {NaturalCC: An Open-Source Toolkit for Code Intelligence},
              booktitle   = {Proceedings of 44th International Conference on Software Engineering, Companion Volume},
              publisher = ACM,
              year      = {2022}
            }