• Stars
    star
    206
  • Rank 190,504 (Top 4 %)
  • Language
    Python
  • License
    BSD 3-Clause "New...
  • Created over 6 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Neural Code Comprehension: A Learnable Representation of Code Semantics

Neural Code Comprehension: A Learnable Representation of Code Semantics

ncc (Neural Code Comprehension) is a general Machine Learning technique to learn semantics from raw code in virtually any programming language. It relies on inst2vec, an embedding space and graph representation of LLVM IR statements and their context.

ncc_scheme

This repository contains the code used in [paper]:

Neural Code Comprehension: A Learnable Representation of Code Semantics, Tal Ben-Nun, Alice Shoshana Jakobovits, Torsten Hoefler

Please cite as:

@incollection{ncc,
title = {Neural Code Comprehension: A Learnable Representation of Code Semantics},
author = {Ben-Nun, Tal and Jakobovits, Alice Shoshana and Hoefler, Torsten},
booktitle = {Advances in Neural Information Processing Systems 31},
editor = {S. Bengio and H. Wallach and H. Larochelle and K. Grauman and N. Cesa-Bianchi and R. Garnett},
pages = {3588--3600},
year = {2018},
publisher = {Curran Associates, Inc.},
url = {http://papers.nips.cc/paper/7617-neural-code-comprehension-a-learnable-representation-of-code-semantics.pdf}
}

Code

Requirements

For training inst2vec embeddings:

  • GNU / Linux or Mac OS
  • Python (3.6.5)
    • tensorflow (1.7.0) or preferably: tensorflow-gpu (1.7.0)
    • networkx (2.1)
    • scipy (1.1.0)
    • absl-py (0.2.2)
    • jinja2 (2.10)
    • bokeh (0.12.16)
    • umap (0.1.1)
    • sklearn (0.0)
    • wget (3.2)

Additionally, for training ncc models:

  • GNU / Linux or Mac OS
  • Python (3.6.5)
    • labm8 (0.1.2)
    • keras (2.2.0)

Running the code

1. Training inst2vec embeddings

By default, inst2vec will be trained on publicly available code. Some additional datasets are available on demand and you may add them manually to the training data. For more information on how to do this as well as on the datasets in general, see datasets.

$ python train_inst2vec.py --helpfull # to see the full list of options
$ python train_inst2vec.py \
>  # --context_width ... (default: 2)
>  # --data ... (default: data/, automatically generated one. You may provide your own)

Alternatively, you may skip this step and use pre-trained embeddings.

2. Evaluating inst2vec embeddings

$ python train_inst2vec.py \
> --embeddings_file ... (path to the embeddings p-file to evaluate)
> --vocabulary_folder ... (path to the associated vocabulary folder)

3. Training on tasks with ncc

We provide the code for training three downstream tasks using the same neural architecture (ncc) and inst2vec embeddings.

Algorithm classification

Task: Classify applications into 104 classes given their raw code.
Code and classes provided by https://sites.google.com/site/treebasedcnn/ (see Convolutional neural networks over tree structures for programming language processing)

Train:

$ python train_task_classifyapp.py --helpfull # to see the full list of options
$ python train_task_classifyapp.py

Alternatively, display results from a pre-trained model.

Optimal device mapping prediction

Task: Predict the best-performing compute device (e.g., CPU, GPU) Code and classes provided by https://github.com/ChrisCummins/paper-end2end-dl (see End-to-end Deep Learning of Optimization Heuristics)

Train:

$ python train_task_devmap.py --helpfull # to see the full list of options
$ python train_task_devmap.py

Alternatively, display results from a pre-trained model.

Optimal thread coarsening factor prediction

Code and classes provided by https://github.com/ChrisCummins/paper-end2end-dl (see End-to-end Deep Learning of Optimization Heuristics)

Train:

$ python train_task_threadcoarsening.py --helpfull # to see the full list of options
$ python train_task_threadcoarsening.py

Alternatively, display results from a pre-trained model.

Contact

We would be thrilled if you used and built upon this work. Contributions, comments, and issues are welcome!

License

NCC is published under the New BSD license, see LICENSE.

More Repositories

1

graph-of-thoughts

Official Implementation of "Graph of Thoughts: Solving Elaborate Problems with Large Language Models"
Python
2,096
star
2

dace

DaCe - Data Centric Parallel Programming
Python
491
star
3

gemm_hls

Scalable systolic array-based matrix-matrix multiplication implemented in Vivado HLS for Xilinx FPGAs.
C++
297
star
4

QuaRot

Code for QuaRot, an end-to-end 4-bit inference of large language models.
Python
259
star
5

pymlir

Python interface for MLIR - the Multi-Level Intermediate Representation
Python
210
star
6

hls_tutorial_examples

Examples shown as part of the tutorial "Productive parallel programming on FPGA with high-level synthesis".
C++
189
star
7

MRAG

Official Implementation of "Multi-Head RAG: Solving Multi-Aspect Problems with LLMs"
Python
161
star
8

serverless-benchmarks

SeBS: serverless benchmarking suite for automatic performance analysis of FaaS platforms.
Python
143
star
9

substation

Research and development for optimizing transformers
Python
121
star
10

pspin

PsPIN: A RISC-V in-network accelerator for flexible high-performance low-power packet processing
SystemVerilog
95
star
11

deep-weather

Deep Learning for Post-Processing Ensemble Weather Forecasts
Jupyter Notebook
86
star
12

daceml

A Data-Centric Compiler for Machine Learning
Python
81
star
13

FBLAS

BLAS implementation for Intel FPGA
C++
75
star
14

open-earth-compiler

development repository for the open earth compiler
MLIR
75
star
15

npbench

NPBench - A Benchmarking Suite for High-Performance NumPy
Python
73
star
16

ucudnn

Accelerating DNN Convolutional Layers with Micro-batches
C++
64
star
17

rFaaS

rFaaS: a high-performance FaaS platform with RDMA acceleration for low-latency invocations.
C++
48
star
18

haystack

Haystack is an analytical cache model that given a program computes the number of cache misses.
C++
42
star
19

sparsity-in-deep-learning

Bibtex for Sparsity in Deep Learning paper (https://arxiv.org/abs/2102.00554) - open for pull requests
TeX
40
star
20

mlir-dace

Data-Centric MLIR dialect
C++
37
star
21

redmark

ReDMArk: Bypassing RDMA Security Mechanisms.
C++
37
star
22

apfp

FPGA acceleration of arbitrary precision floating point computations.
C++
34
star
23

NoPFS

Near-optimal Prefetching System
32
star
24

sten

Sparsity support for PyTorch
Python
31
star
25

rapidchiplet

A toolchain for rapid design space exploration of chiplet architectures
C++
27
star
26

ens10

Scripts and examples for the ENS-10 Ensemble Prediction System machine learning dataset
Python
25
star
27

gms

GraphMineSuite (GMS): a benchmarking suite for graph mining algorithms such as graph pattern matching or graph learning
C++
25
star
28

sage

Python
24
star
29

liblsb

Rebol
23
star
30

smoe

Spatial Mixture-of-Experts
Python
19
star
31

CoRM

CoRM: Compactable Remote Memory over RDMA
C++
19
star
32

dace-vscode

Rich editor for SDFGs with included profiling and debugging, static analysis, and interactive optimization.
TypeScript
18
star
33

kafkadirect

RDMA-enabled Apache Kafka
Java
17
star
34

faaskeeper

A fully serverless implementation of the ZooKeeper coordination protocol.
Python
17
star
35

fmi

Function Message Interface (FMI): library for message-passing and collective communication for serverless functions.
C++
15
star
36

SMI

Streaming Message Interface: High-Performance Distributed Memory Programming on Reconfigurable Hardware
C++
15
star
37

stencilflow

Python
15
star
38

naos

Naos: Serialization-free RDMA networking in Java
Java
15
star
39

absinthe

Absinthe is an optimization framework to fuse and tile stencil codes in one shot
Python
14
star
40

NNCompression

Compressing weather and climate data into neural networks
Python
13
star
41

DNN-cpp-proxies

C++/MPI proxies for distributed training of deep neural networks.
C++
13
star
42

arrow-matrix

Arrow Matrix Decomposition - Communication-Efficient Distributed Sparse Matrix Multiplication
Python
13
star
43

CheckEmbed

Official Implementation of "CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks"
Python
12
star
44

.github

10
star
45

LogGOPSim

A LogGOPS (LogP, LogGP, LogGPS) Simulator and Simulation Framework
C
10
star
46

vldb19-distributed-locking

This repository hosts the code used for the following paper: Claude Barthels, Ingo MΓΌller, Konstantin Taranov, Torsten Hoefler, Gustavo Alonso. "Strong consistency is not hard to get: Two-Phase Locking and Two-Phase Commit on Thousands of Cores." In: PVLDB, 2020.
C++
10
star
47

SimFS

SimFS: A Virtualizing Simulation Data File System Interface
C++
8
star
48

CLaMPI

Caching Layer for MPI
C
8
star
49

FBACode

Python
8
star
50

nbody_hls

Implementation of the N^2-formulation of N-body simulation with Vivado HLS for SDAccel platforms.
C++
8
star
51

GDI-RMA

Official Implementation of "The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores"
C
8
star
52

DiffDA

Python
7
star
53

stencil_hls

Implementation of time and space-tiled stencil in Vivado HLS.
C++
7
star
54

open-earth-benchmarks

Open repository for climate and weather benchmark kernels
C++
7
star
55

cppless

C++
6
star
56

polybench-comparator

Regression and comparison tools for the Polybench benchmark
Shell
6
star
57

nevermore

The source code for the Nevermore paper at ACM CCS'22
C++
6
star
58

foMPI-NA

C
6
star
59

perf-taint

Taint-based program analysis framework for empirical performance modeling.
LLVM
5
star
60

streamingsched

Streaming Task Scheduling
Python
5
star
61

faaskeeper-python

Python client library for FaaSKeeper, the serverless ZooKeeeper.
Python
5
star
62

muliticast-based-allgather

C
4
star
63

smat

Code for High Performance Unstructured SpMM Computation Using Tensor Cores
Emacs Lisp
4
star
64

libNBC

Shell
3
star
65

climetlab-maelstrom-ens10

MAELSTROM ENS10 dataset plugin for CliMetLab
Jupyter Notebook
3
star
66

dace-webclient

Web-based SDFG viewer for DaCe
JavaScript
3
star
67

spatial-collectives

Optimized communication collectives for the Cerebras waferscale engine
Python
3
star
68

libhear

C++
3
star
69

TCPunch

C++
3
star
70

LGSxNS3

Python
2
star
71

cppless-clang

2
star
72

c2dace

C
2
star
73

probgraph

Emacs Lisp
2
star
74

LogGOPSim2

C++
2
star
75

fflib

C
2
star
76

serverless-benchmarks-data

TeX
2
star
77

rivets

C
2
star
78

conflux

C++
1
star
79

fuzzyflow-artifact

Computational artifacts for the FuzzyFlow publication
Shell
1
star
80

SAILOR

Python
1
star
81

praas-benchmarks

Jupyter Notebook
1
star
82

HTSIM-old

C++
1
star
83

faas-profiler

Python
1
star
84

UPM

User-guided Page Merging: Memory Deduplication for Serverless
C
1
star
85

f2dace-artifact

Fortran
1
star