src-d/datasets

Stars
323
Rank 130,051 (Top 3 %)
Language
Jupyter Notebook
License
Other
Created almost 7 years ago
Updated about 5 years ago

src-d/datasets

src-d

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

source{d} datasets ("big code") for source code analysis and machine learning on source code

source{d} Datasets

source{d} datasets for source code analysis and machine learning on source code (ML on Code).

This repository contains all the needed tools and scripts to reproduce the datasets, as well as the academic papers they may relate to.

Available datasets

Public Git Archive

Public Git Archive
Size: 6TB
Description: 260k+ top-bookmarked repositories from GitHub, consisting of 136M+ files and ~28 billion lines of code.

Programming Language Identifiers

Programming Language Identifiers
Size: 1GB
Description: ~49M distinct identifiers extracted from 10+ programming languages.

Code duplicates

Manually labelled pairs of files and functions
Size: 250MB
Description: 2k Java file and 600 Java function pairs labeled as similar or different by several programmers.

Pull Request review comments

PR review comments
Size: 1.5GB
Description: 25.3 million GitHub PR review comments since January 2015 till December 2018.

Commit messages

Commit messages
Size: 46GB
Description: 1.3 billion GitHub commit messages till March 2019.

Structural commit features

Structural commit features
Size: 1.9GB
Description: 1.6 million commits in 622 Java repositories on GitHub.

DockerHub Metadata

DockerHub Metadata
Size: 1.4GB
Description: 1.46 million Docker image configuration and manifest files on DockerHub fetched in June 2019.

DockerHub Packages

DockerHub Packages
Size: 15GB
Description: 419092 analyzed Docker images: lists of native, Python and Node packages on DockerHub fetched in summer 2019.

Typos

Typos
Size: 1MB
Description: 7375 typos in source code identifier names found in GitHub repositories.

NuGet Namespaces

NugetNamespaces
Size: 13MB
Description: information about 681,858 .NET namespaces extracted from 227,839 NuGet packages.

Contributions

Contributions are very welcome, please see CONTRIBUTING.md and code of conduct.

License

The tools and scripts are licensed under Apache 2.0, see LICENSE.md.

awesome-machine-learning-on-source-code

Cool links & research papers related to Machine Learning applied to source code (MLonCode)

go-git

Project has been moved to: https://github.com/go-git/go-git

hercules

Gaining advanced insights from Git repository history.

gitbase

SQL interface to git repositories, written in Go. https://docs.sourced.tech/gitbase

go-mysql-server

An extensible MySQL server implementation in Go.

go-kallax

Kallax is a PostgreSQL typesafe ORM for the Go language.

kmcuda

Large scale K-means and K-nn implementation on NVIDIA GPU / CUDA

Jupyter Notebook

proteus

Generate .proto files from Go source code.

wmd-relax

Calculates Word Mover's Distance Insanely Fast

enry

A faster file programming language detector

guide

Aiming to be a fully transparent company. All information about source{d} and what it's like to work here.

lapjv

Linear Assignmment Problem solver using Jonker-Volgenant algorithm - Python 3 native module.

go-license-detector

Reliable project licenses detector.

engine-deprecated

[DISCONTINUED] Go to https://github.com/src-d/sourced-ce/

go-billy

The missing interface filesystem abstraction for Go

sourced-ce

source{d} Community Edition (CE)

beanstool

Dependency free beanstalkd admin tool

lookout

Assisted code review, running custom code analyzers on pull requests

ml

sourced.ml is a library and command line tools to build and apply machine learning models on top of Universal Abstract Syntax Trees

reading-club

Paper reading club at source{d}

minhashcuda

Weighted MinHash implementation on CUDA (multi-gpu).

go-siva

siva - seekable indexed verifiable archiver

jgit-spark-connector

jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.

gitbase-web

gitbase web client; source{d} CE comes with a new UI, check it at https://docs.sourced.tech/community-edition/

gemini

Advanced similarity and duplicate source code at scale.

apollo

Advanced similarity and duplicate source code proof of concept for our research efforts.

borges

borges collects and stores Git repositories.

okrs

Objectives & Key Results repository for the source{d} team

go-queue

Queue is a generic interface to abstract the details of implementation of queue systems.

vecino

Vecino is a command line application to discover Git repositories which are similar to the one that the user provides.

jgscm

Jupyter support for Google Cloud Storage

code2vec

MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)

coreos-nvidia

Yet another NVIDIA driver container for Container Linux (aka CoreOS)

style-analyzer

Lookout Style Analyzer: fixing code formatting and typos during code reviews

Jupyter Notebook

code-annotation

🐈 Code Annotation Tool

flamingo

Flamingo is a very thin and simple platform-agnostic chat bot framework

blog

sparkpickle

Pure Python implementation of reading SequenceFile-s with pickles written by Spark's saveAsPickleFile()

go-errors

Yet another errors package, implementing error handling primitives.

homebrew

infrastructure-dockerfiles

Dockerfile-s to build the images which power source{d}'s computing infrastructure.

conferences

Tracking events, CfPs, abstracts, slides, and all other even related things

tmsc

models

Machine learning models for MLonCode trained using the source{d} stack

terraform-provider-online

Terraform provider for Online.net

modelforge

Python library to share machine learning models easily and reliably.

identity-matching

source{d} extension to match Git signatures to real people.

tensorflow-swivel

seriate

Optimal ordering of elements in a set given their distance matrix.

gitcollector

go-vitess

An automatic filter-branch of Go libraries from the great Vitess project.

rovers

Rovers is a service to retrieve repository URLs from multiple repository hosting providers.

go-parse-utils

ml-core

source{d} MLonCode foundation - core algorithms and models.

charts

Applications for Kubernetes

role2vec

snippet-ranger

Jupyter Notebook

fsbench

a small tool for benchmarking filesystems

dev-similarity

Jupyter Notebook

go-log

Log is a generic logging library based on logrus

tab-vs-spaces

Jupyter Notebook

ghsync

GitHub API v3 > PostgreSQL

diffcuda

Accelerated bulk diff on GPU

ml-mining

go-billy-siva

A limited go-billy filesystem implementation based on siva.

go-compose-installer

A toolkit to create installers based on docker compose.

github-reminder

A GitHub application to handle deadline reminders in a GitHub idiomatic way.

go-git-fixtures

several git fixtures to run go-git tests

docsrv

docsrv is an app to serve versioned documentation for GitHub projects on demand

go-cli

CLI scaffolding for Go

shell-complete

kubernetes-local-pv-provisioner

Helping you setting up local persistent volumes

engine-analyses

Analyses of open source projects with source{d} Engine

Jupyter Notebook

sourced-ui

gypogit

[UNMAINTAINED] go-git wrapper for Python

go-borges

treediff

engine-tour

Temporary storage for useful guides for the source{d} engine

Jupyter Notebook

jupyter-spark-docker

Dockerfile with jupyter and scala installed

imports

git-validate

k8s-pod-headless-service-operator

landing

landing for source{d}

lookout-terraform-analyzer

This is a lookout analyzer that checks if your PR has been Terraform fmt'ed when submitting it.

swivel-spark-prep

Distributed equivalent of prep.py and fastprep from Swivel using Apache Spark.

ci

Make-based build system for Go projects at source{d}

framework

platform-starter

Starter and basic configuration for platform frontend projects.

metadata-retrieval

lookout-sdk

SDK for lookout analyzers

code-completion

autocompletion prototype

siva-java

siva format implemented in Java

design

All things design at source{d}: branding, guidelines, UI assets, media & co.

berserker

Large scale UAST extractor [DEPRECATED]

combustion

tm-experiments

Topic Modeling Experiments on Source Code

go-YouTokenToMe

lookout-sdk-ml

SDK for ML based Lookout analyzers

go-asdf

Advanced Scientific Data Format reader library in pure Go.

google-cloud-dns-healthcheck