• Stars
    star
    217
  • Rank 182,446 (Top 4 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 6 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[DISCONTINUED] Go to https://github.com/src-d/sourced-ce/
source{d} Engine

Powerful language-agnostic analysis of your source code and git history.

GitHub version Build Status Go Report Card GoDoc source{d} design document

WebsiteDocumentationBlogSlackTwitter

Introduction

source{d} Engine exposes powerful Universal ASTs to analyze your code and a SQL engine to analyze your git history:

  • Code Processing: use git repositories as a dataset.
  • Language-Agnostic Code Analysis: automatically identify languages, parse source code, and extract the pieces that matter with language-independent queries.
  • Git Analysis: powerful SQL based analysis on top of your git repositories.
  • Querying With Familiar APIs: analyze your code through powerful friendly APIs, such as SQL, gRPC, and various client libraries.

You can access a rendered version of this documentation at docs.sourced.tech/engine.

Contents

Quick Start

Follow the steps below to get started with source{d} Engine.

1. Install Docker

Follow these instructions based on your OS:

Docker on macOS

Follow instructions at Docker for Mac. You may also use Homebrew:

brew cask install docker

Docker on Ubuntu Linux

Follow instructions at Docker for Ubuntu Linux:

sudo apt-get update
sudo apt-get install docker-ce

Docker on Arch Linux

Follow instructions at Docker for Arch Linux:

sudo pacman -S docker

Docker on Windows

Docker Desktop for Windows. Make sure to read the system requirements here.

2. Install source{d} Engine

Please note that Docker Toolbox is not supported neither for Windows nor for macOS. In case that you're running Docker Toolbox, please consider updating to newer Docker Desktop for Mac or Docker Desktop for Windows. See previous section

Download the latest release for MacOS (Darwin), Linux or Windows.

Engine on macOS

Double-click on the tar file to extract it.

Open your terminal and move the binary to your local bin folder to make it executable from any directory:

sudo mv ~/replace/path/to/engine_darwin_amd64/srcd /usr/local/bin/

If you want to setup bash completion, add the following to your $HOME/.bash_profile file:

# source{d} Engine auto completion
source <(srcd completion)

Engine on Linux

Extract the contents of the tar file from your terminal:

tar -xvf ~/replace/path/to/engine_REPLACEVERSION_linux_amd64.tar.gz

Move the binary to your local bin folder to be executable from any directory:

sudo mv engine_linux_amd64/srcd /usr/local/bin/

If you want to setup bash completion, add the following to your $HOME/.bashrc file:

# source{d} Engine auto completion
source <(srcd completion)

Engine on Windows

Please note that from now on we assume that the commands are executed in powershell and not in cmd. Running them in cmd is not guaranteed to work. Proper support may be added in future releases.

To run the following preparatory steps you need to run powershell as administrator.

mkdir 'C:\Program Files\srcd'
# Add the directory to the `%path%` to make it available from anywhere
setx /M PATH "$($env:path);C:\Program Files\srcd"
# Now open a new powershell to apply the changes

Extract the tar file with the tool you prefer. Copy srcd.exe into the directory you created:

mv engine_windows_amd64\srcd.exe 'C:\Program Files\srcd'

3. Start source{d} Engine With Your Local Repositories

Now it's time to initialize source{d} Engine and provide it with some repositories to analyze:

# Without a path Engine operates on the local directory,
# it works with nested or sub-directories.
srcd init

# You can also provide a path
srcd init <path>

You can also optionally provide a configration file through the --config flag as follows:

srcd init --config <config file path>
# Or
srcd init <path> --config <config file path>

See the documentation for the --config flag here.

Note: Once Engine is initialized with a working dir, it does not watch for new repository creation, so if you want to add (or delete) repositories, you need to init again. Also, database indexes are not updated automatically when its contents change, so in that cases, the index must be manually recreated. If you want to inspect different datasets, it needs to be done separatelly: you can init one working directory, perform your queries, and then change to the other dataset performing an init with the working directory containing it. In such situations, indexes are kept, so you can change from one dataset to the other, without having to recreate them.

Note for MacOS: Docker for Mac requires enabling file sharing for any path outside of /Users.

4. Explore source{d} Engine

For the full list of the commands supported by srcd and those that have been planned, please read commands.md.

Note for Windows: Docker for Windows requires shared drives. Other than that, it's important to use a workdir that doesn't include any sub-directory whose access is not readable by the user running srcd. As an example using C:\Users as workdir will most probably not work. For more details see this issue.

source{d} Engine provides interfaces to query code repositories and to parse code into Universal Abstract Syntax Trees.

In this section we will cover a mix of some commands and interfaces available.

Note: source{d} Engine will download and install Docker images on demand. Therefore, the first time you run some of these commands, they might take a bit of time to start up. Subsequent runs will be faster.

Querying Code

Query Web Interface

To launch the web client for the SQL interface, run the following command and start executing queries:

# Launch the query web client
srcd web sql

This should open the web interface in your browser. You can also access it directly at http://localhost:8080.

Query Command Line Interface (CLI)

If you prefer to work within your terminal via command line, you can open a SQL REPL that allows you to execute queries against your repositories by executing:

# Launch the query CLI REPL
srcd sql

If you want to run a query directly, you can also execute it as such:

# Run query via CLI
srcd sql "SHOW tables;"

Note: Engine's SQL supports a UAST function that returns a Universal AST for the selected source text. UAST values are returned as binary blobs, and are best visualized in the web sql interface rather than the CLI where are seen as binary data.

From Any MySQL Client

You may also connect directly to gitbase using any MySQL compatible client. Use the login user root, no password, and database name gitbase.

# Start the component if needed
srcd start srcd-cli-gitbase

mysql --user=root --host=127.0.0.1 --port=3306 gitbase

Parsing Code

Sometimes you may want to parse files directly as UASTs.

To see which languages are available, check the table of supported languages.

Parse Web Client

If you want a playground to see examples of the UAST, or run your own, you can launch the parse web client.

# Launch the parse web client
srcd web parse

This should open the web interface in your browser. You can also access it directly at http://localhost:8081.

Parse Command Line Interface (CLI)

Alternatively, you can also start parsing files on the command line:

# Parse file via CLI
srcd parse uast /path/to/file.java

To parse a file specifying the programming language:

srcd parse uast --lang=LANGUAGE /path/to/file

To see the installed language drivers:

srcd parse drivers

5. Start Executing Queries

Understand which tables are available for you to query:

gitbase> show tables;
+--------------+
|    TABLE     |
+--------------+
| blobs        |
| commit_blobs |
| commit_files |
| commit_trees |
| commits      |
| files        |
| ref_commits  |
| refs         |
| remotes      |
| repositories |
| tree_entries |
+--------------+
gitbase> DESCRIBE TABLE commits;
+---------------------+-----------+
|        NAME         |   TYPE    |
+---------------------+-----------+
| repository_id       | TEXT      |
| commit_hash         | TEXT      |
| commit_author_name  | TEXT      |
| commit_author_email | TEXT      |
| commit_author_when  | TIMESTAMP |
| committer_name      | TEXT      |
| committer_email     | TEXT      |
| committer_when      | TIMESTAMP |
| commit_message      | TEXT      |
| tree_hash           | TEXT      |
| commit_parents      | JSON      |
+---------------------+-----------+

Show me the repositories I am analyzing:

SELECT * FROM repositories;

Top 10 repositories by commit count in HEAD:

SELECT repository_id,commit_count
FROM (
    SELECT r.repository_id, COUNT(*) AS commit_count
    FROM ref_commits r
    WHERE r.ref_name = 'HEAD'
    GROUP BY r.repository_id
) AS q
ORDER BY commit_count
DESC
LIMIT 10;

Query all files from HEAD:

SELECT cf.file_path, f.blob_content
FROM ref_commits r
NATURAL JOIN commit_files cf
NATURAL JOIN files f
WHERE r.ref_name = 'HEAD'
AND r.history_index = 0;

Retrieve the UAST for all files from HEAD:

SELECT * FROM (
    SELECT cf.file_path,
           UAST(f.blob_content, LANGUAGE(f.file_path,  f.blob_content)) as uast
    FROM ref_commits r
    NATURAL JOIN commit_files cf
    NATURAL JOIN files f
    WHERE r.ref_name = 'HEAD'
    AND r.history_index = 0
) t WHERE uast != '';

Query for all LICENSE & README files across history:

SELECT repository_id, blob_content
FROM files
WHERE file_path = 'LICENSE'
OR file_path = 'README.md';

You can find further sample queries in the examples folder.

Other Guides and Examples

Here is a collection of documentation, guides, and examples of the components exposed by source{d} Engine:

Architecture

source{d} Engine functions as a command-line interface tool that provides easy access to components of source{d} stack for Code As Data.

It consists of a daemon managing all of the services, which are packaged as Docker containers:

For more details on the architecture of this project, read docs/architecture.md.

diagram of source{d} Engine

Babelfish UAST

Definition

One of the most important components of source{d} Engine is the UAST, which stands for: Universal Abstract Syntax Tree.

UASTs are a normalized form of a programming language's AST, annotated with language-agnostic roles and transformed with language-agnostic concepts (e.g. Functions, Imports etc.).

These enable advanced static analysis of code and easy feature extraction for statistics or Machine Learning on Code.

UAST Usage

To parse a file for a UAST using source{d} Engine, head to the Parsing Code section of this document.

Supported Languages

To see which languages are available, check the table of Babelfish supported languages.

Clients and Connectors

For connecting to the language parsing server (Babelfish) and analyzing the UAST, there are several language clients currently supported and maintained:

The Gitbase Spark connector is under development, which aims to allow for an easy integration with Spark & PySpark:

Community

source{d} has an amazing community of developers and contributors who are interested in Code As Data and/or Machine Learning on Code. Please join us! 👋

Contributing

Contributions are welcome and very much appreciated 🙌 Please refer to our Contribution Guide for more details.

Credits

This software uses code from open source packages. We'd like to thank the contributors for all their efforts:

License

Apache 2.0 License

More Repositories

1

awesome-machine-learning-on-source-code

Cool links & research papers related to Machine Learning applied to source code (MLonCode)
6,247
star
2

go-git

Project has been moved to: https://github.com/go-git/go-git
Go
4,904
star
3

hercules

Gaining advanced insights from Git repository history.
Go
2,613
star
4

gitbase

SQL interface to git repositories, written in Go. https://docs.sourced.tech/gitbase
Go
2,063
star
5

go-mysql-server

An extensible MySQL server implementation in Go.
Go
1,040
star
6

go-kallax

Kallax is a PostgreSQL typesafe ORM for the Go language.
Go
858
star
7

kmcuda

Large scale K-means and K-nn implementation on NVIDIA GPU / CUDA
Jupyter Notebook
800
star
8

proteus

Generate .proto files from Go source code.
Go
734
star
9

wmd-relax

Calculates Word Mover's Distance Insanely Fast
Python
461
star
10

enry

A faster file programming language detector
Go
460
star
11

datasets

source{d} datasets ("big code") for source code analysis and machine learning on source code
Jupyter Notebook
323
star
12

guide

Aiming to be a fully transparent company. All information about source{d} and what it's like to work here.
JavaScript
294
star
13

lapjv

Linear Assignmment Problem solver using Jonker-Volgenant algorithm - Python 3 native module.
C++
252
star
14

go-license-detector

Reliable project licenses detector.
Go
237
star
15

go-billy

The missing interface filesystem abstraction for Go
Go
199
star
16

sourced-ce

source{d} Community Edition (CE)
Go
188
star
17

beanstool

Dependency free beanstalkd admin tool
Go
151
star
18

lookout

Assisted code review, running custom code analyzers on pull requests
Go
149
star
19

ml

sourced.ml is a library and command line tools to build and apply machine learning models on top of Universal Abstract Syntax Trees
Python
141
star
20

reading-club

Paper reading club at source{d}
115
star
21

minhashcuda

Weighted MinHash implementation on CUDA (multi-gpu).
C++
114
star
22

go-siva

siva - seekable indexed verifiable archiver
Go
98
star
23

jgit-spark-connector

jgit-spark-connector is a library for running scalable data retrieval pipelines that process any number of Git repositories for source code analysis.
Scala
71
star
24

gitbase-web

gitbase web client; source{d} CE comes with a new UI, check it at https://docs.sourced.tech/community-edition/
Go
57
star
25

gemini

Advanced similarity and duplicate source code at scale.
Scala
54
star
26

apollo

Advanced similarity and duplicate source code proof of concept for our research efforts.
Python
52
star
27

borges

borges collects and stores Git repositories.
Go
52
star
28

okrs

Objectives & Key Results repository for the source{d} team
48
star
29

go-queue

Queue is a generic interface to abstract the details of implementation of queue systems.
Go
47
star
30

vecino

Vecino is a command line application to discover Git repositories which are similar to the one that the user provides.
Python
46
star
31

jgscm

Jupyter support for Google Cloud Storage
Python
45
star
32

code2vec

MLonCode community effort to implement Learning Distributed Representations of Code (https://arxiv.org/pdf/1803.09473.pdf)
Python
40
star
33

coreos-nvidia

Yet another NVIDIA driver container for Container Linux (aka CoreOS)
Makefile
38
star
34

style-analyzer

Lookout Style Analyzer: fixing code formatting and typos during code reviews
Jupyter Notebook
32
star
35

code-annotation

🐈 Code Annotation Tool
JavaScript
28
star
36

flamingo

Flamingo is a very thin and simple platform-agnostic chat bot framework
Go
27
star
37

blog

source{d} blog
HTML
27
star
38

sparkpickle

Pure Python implementation of reading SequenceFile-s with pickles written by Spark's saveAsPickleFile()
Python
24
star
39

go-errors

Yet another errors package, implementing error handling primitives.
Go
23
star
40

homebrew

Real homebrew!
22
star
41

infrastructure-dockerfiles

Dockerfile-s to build the images which power source{d}'s computing infrastructure.
Dockerfile
22
star
42

conferences

Tracking events, CfPs, abstracts, slides, and all other even related things
22
star
43

tmsc

Python
21
star
44

models

Machine learning models for MLonCode trained using the source{d} stack
19
star
45

terraform-provider-online

Terraform provider for Online.net
Go
19
star
46

modelforge

Python library to share machine learning models easily and reliably.
Python
18
star
47

identity-matching

source{d} extension to match Git signatures to real people.
Go
17
star
48

tensorflow-swivel

C++
16
star
49

seriate

Optimal ordering of elements in a set given their distance matrix.
Python
16
star
50

gitcollector

Go
15
star
51

go-vitess

An automatic filter-branch of Go libraries from the great Vitess project.
Go
15
star
52

rovers

Rovers is a service to retrieve repository URLs from multiple repository hosting providers.
HTML
14
star
53

go-parse-utils

Go
14
star
54

ml-core

source{d} MLonCode foundation - core algorithms and models.
Python
14
star
55

charts

Applications for Kubernetes
Smarty
12
star
56

role2vec

TeX
12
star
57

snippet-ranger

Jupyter Notebook
12
star
58

fsbench

a small tool for benchmarking filesystems
Go
11
star
59

dev-similarity

Jupyter Notebook
11
star
60

go-log

Log is a generic logging library based on logrus
Go
11
star
61

tab-vs-spaces

Jupyter Notebook
10
star
62

ghsync

GitHub API v3 > PostgreSQL
Go
9
star
63

diffcuda

Accelerated bulk diff on GPU
C
9
star
64

ml-mining

Python
8
star
65

go-billy-siva

A limited go-billy filesystem implementation based on siva.
Go
8
star
66

go-compose-installer

A toolkit to create installers based on docker compose.
Go
8
star
67

github-reminder

A GitHub application to handle deadline reminders in a GitHub idiomatic way.
Go
8
star
68

go-git-fixtures

several git fixtures to run go-git tests
Go
8
star
69

docsrv

docsrv is an app to serve versioned documentation for GitHub projects on demand
Go
7
star
70

go-cli

CLI scaffolding for Go
Go
7
star
71

shell-complete

Python
7
star
72

kubernetes-local-pv-provisioner

Helping you setting up local persistent volumes
Go
7
star
73

engine-analyses

Analyses of open source projects with source{d} Engine
Jupyter Notebook
7
star
74

sourced-ui

source{d} UI
JavaScript
7
star
75

gypogit

[UNMAINTAINED] go-git wrapper for Python
Python
6
star
76

go-borges

Go
6
star
77

treediff

Python
6
star
78

engine-tour

Temporary storage for useful guides for the source{d} engine
Jupyter Notebook
6
star
79

jupyter-spark-docker

Dockerfile with jupyter and scala installed
Dockerfile
6
star
80

imports

Go
6
star
81

git-validate

Go
6
star
82

k8s-pod-headless-service-operator

Go
6
star
83

landing

landing for source{d}
HTML
5
star
84

lookout-terraform-analyzer

This is a lookout analyzer that checks if your PR has been Terraform fmt'ed when submitting it.
Go
5
star
85

swivel-spark-prep

Distributed equivalent of prep.py and fastprep from Swivel using Apache Spark.
Scala
5
star
86

ci

Make-based build system for Go projects at source{d}
Shell
5
star
87

framework

[DEPRECATED]
Go
4
star
88

platform-starter

Starter and basic configuration for platform frontend projects.
Go
4
star
89

metadata-retrieval

Go
4
star
90

lookout-sdk

SDK for lookout analyzers
Python
4
star
91

code-completion

autocompletion prototype
Python
4
star
92

siva-java

siva format implemented in Java
Java
4
star
93

design

All things design at source{d}: branding, guidelines, UI assets, media & co.
4
star
94

berserker

Large scale UAST extractor [DEPRECATED]
Shell
4
star
95

combustion

Go
3
star
96

tm-experiments

Topic Modeling Experiments on Source Code
Python
3
star
97

go-YouTokenToMe

Go
3
star
98

lookout-sdk-ml

SDK for ML based Lookout analyzers
Python
3
star
99

go-asdf

Advanced Scientific Data Format reader library in pure Go.
Go
3
star
100

google-cloud-dns-healthcheck

Go
3
star