GitHub Search ·

This project is made of two components:

1. A Spring Boot powered back-end, responsible for:
   1. Continuously crawling GitHub API endpoints for repository information, and storing it in a central database;
   2. Acting as an API for providing access to the stored data.
2. A Bootstrap-styled and jQuery powered web user interface, serving an accessible front for the API, available at http://seart-ghs.si.usi.ch

Running Locally

Prerequisites

Database

Before choosing whether to start with a clean slate or pre-populated database, make sure the following requirements are met:

1. The database timezone is set to +00:00. You can verify this via:

   SELECT @@global.time_zone, @@session.time_zone;

2. The event scheduler is turned ON. You can verify this via:

   SELECT @@global.event_scheduler;

3. The binary logging during the creation of stored functions is set to 1. You can verify this via:

   SELECT @@global.log_bin_trust_function_creators;

4. The gse database exists. To create it:

   CREATE DATABASE gse CHARACTER SET utf8 COLLATE utf8_bin;

5. The gseadmin user exists. To create one, run:

   CREATE USER IF NOT EXISTS 'gseadmin'@'%' IDENTIFIED BY 'Lugano2020';
   GRANT ALL ON gse.* TO 'gseadmin'@'%';

If you want to start with a completely blank database, then no further action is required. The necessary tables will be created by virtue of Flyway migrations, which will run on initial server startup. However, if you want your local database to be pre-initialized with the data we have mined, then you can use the compressed SQL dump we provide. We host said dump, as well as the previous 4 iterations, on Dropbox. Once you select and download a database dump, you can import the data by running:

gzcat < gse.sql.gz | mysql -u gseadmin -pLugano2020 gse

Server

Before attempting to run the server, you must generate your own GitHub personal access token (PAT). GHS relies on the GraphQL API, which is inaccessible without authentication. The token must include the repo scope, in order for it to access the information present in the GitHub API.

Once that is done, you can run the server locally using Maven:

mvn spring-boot:run

If you want to make use of the token when crawling, specify it in the run arguments:

mvn spring-boot:run -Dspring-boot.run.arguments=--ghs.github.tokens=<your_access_token>

Alternatively, you can compile and run the JAR directly:

mvn clean package
ln target/ghs-application-*.jar target/ghs-application.jar
java -Dghs.github.tokens=<your_access_token> -jar target/ghs-application.jar

Here's a list of project-specific arguments supported by the application that you can find in the application.properties:

Web UI

The easiest way to start the front-end is through IntelliJ's built-in web server. After starting the application back-end right click on index.html in the html directory, and select one of the provided launch options from Open In > Browser. Alternatively, you can self-host the web UI by virtue of tools such as http-server:

# install by running: npm install -g http-server
http-server html -p 3030

Regardless of which method you choose for hosting, the back-end CORS restricts you to using either port 3030 or 7030.

Dockerisation 🐳

The deployment stack consists of the following containers:

The service dependency chain can be represented as follows:

graph RL
    gse-migration --> |service_healthy| gse-database
    gse-backup --> |service_completed_successfully| gse-migration
    gse-server --> |service_completed_successfully| gse-migration
    gse-website --> |service_healthy| gse-server
    gse-watchtower --> |service_started| gse-website

Deploying is as simple as, in the docker-compose directory, run:

docker-compose -f docker-compose.yml up -d

It's worth mentioning that the database setup steps outlined in the previous section are not needed when running with docker, as the environment properties passed to the service will create the user and DB on first ever startup. However, the same does not apply to the database data, as the default deployment will create an empty database. If you want to use existing data from the dumps, then you have to override the compose deployment to use a custom database image that comes bundled with the dump. Create your docker-compose.override.yml file, and add to it the following contents:

version: '3.9'
name: 'gse'

services:

  gse-database:
     image: seart/ghs-database:latest

The above image will include the freshest database dump, at most 15 days behind the actual platform data. For a more specific database version, refer to the Docker Hub page. Just remember to specify the override file during deployment:

docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d

The database data itself is kept in the gse-data volume, while detailed back-end logs are kept in a local mount called logs.

You can also use this override file to change the configurations of other services, for instance specifying your own PAT for the crawler:

version: '3.9'
name: 'gse'

services:

  # other services omitted...

  gse-server:
    environment:
      GHS_GITHUB_TOKENS: 'A single or comma-separated list of token(s)'
      GHS_CRAWLER_ENABLED: 'true'

Any of the Spring Boot properties or aforementioned application-specific properties can be overridden. Just keep in mind that ghs.x.y corresponds to the GHS_X_Y service environment setting.

Another example is the automated database backup service, which is disabled by default. Should you chose to re-enable it, you would have to add the following to the override file:

version: '3.9'
name: 'gse'

services:

  # other services omitted...

  gse-backup:
    restart: always
    entrypoint: "/init"

FAQ

How can I report a bug or request a feature or ask a question?

Please add a new issue, and we will get back to you very soon.

How do I extend/modify the existing database schema?

In order to do that, you should be familiar with database migration tools and practices. This project in particular uses Flyway by Redgate. However, the general rule for schema manipulation is: create new migrations, and do not edit existing ones.