• Stars
    star
    221
  • Rank 179,773 (Top 4 %)
  • Language
    Go
  • License
    BSD 2-Clause "Sim...
  • Created over 9 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A fast build system for Docker images.

Conveyor

Conveyor builds Docker images. Fast.

How it works

  1. Conveyor receives a build request via a GitHub commit webhook.
  2. Conveyor builds and tags the resulting image with 3 tags: latest, the git commit sha and the git branch.
  3. It then pushes the image to the Docker registry and adds a commit status to the GitHub commit.

Installation

  1. Conveyor needs access to pull GitHub repositories. The easiest way to do this is to add a bot user to your organization and generate an ssh key for them. Once you've done that, create a new S3 bucket and upload id_rsa and id_rsa.pub to the root of the bucket.
  2. Create a GitHub access token with repo, admin:repo_hook scopes.
  3. Create a new CloudFormation stack using cloudformation.json in this repo.

Configuration

The server command has the following available options:

NAME:
   server - Run an http server to build Docker images whenever a push event happens on GitHub

USAGE:
   command server [command options] [arguments...]

OPTIONS:
   --port '8080'        Port to run the server on [$PORT]
   --github.token         GitHub API token to use when updating commit statuses on repositories. [$GITHUB_TOKEN]
   --github.secret        Shared secret used by GitHub to sign webhook payloads. This secret will be used to verify that the request came from GitHub. [$GITHUB_SECRET]
   --dry          Enable dry run mode. [$DRY]
   --builder.image 'remind101/conveyor-builder' A docker image to use to perform the build. [$BUILDER_IMAGE]
   --logger 'stdout://'       The logger to use. Available options are `stdout://`, or `s3://bucket`. [$LOGGER]
   

Performance

Conveyor is designed to be faster than alternative build systems like the Docker Hub or Quay. It does this by making the following tradeoffs.

  1. It uses the latest version of Docker (1.8), which has a number of performance improvements when building and pushing images.
  2. It pulls the last built image for the branch to maximize the number of layers that can be used from the cache.

Cache

By default, conveyor will pull the last built image for the branch. This isn't always desirable, so you can disable the initial docker pull by adding the following to the git commit description:

[docker nocache]

Scale Out

Conveyor supports two methods to scale out to multiple machines.

Docker Swarm

The first method to scale out Conveyor is to scale out using Docker Swarm. Using this method, Conveyor runs its builds across a cluster of Docker daemons. The advantage of using this method is that you don't need to provide a queue flag since Conveyor can use an in memory queue.

Queue

The recommended way to scale out is to scale out using a build queue. Using this method, you run the conveyor worker subcommand on a machine that hosts a local Docker daemon. The worker process will pull build requests off of the queue and perform the build. The conveyor server command can then run completely separate from the worker nodes.

Conveyor currently supports the following build queues:

  1. SQS

Slack Integration

Conveyor can optionally expose some management tasks via Slack slash commands.

Setup

  1. Add a new Slash command. I'd recommend using /conveyor as the command.
  2. Copy the token and provide it as the --slack.token flag.

Now, you can use Conveyor to automatically manage the GitHub webhook:

/conveyor setup org/repo

API

Conveyor also sports a restful API for triggering builds. You can use this with tooling to, say for example, trigger a build before you deploy.

See schema.md for documentation about the API.

Development

First, bootstrap the remind101/conveyor-builder image, SSH keys and docker config:

$ make bootstrap

Then start it up with docker-compose:

$ docker-compose up

You can test a simple dry run build of the remind101/acme-inc repo with:

$ make test-payload

If you want to test external GitHub webhooks, the easiest way to do that is using ngrok:

$ ngrok $(docker-machine ip default):8080

Then add a new push webhook to a repo, pointed at the ngrok URL. No secret is necessary unless you set GITHUB_SECRET in .env.

NOTE: If you're testing on a private repo, you need to make sure that you've added the generated SSH key to your github account. The generated SSH key can be found in ./builder/docker/data/.ssh/id_rsa.pub

Tests

To run the full test suite. Note that you need to run make bootstrap prior to running this:

$ go test ./...

To run only the unit tests (no need to run make bootstrap):

$ go test ./... -short

More Repositories

1

empire

A PaaS built on top of Amazon EC2 Container Service (ECS)
Go
2,688
star
2

assume-role

Easily assume AWS roles in your terminal.
Go
566
star
3

android-arch-sample

Sample app for MVP Architecture on Android
Java
348
star
4

ssm-env

Expand env variables from AWS Parameter Store
Go
251
star
5

tugboat

Rest API and AngularJS client for deploying github repos.
Go
235
star
6

slashdeploy

GitHub Deployments for Slack
JavaScript
153
star
7

deploy

CLI for GitHub Deployments
Go
134
star
8

jest-transform-graphql

Make .graphql file importing work in Jest
JavaScript
121
star
9

AutoGraph

A GraphQL Client in Swift
Swift
120
star
10

rest-graphql

Middleware for Express to adapt REST requests to GraphQL queries
JavaScript
55
star
11

angular-tooltip

Simple and extensible tooltips for angularjs
JavaScript
41
star
12

migrate

Simple migrations for database/sql
Go
40
star
13

stacker_blueprints

DEPRECATED - moved to:
Python
39
star
14

emp

[DEPRECATED] Command line interface for Empire
Go
37
star
15

mq-go

SQS Consumer Server for Go
Go
28
star
16

dbsnap

Tool to copy and verify AWS RDS snapshots.
Python
24
star
17

pkg

A layer of convenience over the Go stdlib
Go
22
star
18

newrelic

DEPRECATED: Use the official lib here https://github.com/newrelic/go-agent
Go
19
star
19

logspout-kinesis

A Logspout adapter for writing logs to Amazon Kinesis
Go
16
star
20

request_id

Middleware for logging heroku request id's
Ruby
16
star
21

exceptions

A Ruby gem for tracking exceptions.
Ruby
15
star
22

grape-pagination

Pagination helpers for Grape.
Ruby
14
star
23

dockerstats

Easy scraping for the Docker stats api.
Go
14
star
24

auto-value-realm

An extension for Google's AutoValue that allows using Realm's datastore
Java
13
star
25

ecsdog

[DEPRECATED] ECS events are now automatically pulled in with the AWS integration
Go
10
star
26

kinesumer

Kinesis consumer library in Go
Go
10
star
27

turbolinks-redirect

Simple redirect_to support for turbolinks and jquery-rails.
Ruby
9
star
28

docker-build

A small script for building, tagging and pushing Docker images
Shell
9
star
29

dnsdog

DNS metrics in DataDog
Go
9
star
30

empire_ami

Home of the AMI building tools for the Official Empire AMI
Shell
9
star
31

policies

Remind Privacy Policy & Terms of Service from www.remind.com
7
star
32

collective

[DEPRECATED use https://github.com/remind101/r101-datadog instead] It collects metrics and puts it on STDOUT.
Ruby
6
star
33

hubot-deploy

Hubot script for GitHub Deployments.
CoffeeScript
6
star
34

reInvent-2015

Slides and Demo resources for Docker & ECS in Production talk.
Go
4
star
35

ruby-cloud-profiler

Ruby
3
star
36

capybara-mocktime

Ruby gem for synchronizing time between tests and the browser using Timecop and Sinon.
Ruby
3
star
37

activerecord-poro

Associations for plain old ruby objects
Ruby
3
star
38

AutoGraphParser

Swift GQL Parser
Swift
3
star
39

homebrew-formulae

Homebrew tap for Remind tools and utilities.
Ruby
3
star
40

formatted-metrics

Easily produce metrics for consumption by l2met.
Ruby
3
star
41

acme-inc

An app that does nothing.
Go
2
star
42

kinesis

Go program and library for streaming to Amazon Kinesis.
Go
2
star
43

gopheragent

A golang user-agent parser
Go
2
star
44

dockerdog

Better Docker event metrics for DataDog
Go
2
star
45

share-on-remind-extension

Share on Remind Extension
JavaScript
2
star
46

git-deploy

Ruby
2
star
47

amazon-ecs-agent

The official Amazon ECS Agent, with some Remind/Empire specific patches applied.
Makefile
2
star
48

activerecord-pgbouncer

ActiveRecord connection adapter for using PgBouncer safely.
Ruby
2
star
49

cloudsns

SNS polling library for cloudformation events
Python
2
star
50

action-require-reviewer

Github workflow action to require a reviewer on pushed branches
TypeScript
1
star
51

metrics

Go library for printing metrics in an l2met compatible format.
Go
1
star
52

e164.rb

e164.js but ruby
Ruby
1
star
53

all_my_circuits

Mostly threadsafe implementation of the CircuitBreaker pattern for Ruby.
Ruby
1
star
54

migrate_safely

Adds confirmation prompt for rake db:migrate
Ruby
1
star
55

beso

Ruby
1
star
56

pooled-redis

Connection pooled Redis client that utilizes promises.
JavaScript
1
star
57

AutoGraphCodeGen

Swift GraphQL Code Generator
Swift
1
star
58

email-provider

Give it an email address, and get the email provider back.
Ruby
1
star