Use Docker to run your Golang integration tests against third party services on Microsoft Windows, Mac OSX and Linux!

Table of Contents

• Why should I use Dockertest?
• Installing and using Dockertest
  • Using Dockertest
  • Examples
• Troubleshoot & FAQ
  • Out of disk space
  • Removing old containers

When developing applications, it is often necessary to use services that talk to a database system. Unit Testing these services can be cumbersome because mocking database/DBAL is strenuous. Making slight changes to the schema implies rewriting at least some, if not all of the mocks. The same goes for API changes in the DBAL. To avoid this, it is smarter to test these specific services against a real database that is destroyed after testing. Docker is the perfect system for running unit tests as you can spin up containers in a few seconds and kill them when the test completes. The Dockertest library provides easy to use commands for spinning up Docker containers and using them for your tests.

Using Dockertest is straightforward and simple. Check the releases tab for available releases.

To install dockertest, run

go get -u github.com/ory/dockertest/v3

or

dep ensure -add github.com/ory/[email protected]

package dockertest_test

import (
	"database/sql"
	"fmt"
	"log"
	"os"
	"testing"

	_ "github.com/go-sql-driver/mysql"
	"github.com/ory/dockertest/v3"
)

var db *sql.DB

func TestMain(m *testing.M) {
	// uses a sensible default on windows (tcp/http) and linux/osx (socket)
	pool, err := dockertest.NewPool("")
	if err != nil {
		log.Fatalf("Could not construct pool: %s", err)
	}

	// uses pool to try to connect to Docker
	err = pool.Client.Ping()
	if err != nil {
		log.Fatalf("Could not connect to Docker: %s", err)
	}

	// pulls an image, creates a container based on it and runs it
	resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
	if err != nil {
		log.Fatalf("Could not start resource: %s", err)
	}

	// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
	if err := pool.Retry(func() error {
		var err error
		db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
		if err != nil {
			return err
		}
		return db.Ping()
	}); err != nil {
		log.Fatalf("Could not connect to database: %s", err)
	}

	code := m.Run()

	// You can't defer this because os.Exit doesn't care for defer
	if err := pool.Purge(resource); err != nil {
		log.Fatalf("Could not purge resource: %s", err)
	}

	os.Exit(code)
}

func TestSomething(t *testing.T) {
	// db.Query()
}

We provide code examples for well known services in the examples directory, check them out!

Try cleaning up the images with docker-cleanup-volumes.

Sometimes container clean up fails. Check out this stackoverflow question on how to fix this. You may also set an absolute lifetime on containers:

resource.Expire(60) // Tell docker to hard kill the container in 60 seconds

To let stopped containers removed from file system automatically, use pool.RunWithOptions() instead of pool.Run() with config.AutoRemove set to true, e.g.:

postgres, err := pool.RunWithOptions(&dockertest.RunOptions{
	Repository: "postgres",
	Tag:        "11",
	Env: []string{
		"POSTGRES_USER=test",
		"POSTGRES_PASSWORD=test",
		"listen_addresses = '*'",
	},
}, func(config *docker.HostConfig) {
	// set AutoRemove to true so that stopped container goes away by itself
	config.AutoRemove = true
	config.RestartPolicy = docker.RestartPolicy{
		Name: "no",
	}
})

You should add docker dind service to your job which starts in sibling container. That means database will be available on host docker.
You app should be able to change db host through environment variable.

Here is the simple example of gitlab-ci.yml:

stages:
  - test
go-test:
  stage: test
  image: golang:1.15
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_DRIVER: overlay2
    YOUR_APP_DB_HOST: docker
  script:
    - go test ./...

Plus in the pool.Retry method that checks for connection readiness, you need to use $YOUR_APP_DB_HOST instead of localhost.

Gitlab runner can be run in docker executor mode to save compatibility with shared runners.
Here is the simple register command:

gitlab-runner register -n \
 --url https://gitlab.com/ \
 --registration-token $YOUR_TOKEN \
 --executor docker \
 --description "My Docker Runner" \
 --docker-image "docker:19.03.12" \
 --docker-privileged

You only need to instruct docker dind to start with disabled tls.
Add variable DOCKER_TLS_CERTDIR: "" to gitlab-ci.yml above. It will tell docker daemon to start on 2375 port over http.

name: Test with Docker

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      dind:
        image: docker:23.0-rc-dind-rootless
        ports:
          - 2375:2375
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: "1.18"

      - name: Test with Docker
        run: go test -v ./...

Use-case: locally installed docker CLI (client), docker daemon somewhere remotely, environment properly set (ie: DOCKER_HOST, etc..). For example, remote docker can be provisioned by docker-machine.

Currently, dockertest in case of resource.GetHostPort() will return docker host binding address (commonly - localhost) instead of remote docker host. Universal solution is:

func getHostPort(resource *dockertest.Resource, id string) string {
	dockerURL := os.Getenv("DOCKER_HOST")
	if dockerURL == "" {
		return resource.GetHostPort(id)
	}
	u, err := url.Parse(dockerURL)
	if err != nil {
		panic(err)
	}
	return u.Hostname() + ":" + resource.GetPort(id)
}

It will return the remote docker host concatenated with the allocated port in case DOCKER_HOST env is defined. Otherwise, it will fall back to the embedded behavior.