Terraform module for GitLab auto scaling runners on AWS spot instances

• The module
• Prerequisites
• Usage
• Examples
• Contributors ✨
• Requirements
• Providers
• Modules
• Resources
• Inputs
• Outputs

The module

This Terraform modules creates a GitLab CI runner. A blog post describes the original version of the runner. See the post at 040code. The original setup of the module is based on the blog post: Auto scale GitLab CI runners and save 90% on EC2 costs.

💥 BREAKING CHANGE AHEAD: Version 7 of the module rewrites the whole variable section to

• harmonize the variable names
• harmonize the documentation
• remove deprecated variables
• gain a better overview of the features provided

And it also adds

• all possible Docker settings
• the idle_scale_factor

We know that this is a breaking change causing some pain, but we think it is worth it. We hope you agree. And to make the transition as smooth as possible, we have added a migration script to the migrations folder. It will cover almost all cases, but some minor rework might still be possible.

Checkout issue 819

The runners created by the module use spot instances by default for running the builds using the docker+machine executor.

• Shared cache in S3 with life cycle management to clear objects after x days.
• Logs streamed to CloudWatch.
• Runner agents registered automatically.

The name of the runner agent and runner is set with the overrides variable. Adding an agent runner name tag does not work.

# ...
overrides  = {
  name_sg                     = ""
  name_runner_agent_instance  = "Gitlab Runner Agent"
  name_docker_machine_runners = "Gitlab Runner Terraform"
  name_iam_objects = "gitlab-runner"
}

//this doesn't work
agent_tags = merge(local.my_tags, map("Name", "Gitlab Runner Agent"))

The runner supports 3 main scenarios:

GitLab CI docker-machine runner - one runner agent

In this scenario the runner agent is running on a single EC2 node and runners are created by docker machine using spot instances. Runners will scale automatically based on the configuration. The module creates a S3 cache by default, which is shared across runners (spot instances).

GitLab CI docker-machine runner - multiple runner agents

In this scenario the multiple runner agents can be created with different configuration by instantiating the module multiple times. Runners will scale automatically based on the configuration. The S3 cache can be shared across runners by managing the cache outside of the module.

GitLab Ci docker runner

In this scenario not docker machine is used but docker to schedule the builds. Builds will run on the same EC2 instance as the agent. No auto scaling is supported.

Prerequisites

Terraform

Ensure you have Terraform installed. The modules is based on Terraform 0.11, see .terraform-version for the used version. A handy tool to mange your Terraform version is tfenv.

On macOS it is simple to install tfenv using brew.

brew install tfenv

Next install a Terraform version.

tfenv install <version>

AWS

Ensure you have setup your AWS credentials. The module requires access to IAM, EC2, CloudWatch, S3 and SSM.

JQ & AWS CLI

In order to be able to destroy the module, you will need to run from a host with both jq and aws installed and accessible in the environment.

On macOS it is simple to install them using brew.

brew install jq awscli

Service linked roles

The GitLab runner EC2 instance requires the following service linked roles:

• AWSServiceRoleForAutoScaling
• AWSServiceRoleForEC2Spot

By default the EC2 instance is allowed to create the required roles, but this can be disabled by setting the option allow_iam_service_linked_role_creation to false. If disabled you must ensure the roles exist. You can create them manually or via Terraform.

resource "aws_iam_service_linked_role" "spot" {
  aws_service_name = "spot.amazonaws.com"
}

resource "aws_iam_service_linked_role" "autoscaling" {
  aws_service_name = "autoscaling.amazonaws.com"
}

KMS keys

If a KMS key is set via kms_key_id, make sure that you also give proper access to the key. Otherwise, you might get errors, e.g. the build cache can't be decrypted or logging via CloudWatch is not possible. For a CloudWatch example checkout kms-policy.json

GitLab runner token configuration

By default the runner is registered on initial deployment. In previous versions of this module this was a manual process. The manual process is still supported but will be removed in future releases. The runner token will be stored in the AWS SSM parameter store. See example for more details.

To register the runner automatically set the variable gitlab_runner_registration_config["registration_token"]. This token value can be found in your GitLab project, group, or global settings. For a generic runner you can find the token in the admin section. By default the runner will be locked to the target project, not run untagged. Below is an example of the configuration map.

gitlab_runner_registration_config = {
  registration_token = "<registration token>"
  tag_list           = "<your tags, comma separated>"
  description        = "<some description>"
  locked_to_project  = "true"
  run_untagged       = "false"
  maximum_timeout    = "3600"
  # ref_protected runner will only run on pipelines triggered on protected branches. Defaults to not_protected
  access_level       = "<not_protected OR ref_protected>"
}

The registration token can also be read in via SSM parameter store. If no registration token is passed in, the module will look up the token in the SSM parameter store at the location specified by secure_parameter_store_gitlab_runner_registration_token_name.

For migration to the new setup simply add the runner token to the parameter store. Once the runner is started it will lookup the required values via the parameter store. If the value is null a new runner will be registered and a new token created/stored.

# set the following variables, look up the variables in your Terraform config.
# see your Terraform variables to fill in the vars below.
aws-region=<${var.aws_region}>
token=<runner-token-see-your-gitlab-runner>
parameter-name=<${var.environment}>-<${var.secure_parameter_store_runner_token_key}>

aws ssm put-parameter --overwrite --type SecureString  --name "${parameter-name}" --value ${token} --region "${aws-region}"

Once you have created the parameter, you must remove the variable runners_token from your config. The next time your GitLab runner instance is created it will look up the token from the SSM parameter store.

Finally, the runner still supports the manual runner creation. No changes are required. Please keep in mind that this setup will be removed in future releases.

Auto Scaling Group

Scheduled scaling

When enable_schedule=true, the schedule_config variable can be used to scale the Auto Scaling group.

Scaling may be defined with one scale_out scheduled action and/or one scale_in scheduled action.

For example:

  module "runner" {
    # ...
    enable_schedule = true
    schedule_config = {
      # Configure optional scale_out scheduled action
      scale_out_recurrence = "0 8 * * 1-5"
      scale_out_count      = 1 # Default for min_size, desired_capacity and max_size
      # Override using: scale_out_min_size, scale_out_desired_capacity, scale_out_max_size

      # Configure optional scale_in scheduled action
      scale_in_recurrence  = "0 18 * * 1-5"
      scale_in_count       = 0 # Default for min_size, desired_capacity and max_size
      # Override using: scale_out_min_size, scale_out_desired_capacity, scale_out_max_size
    }
  }

Instance Termination

The Auto Scaling Group may be configured with a lifecycle hook that executes a provided Lambda function when the runner is terminated to terminate additional instances that were spawned.

The use of the termination lifecycle can be toggled using the asg_termination_lifecycle_hook_create variable.

When using this feature, a builds/ directory relative to the root module will persist that contains the packaged Lambda function.

Access runner instance

A few option are provided to access the runner instance:

1. Access via the Session Manager (SSM) by setting enable_runner_ssm_access to true. The policy to allow access via SSM is not very restrictive.
2. By setting none of the above, no keys or extra policies will be attached to the instance. You can still configure you own policies by attaching them to runner_agent_role_arn.

GitLab runner cache

By default the module creates a cache for the runner in S3. Old objects are automatically removed via a configurable life cycle policy on the bucket.

Creation of the bucket can be disabled and managed outside this module. A good use case is for sharing the cache across multiple runners. For this purpose the cache is implemented as a sub module. For more details see the cache module. An example implementation of this use case can be found in the runner-public example.

In case you enable the access logging for the S3 cache bucket, you have to add the following statement to your S3 logging bucket policy.

{
    "Sid": "Allow access logging",
    "Effect": "Allow",
    "Principal": {
        "Service": "logging.s3.amazonaws.com"
    },
    "Action": "s3:PutObject",
    "Resource": "<s3-arn>/*"
}

In case you manage the S3 cache bucket yourself it might be necessary to apply the cache before applying the runner module. A typical error message looks like:

Error: Invalid count argument
on .terraform/modules/gitlab_runner/main.tf line 400, in resource "aws_iam_role_policy_attachment" "docker_machine_cache_instance":
  count = var.cache_bucket["create"] || length(lookup(var.cache_bucket, "policy", "")) > 0 ? 1 : 0
The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many
instances will be created. To work around this, use the -target argument to first apply only the resources that the count
depends on.

The workaround is to use a terraform apply -target=module.cache followed by a terraform apply to apply everything else. This is a one time effort needed at the very beginning.

Usage

Configuration

Update the variables in terraform.tfvars according to your needs and add the following variables. See the previous step for instructions on how to obtain the token.

runner_name  = "NAME_OF_YOUR_RUNNER"
gitlab_url   = "GITLAB_URL"
runner_token = "RUNNER_TOKEN"

The base image used to host the GitLab Runner agent is the latest available Amazon Linux 2 HVM EBS AMI. In previous versions of this module a hard coded list of AMIs per region was provided. This list has been replaced by a search filter to find the latest AMI. Setting the filter to amzn2-ami-hvm-2.0.20200207.1-x86_64-ebs will allow you to version lock the target AMI.

Scenario: Basic usage

Below is a basic examples of usages of the module. Regarding the dependencies such as a VPC, have a look at the default example.

module "runner" {
  # https://registry.terraform.io/modules/cattle-ops/gitlab-runner/aws/
  source  = "cattle-ops/gitlab-runner/aws"

  aws_region  = "eu-west-1"
  environment = "spot-runners"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids_gitlab_runner = module.vpc.private_subnets
  subnet_id_runners        = element(module.vpc.private_subnets, 0)

  runners_name       = "docker-default"
  runners_gitlab_url = "https://gitlab.com"

  gitlab_runner_registration_config = {
    registration_token = "my-token"
    tag_list           = "docker"
    description        = "runner default"
    locked_to_project  = "true"
    run_untagged       = "false"
    maximum_timeout    = "3600"
  }

}

Removing the module

As the module creates a number of resources during runtime (key pairs and spot instance requests), it needs a special procedure to remove them.

1. Use the AWS Console to set the desired capacity of all auto scaling groups to 0. To find the correct ones use the var.environment as search criteria. Setting the desired capacity to 0 prevents AWS from creating new instances which will in turn create new resources.
2. Kill all agent ec2 instances on the via AWS Console. This triggers a Lambda function in the background which removes all resources created during runtime of the EC2 instances.
3. Wait 3 minutes so the Lambda function has enough time to delete the key pairs and spot instance requests.
4. Run a terraform destroy or terraform apply (depends on your setup) to remove the module.

If you don't follow the above procedure key pairs and spot instance requests might survive the removal and might cause additional costs. But I have never seen that. You should also be fine by executing step 4 only.

Scenario: Multi-region deployment

Name clashes due to multi-region deployments for global AWS resources create by this module (IAM, S3) can be avoided by including a distinguishing region specific prefix via the cache_bucket_prefix string respectively via name_iam_objects in the overrides map. A simple example for this would be to set region-specific-prefix to the AWS region the module is deployed to.

module "runner" {
  # https://registry.terraform.io/modules/cattle-ops/gitlab-runner/aws/
  source  = "cattle-ops/gitlab-runner/aws"

  aws_region  = "eu-west-1"
  environment = "spot-runners"

  vpc_id                   = module.vpc.vpc_id
  subnet_ids_gitlab_runner = module.vpc.private_subnets
  subnet_id_runners        = element(module.vpc.private_subnets, 0)

  runners_name       = "docker-default"
  runners_gitlab_url = "https://gitlab.com"

  gitlab_runner_registration_config = {
    registration_token = "my-token"
    tag_list           = "docker"
    description        = "runner default"
    locked_to_project  = "true"
    run_untagged       = "false"
    maximum_timeout    = "3600"
  }

  overrides = {
    name_iam_objects = "<region-specific-prefix>-gitlab-runner-iam"
  }

  cache_bucket_prefix = "<region-specific-prefix>"
}

Scenario: Use of Spot Fleet

Since spot instances can be taken over by AWS depending on the instance type and AZ you are using, you may want multiple instances types in multiple AZs. This is where spot fleets come in, when there is no capacity on one instance type and one AZ, AWS will take the next instance type and so on. This update has been possible since the fork of docker-machine supports spot fleets.

We have seen that the fork of docker-machine this module is using consume more RAM using spot fleets. For comparison, if you launch 50 machines in the same time, it consumes ~1.2GB of RAM. In our case, we had to change the instance_type of the runner from t3.micro to t3.small.

Configuration example

module "runner" {
  # https://registry.terraform.io/modules/npalm/gitlab-runner/aws/
  source  = "npalm/gitlab-runner/aws"

  aws_region  = "eu-west-3"
  environment = "spot-runners"

  vpc_id                    = module.vpc.vpc_id
  subnet_id                 = module.vpc.private_subnets[0] # subnet of the agent
  fleet_executor_subnet_ids = module.vpc.private_subnets

  docker_machine_instance_types_fleet       = ["t3a.medium", "t3.medium", "t2.medium"]
  use_fleet                                 = true
  fleet_key_pair_name                       = "<key_pair_name>"

  runners_name       = "docker-machine"
  runners_gitlab_url = "https://gitlab.com"

  gitlab_runner_registration_config = {
    registration_token = "my-token"
    tag_list           = "docker"
    description        = "runner default"
    locked_to_project  = "true"
    run_untagged       = "false"
    maximum_timeout    = "3600"
  }

  overrides = {
    name_iam_objects = "<region-specific-prefix>-gitlab-runner-iam"
  }
}

Examples

A few examples are provided. Use the following steps to deploy. Ensure your AWS and Terraform environment is set up correctly. All commands below should be run from the terraform-aws-gitlab-runner/examples/<example-dir> directory. Don't forget to remove the runners manually from your Gitlab instance as soon as your are done.

Versions

The version of Terraform is locked down via tfenv, see the .terraform-version file for the expected versions. Providers are locked down as well in the providers.tf file.

Configure

The examples are configured with defaults that should work in general. The examples are in general configured for the region Ireland eu-west-1. The only parameter that needs to be provided is the GitLab registration token. The token can be found in GitLab in the runner section (global, group or repo scope). Create a file terraform.tfvars and the registration token.

    registration_token = "MY_TOKEN"

Run

Run terraform init to initialize Terraform. Next you can run terraform plan to inspect the resources that will be created.

To create the runner, run:

  terraform apply

To destroy the runner, run:

  terraform destroy

Contributors ✨

This project exists thanks to all the people who contribute.

Made with contributors-img.

Module Documentation

Requirements

Providers

Modules

Resources

Inputs

object({
    trace_runner_user_data         = optional(bool, false)
    write_runner_config_to_file    = optional(bool, false)
    write_runner_user_data_to_file = optional(bool, false)
  })

{
  "name": [
    "amzn2-ami-hvm-2.*-x86_64-ebs"
  ]
}

[
  "amazon"
]

object({
    enable         = optional(bool, true)
    log_group_name = optional(string, null)
    retention_days = optional(number, 0)
  })

object({
    ca_certificate     = optional(string, "")
    certificate        = optional(string, "")
    registration_token = optional(string, "REPLACED_BY_USER_DATA")
    runner_version     = optional(string, "15.8.2")
    url                = optional(string, "")
    url_clone          = optional(string, "")
  })

object({
    registration_token = optional(string, "")
    tag_list           = optional(string, "")
    description        = optional(string, "")
    locked_to_project  = optional(string, "")
    run_untagged       = optional(string, "")
    maximum_timeout    = optional(string, "")
    access_level       = optional(string, "not_protected") # this is the only mandatory field calling the GitLab get token for executor operation
  })

object({
    amazon_ecr_credential_helper = optional(bool, false)
    docker_machine_download_url  = optional(string, "")
    docker_machine_version       = optional(string, "0.16.2-gitlab.19-cki.2")
    pre_install_script           = optional(string, "")
    post_install_script          = optional(string, "")
    start_script                 = optional(string, "")
    yum_update                   = optional(bool, true)
  })

object({
    additional_tags             = optional(map(string))
    collect_autoscaling_metrics = optional(list(string), null)
    ebs_optimized               = optional(bool, true)
    max_lifetime_seconds        = optional(number, null)
    monitoring                  = optional(bool, true)
    name                        = string
    name_prefix                 = optional(string)
    private_address_only        = optional(bool, true)
    root_device_config          = optional(map(string), {})
    spot_price                  = optional(string, null)
    ssm_access                  = optional(bool, false)
    type                        = optional(string, "t3.micro")
    use_eip                     = optional(bool, false)
  })

{
  "name": "gitlab-runner"
}

object({
    gitlab_check_interval     = optional(number, 3)
    maximum_concurrent_jobs   = optional(number, 10)
    prometheus_listen_address = optional(string, "")
    sentry_dsn                = optional(string, "SENTRY_DSN_REPLACED_BY_USER_DATA")
  })

object({
    http_endpoint               = string
    http_tokens                 = string
    http_put_response_hop_limit = number
    instance_metadata_tags      = string
  })

{
  "http_endpoint": "enabled",
  "http_put_response_hop_limit": 2,
  "http_tokens": "required",
  "instance_metadata_tags": "disabled"
}

object({
    allow_incoming_ping                    = optional(bool, false)
    allow_incoming_ping_security_group_ids = optional(list(string), [])
    security_group_description             = optional(string, "A security group containing gitlab-runner agent instances")
    security_group_ids                     = optional(list(string), [])
  })

list(object({
    cidr_blocks      = list(string)
    ipv6_cidr_blocks = list(string)
    prefix_list_ids  = list(string)
    from_port        = number
    protocol         = string
    security_groups  = list(string)
    self             = bool
    to_port          = number
    description      = string
  }))

[
  {
    "cidr_blocks": [
      "0.0.0.0/0"
    ],
    "description": null,
    "from_port": 0,
    "ipv6_cidr_blocks": [
      "::/0"
    ],
    "prefix_list_ids": null,
    "protocol": "-1",
    "security_groups": null,
    "self": null,
    "to_port": 0
  }
]

object({
    additional_tags                        = optional(map(string))
    allow_iam_service_linked_role_creation = optional(bool, true)
    assume_role_policy_json                = optional(string, "")
    create_role_profile                    = optional(bool, true)
    policy_arns                            = optional(list(string), [])
    role_profile_name                      = optional(string)
  })

{
  "scale_in_count": 0,
  "scale_in_recurrence": "0 18 * * 1-5",
  "scale_in_time_zone": "Etc/UTC",
  "scale_out_count": 1,
  "scale_out_recurrence": "0 8 * * 1-5",
  "scale_out_time_zone": "Etc/UTC"
}

object({
    environment_variables = optional(list(string), [])
    max_jobs              = optional(number, 0)
    output_limit          = optional(number, 4096)
    request_concurrency   = optional(number, 1)
    ssm_access            = optional(bool, false)
    type                  = optional(string, "docker+machine")
  })

object({
    access_log_bucket_id     = optional(string, null)
    access_log_bucket_prefix = optional(string, null)
    authentication_type      = optional(string, "iam")
    bucket                   = optional(string, "")
    bucket_prefix            = optional(string, "")
    create                   = optional(bool, true)
    expiration_days          = optional(number, 1)
    include_account_id       = optional(bool, true)
    policy                   = optional(string, "")
    random_suffix            = optional(bool, false)
    shared                   = optional(bool, false)
    versioning               = optional(bool, false)
  })

{
  "name": [
    "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
  ]
}

[
  "099720109477"
]

list(object({
    periods           = list(string)
    idle_count        = optional(number)
    idle_scale_factor = optional(number)
    idle_count_min    = optional(number)
    idle_time         = optional(number)
    timezone          = optional(string, "UTC")
  }))

object({
    http_tokens                 = string
    http_put_response_hop_limit = number
  })

{
  "http_put_response_hop_limit": 2,
  "http_tokens": "required"
}

list(object({
    cidr_blocks      = list(string)
    ipv6_cidr_blocks = list(string)
    prefix_list_ids  = list(string)
    from_port        = number
    protocol         = string
    security_groups  = list(string)
    self             = bool
    to_port          = number
    description      = string
  }))

[
  {
    "cidr_blocks": [
      "0.0.0.0/0"
    ],
    "description": "Allow all egress traffic for Runner Workers.",
    "from_port": 0,
    "ipv6_cidr_blocks": [
      "::/0"
    ],
    "prefix_list_ids": null,
    "protocol": "-1",
    "security_groups": null,
    "self": null,
    "to_port": 0
  }
]

object({
    enable        = bool
    key_pair_name = optional(string, "fleet-key")
  })

{
  "enable": false
}

object({
    destroy_after_max_builds   = optional(number, 0)
    docker_registry_mirror_url = optional(string, "")
    ebs_optimized              = optional(bool, true)
    idle_count                 = optional(number, 0)
    idle_time                  = optional(number, 600)
    monitoring                 = optional(bool, false)
    name_prefix                = optional(string, "")
    private_address_only       = optional(bool, true)
    root_size                  = optional(number, 8)
    start_script               = optional(string, "")
    subnet_ids                 = optional(list(string), [])
    types                      = optional(list(string), ["m5.large"])
    volume_type                = optional(string, "gp2")
  })

object({
    enable    = optional(bool, true)
    max_price = optional(string, "on-demand-price")
  })

object({
    additional_tags         = optional(map(string), {})
    assume_role_policy_json = optional(string, "")
    policy_arns             = optional(list(string), [])
    profile_name            = optional(string, "")
  })

object({
    allowed_images               = optional(list(string))
    allowed_pull_policies        = optional(list(string))
    allowed_services             = optional(list(string))
    cache_dir                    = optional(string)
    cap_add                      = optional(list(string))
    cap_drop                     = optional(list(string))
    container_labels             = optional(list(string))
    cpuset_cpus                  = optional(string)
    cpu_shares                   = optional(number)
    cpus                         = optional(string)
    devices                      = optional(list(string))
    device_cgroup_rules          = optional(list(string))
    disable_cache                = optional(bool, false)
    disable_entrypoint_overwrite = optional(bool)
    dns                          = optional(list(string))
    dns_search                   = optional(list(string))
    extra_hosts                  = optional(list(string))
    gpus                         = optional(string)
    helper_image                 = optional(string)
    helper_image_flavor          = optional(string)
    host                         = optional(string)
    hostname                     = optional(string)
    image                        = optional(string, "docker:18.03.1-ce")
    isolation                    = optional(string)
    links                        = optional(list(string))
    mac_address                  = optional(string)
    memory                       = optional(string)
    memory_swap                  = optional(string)
    memory_reservation           = optional(string)
    network_mode                 = optional(string)
    oom_kill_disable             = optional(bool)
    oom_score_adjust             = optional(number)
    privileged                   = optional(bool, true)
    pull_policies                = optional(list(string), ["always"])
    runtime                      = optional(string)
    security_opt                 = optional(list(string))
    shm_size                     = optional(number, 0)
    sysctls                      = optional(list(string))
    tls_cert_path                = optional(string)
    tls_verify                   = optional(bool, false)
    user                         = optional(string)
    userns_mode                  = optional(string)
    volumes                      = optional(list(string), ["/cache"])
    volumes_from                 = optional(list(string))
    volume_driver                = optional(string)
    wait_for_services_timeout    = optional(number)
  })

{
  "disable_cache": "false",
  "image": "docker:18.03.1-ce",
  "privileged": "true",
  "pull_policy": "always",
  "shm_size": 0,
  "tls_verify": "false",
  "volumes": [
    "/cache"
  ]
}

list(object({
    name       = string
    alias      = string
    entrypoint = list(string)
    command    = list(string)
  }))

list(object({
    volume  = string
    options = string
  }))

list(object({
    volume  = string
    options = string
  }))

object({
    post_build_script = optional(string, """")
    pre_build_script  = optional(string, """")
    pre_clone_script  = optional(string, """")
  })

Outputs