mvisonneau/gitlab-ci-pipelines-exporter mvisonneau/gitlab-ci-pipelines-exporter

  • Stars
    star
    1,263
  • Rank 37,244 (Top 0.8 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created about 6 years ago
  • Updated 3 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
mvisonneau/gitlab-ci-pipelines-exporter
github

mvisonneau

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights

🦊 gitlab-ci-pipelines-exporter

PkgGoDev Go Report Card Docker Pulls test Coverage Status release gitlab-ci-pipelines-exporter

gitlab-ci-pipelines-exporter allows you to monitor your GitLab CI pipelines with Prometheus or any monitoring solution supporting the OpenMetrics format.

You can find more information on GitLab docs about how it takes part improving your pipeline efficiency.

TL:DR

Here are some Grafana dashboards I was able to craft using those metrics. Otherwise, the detailed list of exported metrics is maintained here.

Pipelines

grafana_dashboard_pipelines

grafana.com dashboard #10620

Jobs

grafana_dashboard_jobs

grafana.com dashboard #13328

Environments / Deployments

grafana_dashboard_environments

grafana.com dashboard #13329

If you want to quickly try them out with your own data, have a look into the examples/quickstart folder which contains documentation to provision test version of the exporter, prometheus and also grafana in ~5min using docker-compose

Install

Go

~$ go run github.com/mvisonneau/gitlab-ci-pipelines-exporter/cmd/gitlab-ci-pipelines-exporter@latest

Snapcraft

~$ snap install gitlab-ci-pipelines-exporter

Homebrew

~$ brew install mvisonneau/tap/gitlab-ci-pipelines-exporter

Docker

~$ docker run -it --rm docker.io/mvisonneau/gitlab-ci-pipelines-exporter
~$ docker run -it --rm ghcr.io/mvisonneau/gitlab-ci-pipelines-exporter
~$ docker run -it --rm quay.io/mvisonneau/gitlab-ci-pipelines-exporter

Scoop

~$ scoop bucket add https://github.com/mvisonneau/scoops
~$ scoop install gitlab-ci-pipelines-exporter

NixOS

~$ nix-env -iA nixos.prometheus-gitlab-ci-pipelines-exporter

Binaries, DEB and RPM packages

Have a look onto the latest release page to pick your flavor and version. Here is an helper to fetch the most recent one:

~$ export GCPE_VERSION=$(curl -s "https://api.github.com/repos/mvisonneau/gitlab-ci-pipelines-exporter/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
# Binary (eg: linux/amd64)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_amd64.tar.gz
~$ tar zxvf gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_amd64.tar.gz -C /usr/local/bin

# DEB package (eg: linux/386)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_386.deb
~$ dpkg -i gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_386.deb

# RPM package (eg: linux/arm64)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_arm64.rpm
~$ rpm -ivh gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_arm64.rpm

HELM

If you want to make it run on kubernetes, there is a helm chart available for this purpose.

You can check the chart's values.yml for complete configuration options.

# Add the helm repository to your local client
~$ helm repo add mvisonneau https://charts.visonneau.fr

# Configure a minimal configuration for the exporter
~$ cat <<EOF > values.yml
config:
  gitlab:
    url: https://gitlab.example.com
    # You can also configure the token using --gitlab-token
    # or the $GCPE_GITLAB_TOKEN environment variable
    token: xrN14n9-ywvAFxxxxxx
  projects:
    - name: foo/project
EOF

# Release the chart on your Kubernetes cluster
~$ helm upgrade -i gitlab-ci-pipelines-exporter mvisonneau/gitlab-ci-pipelines-exporter -f values.yml

Configuration syntax

The configuration syntax is maintained here.

Quickstart

# Write a minimal config file somewhere on disk
~$ cat <<EOF > $(pwd)/config.yml
gitlab:
  url: https://gitlab.example.com
  # You can also configure the token using --gitlab-token
  # or the $GCPE_GITLAB_TOKEN environment variable
  token: <your_token>
projects:
  - name: foo/project
  - name: bar/project
wildcards:
  - owner:
      name: foo
      kind: group
EOF

# If you have installed the binary
~$ gitlab-ci-pipelines-exporter --config /etc/config.yml

# Otherwise if you have docker available, it is as easy as :
~$ docker run -it --rm \
   --name gitlab-ci-pipelines-exporter \
   -v $(pwd)/config.yml:/etc/config.yml \
   -p 8080:8080 \
   mvisonneau/gitlab-ci-pipelines-exporter:latest \
   run --config /etc/config.yml

You should then be able to see the following logs

INFO[0000] starting exporter                             gitlab-endpoint="https://gitlab.com" on-init-fetch-refs-from-pipelines=true pulling-pipelines-every=60s pulling-projects-every=15s pulling-refs-every=10s pulling-workers=2 rate-limit=10rps
INFO[0000] configured wildcards                          count=1
INFO[0000] found new project                             project-name=foo/project wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind=group wildcard-owner-name=foo wildcard-search=
INFO[0000] found new project                             project-name=foo/bar wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind=group wildcard-owner-name=foo wildcard-search=
INFO[0000] configured projects                           count=3
INFO[0000] started, now serving requests                 listen-address=":8080"
INFO[0000] found project refs                            project-path-with-namespace=foo/project project-ref=main
INFO[0000] found project refs                            project-path-with-namespace=bar/project project-ref=main
INFO[0000] found project refs                            project-path-with-namespace=foo/bar project-ref=main

And this is an example of the metrics you should expect to retrieve, the detailed list of exported metrics is maintained here.

~$ curl -s localhost:8080/metrics | grep gitlab_ci
# HELP gitlab_ci_pipeline_last_run_duration_seconds Duration of last pipeline run
# TYPE gitlab_ci_pipeline_last_run_duration_seconds gauge
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/project",ref="dev",topics="",variables=""} 81
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/project",ref="main",topics="",variables=""} 420
gitlab_ci_pipeline_last_run_duration_seconds{project="bar/project",ref="main",topics="",variables=""} 334
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 55
# HELP gitlab_ci_pipeline_last_run_id ID of the most recent pipeline
# TYPE gitlab_ci_pipeline_last_run_id gauge
gitlab_ci_pipeline_last_run_id{project="foo/project",ref="dev",topics="",variables=""} 4.0059611e+07
gitlab_ci_pipeline_last_run_id{project="foo/project",ref="main",topics="",variables=""} 1.25351545e+08
gitlab_ci_pipeline_last_run_id{project="bar/project",ref="main",topics="",variables=""} 1.33308085e+08
gitlab_ci_pipeline_last_run_id{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 1.40420947e+08
# HELP gitlab_ci_pipeline_last_run_status Status of the most recent pipeline
# TYPE gitlab_ci_pipeline_last_run_status gauge
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="failed",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="success",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="failed",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="main",status="success",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="failed",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="main",status="success",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="canceled",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="failed",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="manual",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="pending",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="running",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="skipped",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="main",status="success",topics="",variables="FOO:BAR"} 1
# HELP gitlab_ci_pipeline_run_count GitLab CI pipeline run count
# TYPE gitlab_ci_pipeline_run_count counter
gitlab_ci_pipeline_run_count{project="foo/project",ref="dev",topics="",variables=""} 1
gitlab_ci_pipeline_run_count{project="foo/project",ref="main",topics="",variables=""} 2
gitlab_ci_pipeline_run_count{project="bar/project",ref="main",topics="",variables=""} 1
gitlab_ci_pipeline_run_count{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 2
# HELP gitlab_ci_pipeline_time_since_last_run_seconds Elapsed time since most recent GitLab CI pipeline run.
# TYPE gitlab_ci_pipeline_time_since_last_run_seconds gauge
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/project",ref="dev",topics="",variables=""} 4.3368877e+07
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/project",ref="main",topics="",variables=""} 4.151883e+06
gitlab_ci_pipeline_time_since_last_run_seconds{project="bar/project",ref="main",topics="",variables=""} 1.907042e+06
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/bar",ref="main",topics="",variables="FOO:BAR"} 65456

HA implementation

It supports running multiple instances of the exporter in an HA fashion leveraging redis as storage middleware. You simply need to set a redis URL in the config.yml or using the --redis-url flag or $GCPE_REDIS_URL env variable. A quick example using docker-compose is also available here: examples/ha-setup

How it works

  • Pulling of all of the GitLab resources (projects, refs, pipelines, jobs, etc..) is spread evenly across all the running instances
  • Rate limit is global across the workers. eg: 3 workers at a 10 rps limit will result in a ~3.3rps limit/worker
  • Exported metrics are fetched from the shared storage layer on each call to ensure data integrity/consistency of the requests across the instances

Push based implementation (leveraging GitLab webhooks)

The exporter supports receiving project pipeline events through GitLab webhooks on the /webhook path. This feature is not enabled by default and requires the following parameters to be set in the config.yml:

server:
   webhook:
      enabled: true
      secret_token: <a_secret_token>

A complete example is available here: examples/webhooks. You can also refer to the configuration syntax for me information.

Usage

~$ gitlab-ci-pipelines-exporter --help
NAME:
   gitlab-ci-pipelines-exporter - Export metrics about GitLab CI pipelines statuses

USAGE:
   gitlab-ci-pipelines-exporter [global options] command [command options] [arguments...]

COMMANDS:
   run      start the exporter
   monitor  display information about the currently running exporter
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --internal-monitoring-listener-address value, -m value  internal monitoring listener address [$GCPE_INTERNAL_MONITORING_LISTENER_ADDRESS]
   --help, -h                                              show help (default: false)
   --version, -v                                           print the version (default: false)

run

~$ gitlab-ci-pipelines-exporter run --help
NAME:
   gitlab-ci-pipelines-exporter run - start the exporter

USAGE:
   gitlab-ci-pipelines-exporter run [command options] [arguments...]

OPTIONS:
   --config file, -c file        config file (default: "./gitlab-ci-pipelines-exporter.yml") [$GCPE_CONFIG]
   --redis-url url               redis url for an HA setup (format: redis[s]://[:password@]host[:port][/db-number][?option=value]) (overrides config file parameter) [$GCPE_REDIS_URL]
   --gitlab-token token          GitLab API access token (overrides config file parameter) [$GCPE_GITLAB_TOKEN]
   --webhook-secret-token token  token used to authenticate legitimate requests (overrides config file parameter) [$GCPE_WEBHOOK_SECRET_TOKEN]
   --help, -h                    show help (default: false)

monitor

~$ gitlab-ci-pipelines-exporter monitor --help
NAME:
   gitlab-ci-pipelines-exporter monitor - display information about the currently running exporter

USAGE:
   gitlab-ci-pipelines-exporter monitor [command options] [arguments...]

OPTIONS:
   --help, -h  show help (default: false)

Monitor / Troubleshoot

monitor_cli_example

If you need to dig into your exporter's internal, you can leverage the internal CLI monitoring endpoint. This will get you insights about the following:

  • Live telemetry regarding:
    • GitLab API requests
    • Tasks buffer usage
    • Projects count and schedules
    • Environments count and schedules
    • Refs count and schedules
    • Metrics count and schedules
  • Parsed configuration details

To use it, you have to start your exporter with the following flag --internal-monitoring-listener-address, -m or the GCPE_INTERNAL_MONITORING_LISTENER_ADDRESS env variable.

You can whether use a TCP or UNIX socket eg:

~$ gitlab-ci-pipelines-exporter -m 'unix://gcpe-monitor.sock' run
~$ gitlab-ci-pipelines-exporter -m 'tcp://127.0.0.1:9000' run

To use the monitor CLI, you need to be able to access the monitoring socket and reuse the same flag:

export GCPE_INTERNAL_MONITORING_LISTENER_ADDRESS='unix://gcpe-monitor.sock'
~$ gitlab-ci-pipelines-exporter run &
~$ gitlab-ci-pipelines-exporter monitor

Develop / Test

If you use docker, you can easily get started using :

~$ make dev-env
# You should then be able to use go commands to work onto the project, eg:
~docker$ make fmt
~docker$ gitlab-ci-pipelines-exporter

Contribute

Contributions are more than welcome! Feel free to submit a PR.

More Repositories

1

docker-tailscale

Run a Tailscale agent/relay in a container
Shell
86
star
2

docker-generatedata

Docker packaged version of the benkeen/generatedata project
Dockerfile
79
star
3

helm-charts

Personal Helm charts repository
Mustache
67
star
4

strongbox

Securely store secrets at rest using Hashicorp Vault
Go
56
star
5

tfcw

Terraform Cloud Wrapper
Go
17
star
6

terraform-provider-updown

Terraform provider for updown.io
Go
17
star
7

docker-yubikey-validation-server

Dockerized stack of Yubico yubikey-ksm and yubikey-val
16
star
8

s5

Safely Store Super Sensitive Stuff
Go
13
star
9

vac

AWS credentials management leveraging Vault
Go
12
star
10

docker-registry-gitlab-cleanup

Small hack to cleanup old docker images in a GitLab embedded docker registry
Python
9
star
11

go-ebsnvme

Golang version of the AWS ebsnvme-id script
Go
8
star
12

approuvez

command line helper to obtain live confirmation from relevant people
Go
8
star
13

puppet-stack-terraform

Provision a Puppet stack on AWS using Terraform
HCL
7
star
14

gitlab-merger

Automated merge request creation for GitLab projects
Go
5
star
15

lambda-sftp2S3

Lamda function that transfer files from an SFTP server onto an S3 bucket
Python
5
star
16

automount

Go binary that allows you to mount hard drives on a Linux host
Go
5
star
17

puppetconf2016-tf-ecs

Codebase for the demo I did during my talk at PuppetConf 2016 around scaling puppet on top of AWS using Terraform and ECS
HCL
5
star
18

slack-git-compare

Compare git references within Slack
Go
4
star
19

ahs

AWS EC2 Hostname Setter
Go
4
star
20

ocalver

Opinionated CalVer generator
Go
3
star
21

puppet-centreon

Management of the Centreon provisioning with Puppet
Ruby
3
star
22

docker-goldfish

Docker container of the open source Vault UI project : Goldfish
Shell
2
star
23

gpcd

GoPro Cloud Downloader
Go
2
star
24

puppet-openswan

Provides the installation and configuration of an ipsec over xl2tp VPN server with Puppet
Puppet
1
star
25

atom-s5

Atom wrapper for S5
JavaScript
1
star
26

puppet-curator

Installs, configures, and manages Elasticsearch Curator.
Ruby
1
star