• Stars
    star
    133
  • Rank 272,600 (Top 6 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 7 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Executes Terraform configuration as job/pod inside a Kubernetes cluster.

Terraformer

CI Build status

Terraformer is a tool that can execute Terraform commands (apply, destroy and validate) and can be run as a Pod inside a Kubernetes cluster. The Terraform configuration and state files (main.tf, variables.tf, terraform.tfvars and terraform.tfstate) are stored as ConfigMaps and Secrets in the Kubernetes cluster and will be retrieved and updated by Terraformer.

For talking to the API server, Terraformer will either use the service account of its Pod or the provided kubeconfig (via the --kubeconfig flag). Names of the ConfigMaps and Secrets have to be provided via command line flags. The namespace of the objects can be specified via --namespace or will be defaulted to the Pod's namespace.

For more details and example Kubernetes manifests see the example directory.

⚠️ Running Terraformer as a Job

Usually, terraformer apply|destroy|validate runs within a single Pod. Please note, that running Terraformer as a Job is not recommended. The Job object will start a new Pod if the first Pod fails or is deleted (for example due to a Node hardware failure or a reboot). Thus, you may end up in a situation with two running Terraformer Pods at the same time which can fail with conflicts.

State file watcher + update worker

While Terraform itself is running, Terraformer watches the state file for changes and updates the state ConfigMap as soon as a change occurs. Internally, this is achieved by leveraging standard Kubernetes controller mechanisms: the file watcher inserts an update key into a queue for each file write event, and a separate worker goroutine reads from the queue and updates the state ConfigMap for every key.

After Terraform exits, Terraformer tries to update the state ConfigMap one last time, and retries the operation with the exponential backoff until it succeeds or times out.

Signal handling

Apart from dealing with Terraform configuration and state, Terraformer also handles Pod lifecycle event, i.e. shutdown signals. It will try to relay SIGINT and SIGTERM to the running Terraform process in order to stop the ongoing infrastructure operations on Pod deletion.

Termination log

If the Terraform command execution fails (e.g. because of invalid credentials or similar), Terraformer will copy Terraform's logs to $base_dir/terraform-termination-log (where $base_dir is the directory specified in the --base-dir command line flag (defaults to /)). Controllers deploying Terraformer Pods for infrastructure management can set spec.containers[].terminationMessagePath in the Pod specification to this file path to have the kubelet populate the .status.containerStatuses[].lastState.terminated.message field (see this doc for reference). It can then be used to detect error codes from the Terraform logs without fetching all Pods logs, that also include info messages from Terraformer.

The Pod status will then look similar to this:

status:
  containerStatuses:
  - name: terraformer
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        exitCode: 1
        finishedAt: "2021-04-26T10:28:40Z"
        message: "\nError: error configuring Terraform AWS Provider: error validating
          provider credentials: error calling sts:GetCallerIdentity: SignatureDoesNotMatch:
          The request signature we calculated does not match the signature you provided.
          Check your AWS Secret Access Key and signing method. Consult the service
          documentation for details.\n\tstatus code: 403, request id: 49163947-9edf-4c44-ae7d-b013b119442a\n\n\n"
        reason: Error
        startedAt: "2021-04-26T10:28:32Z"

How to run it locally

The Makefile specifies targets for running and developing Terraformer locally:

make run

make run (with an optional variable COMMAND=[command]) will run the given terraformer command (or apply by default) locally on your machine using go run. You will have to point Terraformer to a running cluster, where the config and state is stored, via the KUBECONFIG environment variable. This can be any Kubernetes cluster, either a local one (setup using minikube, kind or the local nodeless garden) or even a remote one.

$ make run COMMAND=apply
# running `go run ./cmd/terraformer apply`
...

make start

make start is similar to make run, but instead of directly running terraformer on your machine it starts a docker development container with all the prerequisites installed (like go, terraform and the terraform aws provider plugin) and starts terraform via go run in it. This allows running terraformer in an environment that is very close to the environment that it is executed in, when running it in a Kubernetes Pod.

Just as with make run, you will have to set your KUBECONFIG to point to a Kubernetes cluster used for hosting config and state. If you are using kind, you might want to use the example cluster config to create a cluster with certificates, that will be trusted from inside the docker VM (on Mac or Windows).

When running this for the first time, the development container will have to be built, which might take a minute. After that, the image will be reused and only rebuilt if you change something in the files relevant for the image build.

make start-dev-container

The mentioned development container can also be started directly by executing running make start-dev-container. This will open a shell in this container, where you can execute all commands for development and testing like make, go, ginkgo and so on.

How to test it

Terraformer currently comes with three suites of tests: a unit test, a binary e2e and a Pod e2e test suite.

Unit Tests and Binary E2E tests

The unit tests and binary e2e tests can be executed by running make test. Most tests are executed against a local control plane by leveraging the envtest package. Therefore, the kube-apiserver and etcd binaries are fetched to bin/kubebuilder and will be started by the tests.

You can also run the tests against a different cluster for debugging or other purposes by setting the USE_EXISTING_CLUSTER and KUBECONFIG environment variable:

KUBECONFIG=$HOME/.kube/configs/local/garden.yaml USE_EXISTING_CLUSTER=true make test

Pod E2E tests

The Pod E2E tests can be executed via make test-e2e. This will run e2e tests against a terraformer Pod with the AWS plugin installed. It uses an existing cluster (given by the KUBECONFIG env var) and deploys a terraformer apply Pod, that will create some lightweight resource (ec2 keypair) on AWS. The test validates, that the resource was created on AWS using the AWS go-sdk and that the state ConfigMap has been updated accordingly. After that, the test deploys a terraformer destroy Pod and validates again the changes on AWS and the state ConfigMap.

In order to execute this test suite, you will need an existing Kubernetes cluster, where the terraformer Pod will be deployed (pointed to by the KUBECONFIG env var). This can be any Cluster (either local, e.g. via kind) or a remote one running in the Cloud. Additionally, you will need a set of AWS credentials with which the test resources will be created (stored under .kube-secrets/aws/{access_key_id,secret_access_key}.secret).

$ make test-e2e
# Executing pod e2e test with terraformer image eu.gcr.io/gardener-project/gardener/terraformer-aws:v2.1.0-dev-b705a1b4b9bfd47a106998892f48ced0dc8caa56
# If the image for this tag is not built/pushed yet, you have to do so first.
# Or you can use a specific image tag by setting the IMAGE_TAG variable
# like this: `make test-e2e IMAGE_TAG=v2.0.0`
=== RUN   TestTerraformer
Running Suite: Terraformer Pod E2E Suite
========================================
...

Docker Images

Terraformer images are built with every pipeline run and pushed to a public GCR repository. The list of existing images and tags can be found in eu.gcr.io/gardener-project/gardener.

Image variants

This repo features different container image variants, all and a few different provider-specific variants. Each image variant includes the terraformer binary itself, plus terraform and some terraform provider plugins.

The image variants are specified under build, each directory represents one variant. Each variant defines which version of terraform (TF_VERSION file) and which terraform provider plugins (with their respective versions, see (terraform-bundle.hcl file)) should be packaged into the image as well.

Historically, terraformer images included provider plugins for all Gardener provider extensions that were using terraformer for managing a Shoot cluster's infrastructure. This image is equivalent to the all variant. Packaging all provider plugins makes terraformer's images quite large and thus unnecessarily increases image pull time, network traffic and cost. With the different image variants, Gardener provider extensions can now deploy terraformer images with only the needed plugins inside. Also, the different extensions don't have to agree on a common terraform version, but are able to choose the terraform version which they want to use in their provider-specific image.

The all image variant is tagged as eu.gcr.io/gardener-project/gardener/terraformer, while the provider-specific image variants are tagged as eu.gcr.io/gardener-project/gardener/terraformer-{aws,gcp,...}.

Building images locally

You can use make docker-images to build all image variants locally.

Alternatively, make docker-image PROVIDER={all,aws,gcp,...} can be used to build only one specific image variant.

Find out more & Get in touch!

Terraformer was presented in the Gardener Community Call on Nov, 13th 2020.
Watch the recording to learn the story behind terraformer v2 and how it is used in our provider extensions!

Feedback and contributions are always welcome!

Reach out to us:

Please find further resources about our project here:

More Repositories

1

gardener

Homogeneous Kubernetes clusters at scale on any infrastructure using hosted control planes.
Go
2,869
star
2

etcd-backup-restore

Collection of components to backup and restore the etcd of a Kubernetes cluster.
Go
285
star
3

machine-controller-manager

Declarative way of managing machines for Kubernetes cluster
Go
253
star
4

dashboard

Web-based GUI for Gardener installations.
JavaScript
209
star
5

external-dns-management

Environment to manage external DNS entries for a kubernetes cluster
Go
80
star
6

garden-setup

Describes Gardener components for installation of a Gardener landscape using sow
Shell
78
star
7

etcd-druid

An etcd operator to configure, provision, reconcile and monitor etcd clusters.
Go
70
star
8

landscaper

Development of Landscaper - A deployer for K8S workloads with integrated data flow engine.
Go
56
star
9

oidc-webhook-authenticator

Kubernetes Webhook Authenticator that allows for dynamic registration of OpenID Connect providers
Go
47
star
10

documentation

Documentation and website
Shell
34
star
11

cert-management

Manages TLS certificates in Kubernetes clusters using custom resources
Go
32
star
12

hvpa-controller

Controller for weight based horizontal and vertical scaling
Go
31
star
13

gardener-extension-provider-openstack

Gardener extension controller for the OpenStack cloud provider (https://openstack.org).
Go
25
star
14

gardenctl-v2

Command-line client for the Gardener with focus on operations
Go
24
star
15

cc-utils

CI/CD utils for gardener project
Python
21
star
16

gardener-extension-provider-aws

Gardener extension controller for the AWS cloud provider (https://aws.amazon.com).
Go
18
star
17

hyperkube

Shell
17
star
18

logging

Components needed for Gardener logging
Go
15
star
19

ops-toolbelt

Useful tools and operations guide for gardener landscapes
Shell
14
star
20

docforge

Scalable build tool for distributed documentation sources
Go
13
star
21

gardener-extension-networking-cilium

Gardener extension controller for the Cilium CNI network plugin.
Go
13
star
22

gardener-extension-provider-gcp

Gardener extension controller for the GCP cloud provider (https://cloud.google.com).
Go
13
star
23

apiserver-proxy

SNI Passthrough proxy for kube-apiservers
Shell
13
star
24

kupid

Inject scheduling criteria into target pods orthogonally by policy definition.
Go
11
star
25

test-infra

Test machinery for orchestration of integration/e2e/smoke style tests
Go
11
star
26

sow

A tiny installation tool based on spiff and plugins
Shell
11
star
27

controller-manager-library

Library easing implementation of kubernetes controllers
Go
11
star
28

gardener-extension-provider-alicloud

Gardener extension controller for the Alibaba cloud provider (https://alibabacloud.com).
Go
10
star
29

website-generator

Tools to generate the public facing gardener website
SCSS
10
star
30

gardenlogin

kubectl credential plugin for shoot cluster admin authentication
Go
10
star
31

gardener-extension-provider-azure

Gardener extension controller for the Azure cloud provider (https://azure.microsoft.com).
Go
10
star
32

gardener-extension-shoot-cert-service

Gardener extension controller for certificate services for shoot clusters.
Go
10
star
33

gardener-extension-os-gardenlinux

Gardener extension controller for the Garden Linux operating system
Go
9
star
34

gardener-extension-provider-equinix-metal

Gardener extension controller for the Equinix Metal cloud provider (https://equinixmetal.com/).
Go
8
star
35

gardener-extension-provider-vsphere

Gardener extension controller for the vSphere cloud provider (https://www.vmware.com).
Go
8
star
36

gardener-extension-networking-calico

Gardener extension controller for the Calico CNI network plugin.
Go
8
star
37

machine-controller-manager-provider-aws

Gardener machine controller manager provider for AWS
Go
8
star
38

gardener-extension-shoot-dns-service

Gardener extension controller for DNS services for shoot clusters.
Go
8
star
39

homebrew-tap

Provides Gardener tools via Homebrew package
Ruby
7
star
40

gardener-extension-registry-cache

Gardener extension controller which deploys pull-through caches for container registries.
Go
7
star
41

diki

Go
7
star
42

ci-infra

Test infrastructure for the Gardener project.
Go
6
star
43

machine-controller-manager-provider-azure

This repository is the out of tree implementation of the machine driver for Azure cloud provider
Go
6
star
44

service-account-issuer-discovery

A simple server that exposes the OpenID discovery documents of a Kubernetes cluster.
Go
6
star
45

landscapercli

Go
6
star
46

etcd-wrapper

Configures and starts an embedded ETCD
Go
5
star
47

chaos-engineering

Chaos engineering tools for Gardener-managed clusters
Python
5
star
48

machine-controller-manager-provider-vsphere

Gardener machine controller manager provider for VMware vSphere
Go
5
star
49

gardener-extension-os-suse-chost

Gardener extension controller for the SUSE Container Host operating system (CHost).
Go
5
star
50

gardener-extension-shoot-oidc-service

Gardener extension controller for OpenID Connect services for shoot clusters.
Go
5
star
51

vpn2

Network connector between the control plane (deployed in a Seed cluster) and a Shoot cluster superseding the vpn repository.
Go
5
star
52

machine-controller-manager-provider-equinix-metal

Out of tree (controller based) implementation for `Equinix Metal` as provider.
Go
5
star
53

remedy-controller

Remedy controller for automatic application of remedies for platform issues
Go
4
star
54

gardener-metrics-exporter

A Prometheus exporter for Gardener service-level metrics.
Go
4
star
55

landscaper-examples

Landscaper Examples
Shell
4
star
56

network-problem-detector

Network problem detector agent and cli
Go
4
star
57

machine-controller-manager-provider-openstack

Out of tree implementation for machine-controller-manager's provider-openstack
Go
4
star
58

terminal-controller-manager

Used for the webterminal feature of the gardener/dashboard
Go
4
star
59

gardener-extension-os-coreos

Gardener extension controller for the CoreOS/FlatCar Container Linux operating system.
Go
4
star
60

gardener-extension-runtime-gvisor

Gardener extension controller for the gVisor container runtime sandbox (https://gvisor.dev).
Go
4
star
61

machine-controller-manager-provider-alicloud

Gardener machine controller manager provider for Alicloud
Go
4
star
62

landscaper-service

A service managing multi-tenant landscaper installations.
Go
4
star
63

dependency-watchdog

This controller checks the status of etcd and restarts control plane components which are in a state of crashloop-backoff over an extensive period of time.
Go
4
star
64

gardener-extension-shoot-lakom-service

A k8s admission controller verifying pods are using signed images (cosign signatures) and a gardener extension to install it for shoots and seeds.
Go
3
star
65

machine-controller-manager-provider-sampleprovider

Out of tree implementation for SampleProvider as a new cloud provider
Go
3
star
66

machine-controller-manager-provider-gcp

Gardener machine controller manager provider for GCP
Go
3
star
67

etcd-custom-image

Custom etcd container image
Shell
3
star
68

k8syncer

Syncing k8s resources from the cluster into some kind of storage.
Go
3
star
69

gardener-extension-shoot-networking-problemdetector

Gardener extension for deploying network problem detector
Go
3
star
70

gardener-extension-shoot-networking-filter

Gardener extension controller for networking policy filter.
Go
2
star
71

gardener-extension-os-ubuntu

Gardener extension controller for the Ubuntu operating system.
Go
2
star
72

website

2
star
73

alpine-conntrack

Alpine image with pre-installed conntrack tools
Dockerfile
2
star
74

ext-authz-server

External authorization server for envoy.
Shell
2
star
75

oidc-apps-controller

A kubernetes controller enabling OIDC authentication and RBAC authorisation proxies to target workloads.
Go
2
star
76

chocolatey-packages

This repository contains chocolatey packages for tools from `github.com/gardener`.
PowerShell
1
star
77

monitoring

Components needed for Gardener monitoring
1
star
78

aws-custom-route-controller

Custom route controller for AWS
Go
1
star
79

aws-ipam-controller

AWS ipam controller for kubernetes
Go
1
star
80

etcd-druid-api

External APIs to manage an etcd cluster and its individual members
Go
1
star
81

falco-event-ingestor

Receive, validate, and store events from Falco deployed by Gardener.
Go
1
star
82

quic-reverse-http-tunnel

A reverse HTTP Tunnel using QUIC protocol
Go
1
star
83

kube-rbac-proxy-watcher

A configuration watcher managing the lifecycle of a child process upon changes on the watched resources.
Go
1
star
84

ingress-default-backend

Default ingress backend for Shoot clusters.
Shell
1
star