aws-operator

The aws-operator manages Kubernetes clusters running on AWS.

Branches

• thiccc
  • Up to and including version v5.4.0.
  • Contains all versions of legacy controllers (reconciling AWSConfig CRs) up to and including v5.4.0.
• legacy
  • From version v5.5.0 up to and including v5.x.x.
  • Contains only the latest version of legacy controllers (reconciling AWSConfig CRs).
• master
  • From version v6.0.0.
  • Contains only the latest version of controllers (reconciling cluster API objects).

Getting the Project

Download the latest release: https://github.com/giantswarm/aws-operator/releases/latest

Clone the git repository: https://github.com/giantswarm/aws-operator.git

Download the latest docker image from here: https://quay.io/repository/giantswarm/aws-operator

How to build

Build the standard way.

go build github.com/giantswarm/aws-operator

Architecture

The operator uses our operatorkit framework. It manages an awsconfig CRD using a generated client stored in our apiextensions repo. Releases are versioned using version bundles.

The operator provisions guest Kubernetes clusters running on AWS. It runs in a host Kubernetes cluster also running on AWS.

CloudFormation

The guest Kubernetes clusters are provisioned using AWS CloudFormation. The resources are split between CloudFormation stacks:

In control plane account

• tccpi - Tenant cluster control plane role setup.
• tccpf - Tenant cluster control plane routes setup.
• tcnpf - Tenant cluster nodepool peering.

In tenant account:

• tccp - Tenant cluster network setup.
• tccpn - Tenant cluster control plane resources (masters).
• tcnp - Tenant cluster nodepool resources (workers).

Other AWS Resources

As well as the CloudFormation stacks we also provision a KMS key and S3 bucket per cluster. This is to upload cloudconfigs for the cluster nodes. The cloudconfigs contain TLS certificates which are encrypted using the KMS key.

Kubernetes Resources

The operator also creates a Kubernetes namespace per guest cluster with a service and endpoints. These are used by the host cluster to access the guest cluster.

Certificates

Authentication for the cluster components and end-users uses TLS certificates. These are provisioned using Hashicorp Vault and are managed by our cert-operator.

Secret

Here the AWS IAM credentials have to be inserted.

service:
  aws:
    accesskey:
      id: 'TODO'
      secret: 'TODO'

Here the base64 representation of the data structure above has to be inserted.

apiVersion: v1
kind: Secret
metadata:
  name: aws-operator-secret
  namespace: giantswarm
type: Opaque
data:
  secret.yml: 'TODO'

To create the secret manually do this.

kubectl create -f ./path/to/secret.yml

We also need a key to hold the SSH public key

apiVersion: v1
kind: Secret
metadata:
  name: aws-operator-ssh-key-secret
  namespace: giantswarm
type: Opaque
data:
  id_rsa.pub: 'TODO'

Node VM Images (AMIs)

This operator holds a static mapping of versions and regions to AMI IDs (VM image IDs, region specific) used for tenant cluster nodes in service/controller/key/ami.go. The file is generated by devctl and should not be edited manually. When a new version of the OS is released and new images have been published on AWS, this mapping can be updating using devctl gen ami --dir service/controller/key.

Live editing operator inside an installation

• Download Okteto latest release from https://github.com/okteto/okteto/releases

• okteto init -n giantswarm

• Set correct label app.giantswarm.io/branch: $BRANCH in the manifest

• Change your kubeconfig to the giantswarm namespace

• Modify PSP of the current operator kubectl patch psp aws-operator-$BRANCH-psp -p '{"spec":{"runAsGroup":{"ranges":null,"rule":"RunAsAny"},"runAsUser":{"rule":"RunAsAny"},"volumes":["secret","configMap","hostPath","persistentVolumeClaim","emptyDir"]}}'

• okteto up

• From this point on, you can modify files locally and will be synced to the remote pod

In order to start the operator, you can build it and execute it inside the pod

• go build
• aws-operator daemon --config.dirs=/var/run/aws-operator/configmap/ --config.dirs=/var/run/aws-operator/secret/ --config.files=config --config.files=secret

Live reload code

• cd /tmp && go get -u github.com/cosmtrek/air && cd /okteto
• air -c air.conf

For live debugging in VS Code

• Install delve debugger: go get github.com/go-delve/delve/cmd/dlv
• dlv debug --headless --listen=:2345 --log --api-version=2 -- daemon --config.dirs=/var/run/aws-operator/configmap/ --config.dirs=/var/run/aws-operator/secret/ --config.files=config --config.files=secret or ./debug_server.sh
• Create debugging connection:

  {
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Connect to okteto",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/okteto",
            "port": 2345,
            "host": "127.0.0.1"
        }
    ]
  }

• Wait until debug server is up and create some breakpoints, start the debugger :)
• If you want to edit the code you will need to stop debugging session and stop the server
• okteto down -v (-v will delete volume with go cache)
• Revert psp with kubectl patch psp aws-operator-$BRANCH-psp -p '{"spec":{"runAsGroup":{"ranges": [{"max":65535, "min":1}],"rule":"MustRunAs"},"runAsUser":{"rule":"MustRunAsNonRoot"},"volumes":["secret","configMap"]}}' or redeploy application

Contact

• Mailing list: giantswarm
• Bugs: issues

Contributing & Reporting Bugs

See CONTRIBUTING for details on submitting patches, the contribution workflow as well as reporting bugs.

For security issues, please see the security policy.

License

aws-operator is under the Apache 2.0 license. See the LICENSE file for details.

Credit

• https://github.com/giantswarm/microkit