Pulumi Kubernetes Operator
A Kubernetes operator that provides a CI/CD workflow for Pulumi stacks using Kubernetes primitives. To learn more about the Pulumi Kubernetes Operator visit the Pulumi documentation.
Overview
- Pulumi Kubernetes Operator
What is Pulumi?
Pulumi is an open source infrastructure-as-code tool for creating, deploying, and managing cloud infrastructure in the programming language of your choice. If you are new to Pulumi, please consider visiting the getting started first to familiarize yourself with Pulumi and concepts such as Pulumi stacks and backends.
When To Use the Pulumi Kubernetes Operator?
The Pulumi Kubernetes Operator enables Kubernetes users to create a Pulumi Stack as a first-class Kubernetes API resource, and use the StackController to drive the updates. It allows users to adopt a GitOps workflow for managing their cloud infrastructure using Pulumi. This infrastructure includes Kubernetes resources in addition to over 60 cloud providers including AWS, Azure, and Google Cloud. The operator provides an alternative to Pulumi's other CI/CD integrations such as Github Actions, Gitlab CI, Jenkins etc. See the full list of Pulumi's CI/CD integrations here. Since the Pulumi Kubernetes Operator can be deployed on any Kubernetes cluster, it provides turnkey GitOps functionality for Pulumi users running in self-hosted or restricted settings. The Kubernetes Operator pattern, lends itself nicely to automation scenarios by driving to the specified state and automatically retrying if transient failures are encountered.
Prerequisites
The following steps should be completed before starting on Pulumi:
Install Pulumi CLI
Follow the Pulumi installation instructions for your OS. For instance, on Mac OS, the easiest way to install Pulumi CLI is from Homebrew:
$ brew install pulumiLogin to Your Chosen State Backend
The operator stores additional metadata about provisioned resources. By default, Pulumi (and the Pulumi Kubernetes Operator) uses the Pulumi managed SaaS backend to store this state and manage concurrency. However, in addition to the managed backend, Pulumi also readily integrates with a variety of state backends, like S3, Azure Blob Storage, Google Cloud Storage, etc. See here for a detailed discussion on choosing a state backend.
Login to Pulumi using your chosen state backend. For simplicity we will only cover the Pulumi managed SaaS state backend and AWS S3 here:
Pulumi SaaS Backend
$ pulumi loginThis will display a prompt that asks for you to provide an access token or automatically request an access token:
Manage your Pulumi stacks by logging in.
Run `pulumi login --help` for alternative login options.
Enter your access token from https://app.pulumi.com/account/tokens
or hit <ENTER> to log in using your browser :In order to configure the Pulumi Kubernetes Operator to use Stacks with state stored on the SaaS backend, you will also need to manually generate access tokens.
This can be done by accessing the Access Tokens page. Setting the environment variable PULUMI_ACCESS_TOKEN to the manually generated token will obviate the need for a pulumi login.
At this point your pulumi CLI is configured to work with the Pulumi SaaS backend.
AWS S3 Backend
- First, you will need to create an S3 bucket manually, either through the AWS CLI or the AWS Console.
- If you have already configured the AWS CLI to use credential files, single sign-on etc., Pulumi will automatically respect and use these settings. Alternatively you can set
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables to the access key and secret access key respectively. - To use the AWS S3 backend, pass the
s3://<bucket-name>as your<backend-url>topulumi login, i.e.:For additional options, refer to the Pulumi documentation.$ pulumi login s3://<bucket-name> - You will need the AWS credentials when configuring Stack CRs for stacks you wish to be backed by the S3 bucket.
- Lastly you will need to create an AWS Key Management Service (KMS) key. This key will be used by Pulumi to encrypt secret configuration values or outputs associated with stacks. Pulumi ensures all secrets are stored encrypted in transit and at rest. By default, the SaaS backend creates per-stack encryption keys to do this, however, Pulumi can leverage KMS as one of several supported encryption providers instead, thus allowing users to self-manage their encryption keys.
Deploy the Operator
Deploy the operator to a Kubernetes cluster.
You can use an existing cluster, or get started by creating a new managed Kubernetes cluster. We will assume that your target Kubernetes cluster is already created and you have configured kubectl to point to it. Note that Pulumi doesn't actually use kubectl but for convenience can use the same mechanism to authenticate against clusters.
Using kubectl
First, download the latest release source code tar ball and expand it locally.
Deploy the CustomResourceDefinitions (CRDs) for the operator.
kubectl apply -f deploy/crds/Deploy the API resources for the operator.
kubectl apply -f deploy/yamlThis will deploy the operator to the default namespace (which depends on your configuration, and is
usually "default"). To deploy to a different namespace:
kubectl apply -n <namespace> -f deploy/yamlYou can deploy to several namespaces by repeating the above command.
Using Pulumi
First, make sure you have reviewed and performed the tasks identified in the prerequisite section.
We will create a Pulumi project to deploy the operator by using a template, then customize it if
necessary, then use pulumi up to run it. There is a choice of template, related to the programming
language and environment you wish to use:
- deploy/deploy-operator-cs (.NET)
- deploy/deploy-operator-go (Go)
- deploy/deploy-operator-py (Python)
- deploy/deploy-operator-ts (TypeScript/NodeJS)
Pick one of those, then create a new project in a fresh directory:
TEMPLATE=deploy/deploy-operator-ts # for example
mkdir deploy-operator
cd deploy-operator
pulumi new https://github.com/pulumi/pulumi-kubernetes-operator/$TEMPLATE
# If using the S3 state backend, you may wish to set the secrets provider here
pulumi stack change-secrets-provider "awskms:///arn:aws:kms:...?region=<region>"You can then set the namespace, or namespaces, in which to deploy the operator:
pulumi config set namespace ns1
# OR deploy to multiple namespaces
pulumi set config --path namespaces[0] ns1
pulumi set config --path namespaces[1] ns2And finally, run the program:
pulumi upUpgrading the operator
For patch and minor version releases, you can just bump the version in your stack config and rerun
pulumi up. For example, if the new version is v1.10.2, you would do this:
pulumi config set operator-version v1.10.2
pulumi upCreate Pulumi Stack CustomResources
The following are examples to create Pulumi Stacks in Kubernetes that are managed and run by the operator.
Using kubectl
Check out Create Pulumi Stacks using kubectl for YAML examples.
Using Pulumi
Check out Create Pulumi Stacks using Pulumi for Typescript, Python, Go, and .NET examples.
Extended Examples
If you'd like to use your own Pulumi Stack, ensure that you have an existing Pulumi program in a git repo, and update the CR with:
- An existing github
projectand/orcommit, - A Pulumi
stackname that exists and will be selected, or a new stack that will be created and selected. - A Kubernetes Secret for your Pulumi API
accessToken, - A Kubernetes Secret for other sensitive settings like cloud provider credentials, and
- Environment variables and stack config as needed.
Stack CR Documentation
Detailed documentation on Stack Custom Resource is available here.
Prometheus Metrics Integration
Details on metrics emitted by the Pulumi Kubernetes Operator as instructions on getting them to flow to Prometheus are available here.
Development
Check out docs/build.md for more details on building and working with the operator locally.