AWS Encryption Provider
This repository is an implementation of the kube-apiserver encryption provider, backed by AWS KMS.
Use with Kubernetes
Assumptions
The following guide makes several assumptions:
- You have an AWS account and permission to manage KMS keys
- You have management access to a Kubernetes API server
- You have already read the Kubernetes documentation page Encrypting Secret Data at Rest
- You have already read the Kubernetes documentation page Using a KMS provider for data encryption
- The AWS KMS encryption provider will need AWS credentials configured in order to call KMS APIs. You can read more about providing credentials by reading the AWS SDK documentation on configuring your application.
Setup
First you'll need to create a KMS master key. For more details you can read the KMS documentation on creating a key. Check the KMS pricing page for up-to-date pricing information.
KEY_ID=$(aws kms create-key --query KeyMetadata.KeyId --output text)
aws kms describe-key --key-id $KEY_ID
{
"KeyMetadata": {
"Origin": "AWS_KMS",
"KeyId": "1234abcd-12ab-34cd-56ef-1234567890ab",
"Description": "",
"KeyManager": "CUSTOMER",
"Enabled": true,
"KeyUsage": "ENCRYPT_DECRYPT",
"KeyState": "Enabled",
"CreationDate": 1502910355.475,
"Arn": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
"AWSAccountId": "111122223333"
}
}Key aliases can be used but it is not recommended. An alias can be updated to a new key, which would break how this encryption provider works. As a result all secrets encrypted before the alias update will become unreadable.
Deploy the aws-encryption-provider plugin
While there are numerous ways you could deploy the aws-encryption-provider plugin, the simplest way for most installations would be a static pod on the same node as each Kubernetes API server. Below is an example pod spec, and you will need to replace the image, key ARN, and region to fit your requirements.
apiVersion: v1
kind: Pod
metadata:
name: aws-encryption-provider
namespace: kube-system
spec:
containers:
- image: 111122223333.dkr.ecr.us-west-2.amazonaws.com/aws-encryption-provider:v0.0.1
name: aws-encryption-provider
command:
- /aws-encryption-provider
- --key=arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
- --region=us-west-2
- --listen=/var/run/kmsplugin/socket.sock
ports:
- containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
volumeMounts:
- mountPath: /var/run/kmsplugin
name: var-run-kmsplugin
volumes:
- name: var-run-kmsplugin
hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreateOnce you have deployed the encryption provider on all the same nodes as your API
servers, you will need to update the kube-apiserver to use the encryption
provider by setting the --encryption-provider-config flag and with the path to
your encryption configuration file. Below is an example:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
name: aws-encryption-provider
endpoint: unix:///var/run/kmsplugin/socket.sock
cachesize: 1000
timeout: 3s
- identity: {}Don't forget, you'll need to mount the directory containing the unix socket that the KMS server is listening on into the kube-apiserver.
Bootstrap during cluster creation (kops)
To use encryption provider during cluster creation, you need to ensure that its running before starting kube-apiserver. For that you need to perform the following high level steps.
Note: These steps have been verified with kops but it should be similar to any other cluster bootstrapping tool.
For exact kops instructions see KOPS.md.
Run aws-encryption-provider as static pod
You need to have encryption provider running before kube-apiserver, and to do that you can
use static pods functionality. For kops, static pod manifests are available at /etc/kubernetes/manifests. You can further use kops file assets functionality to drop
the pod spec file in that directory.
Use Host Network for aws-encryption-provider
As the CNI plugin is not yet available, you need to add hostNetwork: true to pod spec.
Update health port for aws-encryption-provider
When using hostNetwork, the port 8080 used by aws-encryption-provider conflicts with
kube-apiserver which also requires the same port. To fix this, add -health-port=:8083
to args section of pod spec above. Also change the port in containerPort and livenessProbe
sections.
Add /var/run/kmsplugin hostMount to api server spec
Use kops lifecycle hook to run a script/container that can update the kube-apiserver
manifest (available at /etc/kubernetes/manifests) to add /var/run/kmsplugin as hostMount.
Permissions
Ensure master IAM role has permissions to encrypt/decrypt using the kms. You can achieve this using additionalIAMPolicies functionality of kops.
After above changes, the modified pod-spec would look like:
apiVersion: v1
kind: Pod
metadata:
name: aws-encryption-provider
namespace: kube-system
spec:
containers:
- image: 111122223333.dkr.ecr.us-west-2.amazonaws.com/aws-encryption-provider:v0.0.1
name: aws-encryption-provider
command:
- /aws-encryption-provider
- --key=arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
- --region=us-west-2
- --listen=/var/run/kmsplugin/socket.sock
- --health-port=:8083
ports:
- containerPort: 8083
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8083
volumeMounts:
- mountPath: /var/run/kmsplugin
name: var-run-kmsplugin
hostNetwork: true
volumes:
- name: var-run-kmsplugin
hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreateCheck that the provider plugin is working
- First we create a secret:
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata - Then we exec into the etcd-server:
kubectl exec -it -n kube-system $(kubectl get pods -n kube-system | grep etcd-manager-main | awk '{print $1}') bash cd /opt/etcd-v3.3.10-linux-amd64/- Then check the contents of our secret in etcd store by running the following:
ETCDCTL_API=3 etcdctl \
--key /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.key \
--cert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-client.crt \
--cacert /rootfs/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt \
--endpoints "https://etcd-a.internal.${CLUSTER}:4001" get /registry/secrets/default/secret1
Β Β Β -- output should be something like:
0m`οΏ½He.0οΏ½cryption-provider:οΏ½1xοΏ½οΏ½%οΏ½BοΏ½οΏ½οΏ½#JPοΏ½οΏ½JοΏ½οΏ½οΏ½*ΘοΏ½οΏ½οΏ½Ξ@\nοΏ½96οΏ½^οΏ½οΏ½Ϋ¦οΏ½~0| *οΏ½HοΏ½οΏ½
`qοΏ½*οΏ½JοΏ½.PοΏ½οΏ½;&~οΏ½οΏ½o#οΏ½OοΏ½8mοΏ½οΏ½->8LοΏ½οΏ½0οΏ½C3οΏ½οΏ½οΏ½A7οΏ½οΏ½οΏ½οΏ½οΏ½~οΏ½οΏ½οΏ½fοΏ½VοΏ½ά¬οΏ½οΏ½οΏ½XοΏ½οΏ½_οΏ½οΏ½`οΏ½H#οΏ½DοΏ½οΏ½z)+οΏ½81οΏ½οΏ½qWοΏ½οΏ½yοΏ½οΏ½`οΏ½qοΏ½οΏ½}1<LF, οΏ½οΏ½NοΏ½οΏ½pοΏ½οΏ½οΏ½οΏ½i*οΏ½aC#EοΏ½ίΈοΏ½sοΏ½οΏ½οΏ½οΏ½οΏ½οΏ½sοΏ½οΏ½lοΏ½?οΏ½a
οΏ½AΕΊRοΏ½οΏ½οΏ½οΏ½οΏ½οΏ½.οΏ½οΏ½8HοΏ½4οΏ½O
Rotation
If you have configured your KMS master key (CMK) to have rotation enabled, AWS will update the CMK's backing encryption key every year. (You can read more about automatic key rotation at the KMS documentation page) If you are using the aws-encryption-provider with an existing master key, but want to update your cluster to use a new KMS master key, you can by roughly following the below procedure. Be sure to read the Kubernetes documentation on rotating a decryption key, as all of those steps apply to this process.
You will need to run two encryption providers on each API server using different
keys, and you must configure them to each use a different value for the name
field and each provider must listen on a different unix socket. Below is an
example encryption configuration file for all API servers prior to using the new
key.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
# provider using old key
- kms:
name: aws-encryption-provider
endpoint: unix:///var/run/kmsplugin/socket.sock
cachesize: 1000
timeout: 3s
# provider using new Key
- kms:
name: aws-encryption-provider-2
endpoint: unix:///var/run/kmsplugin/socket2.sock
cachesize: 1000
timeout: 3s
- identity: {}After all API servers have been restarted and are able to decrypt using the new key, you can switch the order of the providers with the new key at the beginning of the list and the old key below it. After all secrets have been re-encrypted with the new key, you can remove the encryption provider using the old key from the list.
Community, discussion, contribution, and support
Learn how to engage with the Kubernetes community on the community page.
You can reach the maintainers of this project at:
Code of conduct
Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.