Amazon EKS Pod Identity Webhook
This webhook is for mutating pods that will require AWS IAM access.
Note
After version v0.3.0, --in-cluster=true
no longer works and is deprecated. Please use --in-cluster=false
and manage the cluster certificate with cert-manager or some other external certificate provisioning system.
This is because certificates using the legacy-unknown
signer are no longer signed when using the v1
certificates API.
EKS Walkthrough
- Create an OIDC provider in IAM for your cluster. You can find the OIDC
discovery endpoint by describing your EKS cluster.
And enter "sts.amazonaws.com" as the client-id
aws eks describe-cluster --name $CLUSTER_NAME --query cluster.identity.oidc
- Create an IAM role for your pods and modify the trust policy to allow
your pod's service account to use the role:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.REGION.eks.amazonaws.com/CLUSTER_ID" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "__doc_comment": "scope the role to the service account (optional)", "StringEquals": { "oidc.REGION.eks.amazonaws.com/CLUSTER_ID:sub": "system:serviceaccount:default:my-serviceaccount" }, "__doc_comment": "scope the role to a namespace (optional)", "StringLike": { "oidc.REGION.eks.amazonaws.com/CLUSTER_ID:sub": "system:serviceaccount:default:*" } } } ] }
- Modify your pod's service account to be annotated with the ARN of the role
you want the pod to use
apiVersion: v1 kind: ServiceAccount metadata: name: my-serviceaccount namespace: default annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::111122223333:role/s3-reader" # optional: Defaults to "sts.amazonaws.com" if not set eks.amazonaws.com/audience: "sts.amazonaws.com" # optional: When set to "true", adds AWS_STS_REGIONAL_ENDPOINTS env var # to containers eks.amazonaws.com/sts-regional-endpoints: "true" # optional: Defaults to 86400 for expirationSeconds if not set # Note: This value can be overwritten if specified in the pod # annotation as shown in the next step. eks.amazonaws.com/token-expiration: "86400"
- All new pod pods launched using this Service Account will be modified to use
IAM for pods. Below is an example pod spec with the environment variables and
volume fields added by the webhook.
apiVersion: v1 kind: Pod metadata: name: my-pod namespace: default annotations: # optional: A comma-separated list of initContainers and container names # to skip adding volumes and environment variables eks.amazonaws.com/skip-containers: "init-first,sidecar" # optional: Defaults to 86400, or value specified in ServiceAccount # annotation as shown in previous step, for expirationSeconds if not set eks.amazonaws.com/token-expiration: "86400" spec: serviceAccountName: my-serviceaccount initContainers: - name: init-first image: container-image:version containers: - name: sidecar image: container-image:version - name: container-name image: container-image:version ### Everything below is added by the webhook ### env: - name: AWS_DEFAULT_REGION value: us-west-2 - name: AWS_REGION value: us-west-2 - name: AWS_ROLE_ARN value: "arn:aws:iam::111122223333:role/s3-reader" - name: AWS_WEB_IDENTITY_TOKEN_FILE value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" - name: AWS_STS_REGIONAL_ENDPOINTS value: "regional" volumeMounts: - mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount/" name: aws-token volumes: - name: aws-token projected: sources: - serviceAccountToken: audience: "sts.amazonaws.com" expirationSeconds: 86400 path: token
Usage with Windows container workloads
To ensure workloads are scheduled on windows nodes have the right environment variables, they must have a nodeSelector
targeting windows it must run on. Workloads targeting windows nodes using nodeAffinity
are currently not supported.
nodeSelector:
beta.kubernetes.io/os: windows
Or for Kubernetes 1.14+
nodeSelector:
kubernetes.io/os: windows
Usage with non-root container user
When running a container with a non-root user, you need to give the container access to the token file by setting the fsGroup
field in the securityContext
object.
Usage
Usage of amazon-eks-pod-identity-webhook:
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--annotation-prefix string The Service Account annotation to look for (default "eks.amazonaws.com")
--aws-default-region string If set, AWS_DEFAULT_REGION and AWS_REGION will be set to this value in mutated containers
--enable-debugging-handlers Enable debugging handlers. Currently /debug/alpha/cache is supported
--in-cluster Use in-cluster authentication and certificate request API (default true)
--kube-api string (out-of-cluster) The url to the API server
--kubeconfig string (out-of-cluster) Absolute path to the API server kubeconfig file
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--metrics-port int Port to listen on for metrics and healthz (http) (default 9999)
--namespace string (in-cluster) The namespace name this webhook, the TLS secret, and configmap resides in (default "eks")
--port int Port to listen on (default 443)
--service-name string (in-cluster) The service name fronting this webhook (default "pod-identity-webhook")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--sts-regional-endpoint false Whether to inject the AWS_STS_REGIONAL_ENDPOINTS=regional env var in mutated pods. Defaults to false.
--tls-cert string (out-of-cluster) TLS certificate file path (default "/etc/webhook/certs/tls.crt")
--tls-key string (out-of-cluster) TLS key file path (default "/etc/webhook/certs/tls.key")
--tls-secret string (in-cluster) The secret name for storing the TLS serving cert (default "pod-identity-webhook")
--token-audience string The default audience for tokens. Can be overridden by annotation (default "sts.amazonaws.com")
--token-expiration int The token expiration (default 86400)
--token-mount-path string The path to mount tokens (default "/var/run/secrets/eks.amazonaws.com/serviceaccount")
-v, --v Level number for the log level verbosity
--version Display the version and exit
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
--watch-config-map Enables watching serviceaccounts that are configured through the pod-identity-webhook configmap instead of using annotations
AWS_DEFAULT_REGION Injection
When the aws-default-region
flag is set this webhook will inject AWS_DEFAULT_REGION
and AWS_REGION
in mutated containers if AWS_DEFAULT_REGION
and AWS_REGION
are not already set.
AWS_STS_REGIONAL_ENDPOINTS Injection
When the sts-regional-endpoint
flag is set to true
, the webhook will
inject the environment variable AWS_STS_REGIONAL_ENDPOINTS
with the value set
to regional
. This environment variable will configure the AWS SDKs to perform
the sts:AssumeRoleWithWebIdentity
call to get credentials from the regional
endpoint, instead of the global endpoint in us-east-1
. This is desirable in
almost all cases, unless the STS regional endpoint is disabled in your
account.
You can also enable this per-service account with the annotation
eks.amazonaws.com/sts-regional-endpoints
set to "true"
.
pod-identity-webhook ConfigMap
The purpose of the pod-identity-webhook
ConfigMap is to simplify the mapping of IAM roles and ServiceAccount
when using tools/installers like kOps that directly manage IAM roles and trust policies. When using these tools,
users do not need to configure annotations on the ServiceAccounts as the tools already know the relationship can relay it to the webhook.
When the watch-config-map
flag is set to true
, the webhook will watch the
pod-identity-webhook
ConfigMap in the namespace configured by the --namespace
flag
for additional ServiceAccounts. The webhook will mutate Pods configured to use these
ServiceAccounts even if they have no annotations.
Should the same ServiceAccount both be referenced both in the ConfigMap and have annotations, the annotations takes presedence.
Here is an example ConfigMap:
apiVersion: v1
data:
config: '{"default/myserviceaccount":{"RoleARN":"arn:aws-test:iam::123456789012:role/myserviceaccount.default.sa.minimal.example.com","Audience":"amazonaws.com","UseRegionalSTS":true,"TokenExpiration":0},"myapp/myotherserviceaccount":{"RoleARN":"arn:aws-test:iam::123456789012:role/myotherserviceaccount.myapp.sa.minimal.example.com","Audience":"amazonaws.com","UseRegionalSTS":true,"TokenExpiration":0},"test-*/myserviceaccount":{"RoleARN":"arn:aws-test:iam::123456789012:role/myserviceaccount.test-wildcard.sa.minimal.example.com","Audience":"amazonaws.com","UseRegionalSTS":true,"TokenExpiration":0}}'
kind: ConfigMap
metadata:
annotations:
prometheus.io/port: "443"
prometheus.io/scheme: https
prometheus.io/scrape: "true"
creationTimestamp: null
name: pod-identity-webhook
namespace: kube-system
Container Images
Container images for amazon-eks-pod-identity-webhook can be found on Docker Hub.
Installation
Pre-requisites
You must install cert-manager as it is a pre-requisite for below deployments. (See cert-manager installation)
In-cluster
You can use the provided configuration files in the deploy
directory, along with the provided Makefile
.
make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:latest
This will:
- Create a service account, role, cluster-role, role-binding, and cluster-role-binding that the deployment requires
- Create the deployment, service, ClusterIssuer, certificate, and mutating webhook in the cluster
- Use
in-cluster=false
so that the webhook reloads certificates from the filesystem rather than creating CSRs to request certificates (using CSRs is now deprecated and will not work versions later than v0.3.0).
For self-hosted API server configuration, see see SELF_HOSTED_SETUP.md
On API server
TODO
Notes
With the upgrade to client-go 1.18, certificate_manager_server_expiration_seconds metric has been removed by an upstream commit kubernetes/kubernetes#85874. A new metric certificate_manager_server_rotation_seconds is added which tracks the time a certificate was valid before getting rotated.
Code of Conduct
License
Apache 2.0 - Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. See LICENSE