• Stars
    star
    309
  • Rank 132,022 (Top 3 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 5 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager.

aws-secret-operator

A Kubernetes operator that automatically creates and updates Kubernetes secrets according to what are stored in AWS Secrets Manager.

aws-secret-operator custom resources maps AWS secrets to K8S secrets. Consider K8S secrets as just cached, latest AWS secrets.

Benefits

Security:

By "decryption at rest". No need to create Kubernetes secrets by hand, helm, kustomize, or anything that requires you to decrypt the original secret on CI or your laptop

Scalability:

Relies on Secrets Manager instead of SSM Parameter Store for less chances being throttled by SSM's API rate limit.

Kubernetes secrets act as cache of Secrets Manager secrets, even number of API calls to Secrets Manager is minimum.

Usage

Let's say you've stored a secrets manager secret named prod/mysecret whose value is:

{
  "foo": "bar"
}
$ aws secretsmanager get-secret-value \
    --secret-id prod/mysecret

$ aws secretsmanager create-secret \
    --name prod/mysecret

{
    "ARN": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:prod/mysecret-Ld0PUs",
    "Name": "prod/mysecret"
}

$ aws secretsmanager put-secret-value\
    --secret-id prod/mysecret \
    --secret-string '{"foo":"bar"}'

Let's see the SecretId and VersionId which uniquely identifies the secret:

$ aws secretsmanager describe-secret --secret-id prod/mysecret
{
    "ARN": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:prod/mysecret-Ld0PUs",
    "Name": "prod/mysecret",
    "LastChangedDate": 1543636981.306,
    "LastAccessedDate": 1543622400.0,
    "VersionIdsToStages": {
        "c43e66cb-d0fe-44c5-9b7e-d450441a04be": [
            "AWSCURRENT"
        ]
    }
}

Note that aws-secret-operator intentionally disallow omitting VersionId or specifying VersionStage as it makes you difficult to trigger updates to Pods in response to AWS secrets changes.

Run a script like update-aws-secret-ids in order to automate bumping VersionId in your configuration files.

Create a custom resource awssecret named example that points the SecretsManager secret:

your_example_awssecret.yaml:

apiVersion: mumoshu.github.io/v1alpha1
kind: AWSSecret
metadata:
  name: example
spec:
  stringDataFrom:
    secretsManagerSecretRef:
      secretId: prod/mysecret
      versionId: c43e66cb-d0fe-44c5-9b7e-d450441a04be

The operator then creates a Kubernetes secret named example that looks like:

{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "example",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/secrets/test",
    "uid": "82ef45ee-4fdd-11e8-87bf-00e092001ba4",
    "resourceVersion": "25758",
    "creationTimestamp": "2018-05-04T20:55:43Z"
  },
  "data": {
    "foo": "YmFyCg=="
  },
  "type": "Opaque"
}

Now, your pod should either mount the generated secret as a volume, or set an environment variable from the secret.

Secrets manager has two secret formats: key/value (where the content is a plain JSON key-value map) and plain (a raw string). When the operator spots the key/value version, it expands all the key-value pairs into the nested secret keys:

{
  "apiVersion": "v1",
  "data": {
    "password": "cGFzc3dvcmQ=",
    "user": "dXNlcg=="
  },
  "kind": "Secret",
  "metadata": {
    "creationTimestamp": "2020-06-29T10:35:32Z",
    "name": "example",
    "namespace": "default"
  },
  "type": "Opaque"
}

In case of plain secret format, the whole content of the secret is exposed as a data key of the secret:

{
  "apiVersion": "v1",
  "data": {
    "data": "Zm9vXG5iYXIK"
  },
  "kind": "Secret",
  "metadata": {
    "creationTimestamp": "2020-06-29T10:35:32Z",
    "name": "example",
    "namespace": "default"
  },
  "type": "Opaque"
}

Advanced Configuration

The following spec fields are defined to customize the generated Secret:

  • spec.type maps to generated secret's type field
  • spec.metadata.annotations maps to generated secret's metadata.annotations field
  • spec.metadata.labels maps to genrated secret's metadata.labels field

Note that AWSSecret's metadata.annotations and metadata.labels are not propagated down to the generate secret. Use spec.metadata.annotations and spec.metadata.labels instead.

Installation

# Setup RBAC (Namespaced, more secure)
$ kubectl create -f deploy/namespaced/rbac.yaml

# Setup RBAC (Cluster-scoped, easy to use)
$ kubectl create -f deploy/cluster_scoped/rbac.yaml

# Setup the CRD
$ kubectl create -f deploy/crds/mumoshu.github.io_awssecrets.yaml

# Deploy the app-operator
# CAUTION: replace `ap-northeast-2` with your region e.g. us-west-2, and image tag
$ cat deploy/namespaced/deployment.yaml | sed -e 's/REPLACE_THIS_WITH_YOUR_REGION/ap-northeast-1/' | kubectl create -f -
# or cluster-scoped
$ cat deploy/cluster_scoped/deployment.yaml | sed -e 's/REPLACE_THIS_WITH_YOUR_REGION/ap-northeast-1/' | kubectl create -f -

# Verify that a pod is created
$ kubectl get pod -l app=aws-secret-operator

# Create an AWSSecret resource
$ kubectl create -f your_example_awssecret.yaml

# Verify that a secret is created
$ kubectl get secret

# Cleanup
$ kubectl delete -f your_example_awssecret.yaml
$ kubectl delete -f deploy/namespaced/deployment.yaml
$ kubectl delete -f deploy/namespaced/rbac.yaml
$ kubectl delete -f deploy/crds/mumoshu_v1alpha1_awssecret_crd.yaml

Helm Chart

If you use helm chart, you can install this application by chart.

Why not...

  1. Why not use helm-secrets or sops in combination with e.g. kubectl?

    Because I don't want to give my CI the permission to decrypt secrets.

    Just for example, helm-secrets works by calling sops under the hood to decrypt encrypted values.yaml files, so that helm is able to sees the secrets as unencrypted values.yaml files. This implies that your CI system must have an AWS credential to call kms:Decrypt referring your KMS key. A compromised AWS credential can be used by a malicious user to decrypt those secrets. This is especially problematic when the CI system is a publicly hosted SaaS.

    aws-secret-operator prevents this kind of attacks by enabling your deployment pipeline usually seen in a CI system to submit just the reference to secrets stored in AWS Secrets Manager. The operator then decrypts them to produce Kubernetes Secret objects.

  2. Why not use AWS SSM Parameter Store as a primary source of secrets?

    Pros:

    Parameter Store has an efficient API to batch get multiple secrets sharing a same prefix.

    Cons:

    Its API rate limit is way too low. This has been discussed in several places in the Internet:

  3. Why not use S3 as a primary source of secrets?

    Pros:

    Scalability. This project could have used S3 instead, because S3 supports efficient batch gets with filters by prefixes. An example of such project is chamber. chamber is a CLI wraps SSM Param Store and S3, moving from Parameter Store to S3 due to the issue 1 explained above.

    Cons:

    Tooling. One of benefit of Secrets Manager over S3 is that in theory Secrets Manager has possibilities to deserve attentions of developers who who, for a better U/X, wraps Secrets Manager into a dedicated service/application to manager secrets.

    As using S3 for a primary storage for secrets is not a common practice, S3 can be said to have less possibilities to deserve.

Use in combination with...

  1. Use sops in an independent CI/CD pipeline so you can version-control the "latest master data" of secrets on Git repos. Each pull request that changes the master data results in CI workflows that deploys the master data to Secrets Manager.

    Do provide only a KMS encryption permission to the CI system, so that the compromised AWS credential won't allow the attacker to decrypt your secrets.

Alternatives

  1. Use bitnami-labs/sealed-secrets when:
  1. Use future-simple/helm-secrets when:
  • You don't need to share secrets across apps/namespaces/environments.

    Assuming you're going to manage encrypted secrets within a Git repo, sharing them requires you to copy and possible re-encrypt the secret to multiple git projects.

Acknowledgements

This project is powered by operator-framework. Thanks for building the awesome framework :)

More Repositories

1

kube-airflow

A docker image and kubernetes config files to run Airflow on Kubernetes
Python
649
star
2

variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow.
Go
301
star
3

terraform-provider-eksctl

Manage AWS EKS clusters using Terraform and eksctl
Go
229
star
4

helm-x

Treat any Kustomization or K8s manifests directory as a Helm chart
Go
169
star
5

play2-memcached

A memcached plugin for Play 2.x
Scala
162
star
6

variant2

Turn your bash scripts into a modern, single-executable CLI app today
Go
134
star
7

terraform-provider-helmfile

Deploy Helmfile releases from Terraform
Go
115
star
8

kube-ssm-agent

Secure, access-controlled, and audited terminal sessions to EKS nodes without SSH
Dockerfile
106
star
9

kube-spot-termination-notice-handler

Moved to https://github.com/kube-aws/kube-spot-termination-notice-handler
Shell
97
star
10

concourse-aws

Auto-scaling Concourse CI v2.2.1 on AWS with Terraform (since 2016.04.14)
Go
72
star
11

play2-typescript

TypeScript assets handling for Play 2.0. Compiles .ts files under the /assets dir along with the project.
Scala
72
star
12

crossover

Minimal sufficient Envoy xDS for Kubernetes that knows https://smi-spec.io/
Go
71
star
13

helmfile-operator

Kubernetes operator that continuously syncs any set of Chart/Kustomize/Manifest fetched from S3/Git/GCS to your cluster
Go
62
star
14

okra

Hot-swap Kubernetes clusters while keeping your service up and running.
Go
48
star
15

decouple-apps-and-eks-clusters-with-tf-and-gitops

Smarty
31
star
16

kodedeploy

CodeDeploy for EKS. Parallel and Continuous (re)delivery to multiple EKS clusters. Blue-green deployment "of" EKS clusters.
Shell
29
star
17

config-registry

Switch between kubeconfigs and avoid unintentional operation on your production clusters.
Go
28
star
18

gosh

(Re)write your bash script in Go and make it testable too
Go
24
star
19

tiny-mmo

A work-in-progress example of server-side implementation of a tiny massive multi-player online game, implemented with Scala 2.10.2 and Akka 2.2
Java
23
star
20

geohash-scala

Geohash library for scala
Scala
23
star
21

prometheus-process-exporter

Helml chart for Prometheus process-exporter
Mustache
22
star
22

waypoint-plugin-helmfile

Helmfile deployment plugin for HashiCorp Waypoint
Go
15
star
23

kube-fluentd

An ubuntu-slim/s6-overlay/confd based docker image for running fluentd in Kubernetes pods/daemonsets
Makefile
15
star
24

kube-magic-ip-address

daemonset that assigns a magic IP address to pods. Useful for implementing node-local services on Kubernetes. Use in combination with dd-agent, dd-zipkin-proxy, kube2iam, kiam, and Elastic's apm-server
Shell
15
star
25

node-detacher

Practically and gracefully stop your K8s node on (termination|scale down|maintenance)
Go
13
star
26

kube-node-init

Kubernetes daemonset for node initial configuration. Currently for modifying files and systemd services on eksctl nodes without changing userdata
Smarty
13
star
27

lambda_bot

JavaScript
12
star
28

argocd-clusterset

A command-line tool and Kubernetes controller to sync EKS clusters into ArgoCD cluster secrets
Go
12
star
29

dcind

Docker Compose (and Daemon) in Docker
Shell
12
star
30

flux-repo

Enterprise-grade secrets management for GitOps
Go
10
star
31

brigade-helmfile-demo

Demo for building an enhanced GitOps pipeline with Flux, Brigade and Helmfile
JavaScript
9
star
32

kubeherd

A opinionated toolkit to automate herding your cattles = ephemeral Kubernetes clusters
Shell
9
star
33

sopsed

Spawning and storage of secure environments powered by sops, inspired from vaulted. Out-of-box support for kubectl, kube-aws, helm, helmfile
Go
9
star
34

helmfile-gitops

8
star
35

nodelocaldns

Temporary location of the nodelocaldns chart until it is upstreamed to helm/charts
Smarty
8
star
36

kube-logrotate

An ubuntu-slim/s6-overlay/confd based docker image for running logrotate via Kubernetes daemonsets
Makefile
7
star
37

idea-play

A plugin for IntelliJ IDEA which supports development of Play framework applications.
Java
7
star
38

actions-runner-controller-ci

Test repo for https://github.com/summerwind/actions-runner-controller
Go
6
star
39

kargo

Deploy your Kubernetes application directly or indirectly
Go
6
star
40

shoal

Declarative, Go-embeddable, and cross-platform package manager powered by https://gofi.sh/
Go
6
star
41

ingress-daemonset-controller

Build your own highly available and efficient external ingress loadbalancer on Kubernetes
Go
6
star
42

conflint

Unified lint runners for various configuration files
Go
5
star
43

fluent-plugin-datadog-log

Sends logs to Datadog Log Management: A port/translation of https://github.com/DataDog/datadog-log-agent to a Fluentd plugin
Ruby
5
star
44

play20-zentasks-tests

Added spec2 tests to Play20's 'zentasks
Scala
4
star
45

versapack

Versatile package manager. Version and host anything in Helm charts repositories
4
star
46

nomidroid

Android app to find nearby restaurants, bars, etc with Recruit Web API
Java
4
star
47

vote4music-play20

A Play 2.0 RC1 port of https://github.com/loicdescotte/vote4music
Scala
4
star
48

homebrew-anyenv

Ruby
4
star
49

terraform-provider-kubectl

Run kubectl against multiple K8s clusters in a single terraform-apply. AWS auth/AssumeRole supported.
Go
3
star
50

kube-node-index-prioritizing-scheduler

Kubernetes scheduler extender that uses sorted nodes' positions as priorities. Use in combination with resource request/limit to implement low-latency highly available front proxy cluster
Go
3
star
51

depot

Ruby
2
star
52

kokodoko

JavaScript
2
star
53

brigade-cd

Go
2
star
54

scripts

my scripts for linux machines
Shell
2
star
55

courier

Go
2
star
56

nodejs-chat

JavaScript
2
star
57

ooina

JavaScript
2
star
58

play-websocket-growl-sample

A sample application notifies the server on Growl, and other users via WebSocket
Java
2
star
59

heartbeat-operator

Monitor any http and tcp services via Kubernetes custom resources
Shell
2
star
60

brigade-automation

TypeScript
2
star
61

nbxmpfilter

xmpfilter for NetBeans
2
star
62

wy

Go
2
star
63

play-scala-lobby-example

Java/Scala mixed app with a WebSocketController in Java, while others are in Scala.
JavaScript
2
star
64

docker-squid-gem-proxy

Dockerfile for building docker image to run Squid-based reverse proxies to rubygems.org, for speeding-up `gem install` or `bundle install`
Shell
2
star
65

hcl2-yaml

YAML syntax for HashiCorp Configuration Language Version 2
Go
2
star
66

tweet_alarm_clock_android

Java
2
star
67

testkit

A toolkit for writing Go tests for your cloud apps, infra-as-code, and gitops configs
Go
1
star
68

elasticdrone

A set of tools to configure auto-scaled Drone.io cluster on AWS in ease
JavaScript
1
star
69

kubeaws-cicd-pipeline-golang

An example of the opinionated deployment pipeline for your Kubernetes-on-AWS-native apps. Secure, declarative, customizable, open app deployments at your hand.
Shell
1
star
70

akka-parallel-tasks

Scala
1
star
71

sing

1
star
72

good-code-is-red-example-for-idea-scala-plugin

Scala
1
star
73

share-snippets

JavaScript
1
star
74

scala-scalable-programming-sidenote

Scala
1
star
75

teleport-on-minikube

Shell
1
star
76

genrun

Go
1
star
77

nomidroid_test

tests for nomidroid the android app.
Java
1
star
78

web_smartphonizer

Scala
1
star
79

idobata4s

The Scala client library for idobata.io API
Scala
1
star
80

specs2-sandbox

Scala
1
star
81

kube-distributed-test-runner

Go
1
star
82

syaml

Set YAML values with `cat your.yaml | syaml a.b.c newValue`
Go
1
star
83

delimited-continuation

An example usage of Delimited Continuation in Scala,;implementing an async HTTP client.
Scala
1
star
84

play2-unit-tests-sample

1
star
85

javascript-missiles-and-lasers

JavaScript
1
star
86

scala-debian-package

1
star
87

glabel

A GOverlay implementation for putting text labels on GMap2
JavaScript
1
star
88

gitops-demo

Makefile
1
star
89

division

Control-plane of your microservices and Kubernetes clusters
Go
1
star
90

mumoauth

Yet another implementation of OAuth aiming to cover both OAuth 1.0a and 2, from Client to Server.
Scala
1
star
91

sam-containerized-go-example

Go
1
star
92

kube-veneur

Sidecar container to run local veneur and Kubernetes service for global https://github.com/stripe/veneur. Uses confd and s6-overlay under the hood.
Makefile
1
star
93

tweet_alarm_clock_android_test

Java
1
star
94

javascript-hue_circle

Draw a hue circle in JavaScript
JavaScript
1
star
95

thread_local

An implementation of the thread-local variable for Ruby
Ruby
1
star
96

play-multijpa

a Play! plugin to switch databases for each play.db.jpa.Model subclass
Java
1
star
97

argocd-example-apps

Jsonnet
1
star
98

logrus-bunyan-formatter

Logrus Bunyan Log Formatter
Go
1
star
99

crdb

Custom Resource DB - Kubernetes custom resources without Kubernetes or self-hosted databases. For any cloud, by leveraging cloudprovider-specific managed databases like AWS DynamoDB
Go
1
star