• Stars
    star
    195
  • Rank 199,374 (Top 4 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created about 9 years ago
  • Updated 6 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Vault sidekick

Build Status GoDoc Docker Repository on Quay GitHub version

Vault Side Kick

Summary

Vault Sidekick is a add-on container which can be used as a generic entry-point for interacting with Hashicorp Vault service, retrieving secrets (both static and dynamic) and PKI certs. The sidekick will take care of renewal's and extension of leases for you and renew the credentials in the specified format for you.

Usage

$ sudo docker run --rm quay.io/ukhomeofficedigital/vault-sidekick:v0.3.3 -help
Usage of /vault-sidekick:
  -alsologtostderr
    	log to standard error as well as files
  -auth string
    	a configuration file in json or yaml containing authentication arguments
  -ca-cert string
    	the path to the file container the CA used to verify the vault service
  -cn value
    	a resource to retrieve and monitor from vault
  -dryrun
    	perform a dry run, printing the content to screen
  -exec-timeout duration
    	the timeout applied to commands on the exec option (default 1m0s)
  -format string
    	the auth file format (default "default")
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -logtostderr
    	log to standard error instead of files
  -one-shot
    	retrieve resources from vault once and then exit
  -output string
    	the full path to write resources or VAULT_OUTPUT (default "/etc/secrets")
  -renew-token
      renew vault token according to its ttl
  -stats duration
    	the interval to produce statistics on the accessed resources (default 1h0m0s)
  -stderrthreshold value
    	logs at or above this threshold go to stderr
  -tls-skip-verify
    	whether to check and verify the vault service certificate
  -v value
    	log level for V logs
  -vault string
    	url the vault service or VAULT_ADDR (default "https://127.0.0.1:8200")
  -version
    	show the vault-sidekick version
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging
  -request-token 
        request an authentication token from vault and exit

Building

There is a Makefile in the base repository, so assuming you have make and go: $ make

Example Usage

The below is taken from a Kubernetes pod specification;

spec:
  containers:
  - name: vault-side-kick
    image: quay.io/ukhomeofficedigital/vault-sidekick:v0.3.3
    args:
      - -output=/etc/secrets
      - -cn=pki:project1/certs/example.com:common_name=commons.example.com,revoke=true,update=2h
      - -cn=secret:secret/db/prod/username:file=.credentials
      - -cn=secret:secret/db/prod/password:retries=true
      - -cn=secret:secret/data/db/dev/username:file=.kv2credentials
      - -cn=aws:aws/creds/s3_backup_policy:file=.s3_creds
    volumeMounts:
      - name: secrets
        mountPath: /etc/secrets

The above equates to:

  • Write all the secrets to the /etc/secrets directory
  • Retrieve a dynamic certificate pair for me, with the common name: 'commons.example.com' and renew the cert when it expires automatically
  • Retrieve the two static secrets /db/prod/{username,password} and write them to .credentials and password.secret respectively
  • Retrieve the latest version of static secret /db/dev/username from a v2 kv store and write it to .kv2credentials
  • Apply the IAM policy, renew the policy when required and file the API tokens to .s3_creds in the /etc/secrets directory
  • Read the template at /etc/templates/db.tmpl, produce the content from Vault and write to /etc/credentials file

Authentication

An authentication file can be specified in either yaml of json format which contains a method field, indicating one of the authentication methods provided by vault i.e. userpass, token, github etc and then followed by the required arguments for that plugin.

If the required arguments for that plugin are not contained in the authentication file, fallbacks from environment variables are used. Environment variables are prefixed with VAULT_SIDEKICK, i.e. VAULT_SIDEKICK_USERNAME, VAULT_SIDEKICK_PASSWORD.

Kubernetes Authentication

The Kubernetes auth plugin supports the following environment variables:

  • VAULT_SIDEKICK_ROLE - The Vault role name against which to authenticate (REQUIRED)
  • VAULT_K8S_LOGIN_PATH - If your Kubernetes auth backend is mounted at a path other than kubernetes/ you will need to set this. Default /v1/auth/kubernetes/login
  • VAULT_K8S_TOKEN_PATH - If you mount in-pod service account tokens to a non-default path, you will need to set this. Default /var/run/secrets/kubernetes.io/serviceaccount/token

AppRole Authentication

The AppRole auth plugin supports the following configurations / environment variables:

  • role_id / VAULT_SIDEKICK_ROLE_ID - The approle role_id to authenticate with (REQUIRED)
  • secret_id / VAULT_SIDEKICK_SECRET_ID - The approle secret_id to authenticate with (REQUIRED)
  • login_path / VAULT_APPROLE_LOGIN_PATH - If your AppRole auth backend is mounted at a path other than approle/ you will need to set this. Default /v1/auth/approle/login

Secret Renewals

The default behaviour of vault-sidekick is not to renew a lease, but to retrieve a new secret and allow the previous to expire, in order ensure the rotation of secrets. If you don't want this behaviour on a resource you can override using resource options. For exmaple, your using the mysql dynamic secrets, you want to renew the secret not replace it

[jest@starfury vault-sidekick]$ build/vault-sidekick -cn=mysql:mysql/creds/my_database:fmt=yaml,renew=true
or an iam policy renewed every hour
[jest@starfury vault-sidekick]$ build/vault-sidekick -cn=aws:aws/creds/policy:fmt=yaml,renew=true,update=1h

Or you want to rotate the secret every 1h and revoke the previous one

[jest@starfury vault-sidekick]$ build/vault-sidekick -cn=aws:project/creds/my_s3_bucket:fmt=yaml,update=1h,revoke=true

The format is;

-cn=RESOURCE_TYPE:PATH:OPTIONS

The sidekick supports the following resource types: mysql, postgres, database, pki, aws, gcp, secret, cubbyhole, raw, cassandra and transit

Environment Variable Expansion

The resource paths can contain environment variables which the sidekick will resolve beforehand. A use case being, using a environment or domain within the resource e.g -cn=secret:secrets/myservice/${ENV}/config:fmt=yaml

Output Formatting

The following output formats are supported: json, yaml, ini, txt, cert, csv, bundle, env, credential, aws

Using the following at the demo secrets

[jest@starfury vault-sidekick]$ vault write secret/password this=is demo=value nothing=more
Success! Data written to: secret/password
[jest@starfury vault-sidekick]$ vault read secret/password
Key            	Value
lease_id       	secret/password/7908eceb-9bde-e7de-23da-96131505214a
lease_duration 	2592000
lease_renewable	false
demo           	value
nothing        	more
this           	is

In order to change the output format:

[jest@starfury vault-sidekick]$ build/vault-sidekick -cn=secret:secret/password:fmt=ini -logtostderr=true -dry-run
[jest@starfury vault-sidekick]$ build/vault-sidekick -cn=secret:secret/password:fmt=json -logtostderr=true -dry-run
[jest@starfury vault-sidekick]$ build/vault-sidekick -cn=secret:secret/password:fmt=yaml -logtostderr=true -dry-run

Format: 'cert' is less of a format of more file scheme i.e. is just extracts the 'certificate', 'issuing_ca' and 'private_key' and creates the three files FILE.{ca,key,crt}. The bundle format is very similar in the sense it similar takes the private key and certificate and places into a single file. 'credential' will attempt to decode a GCP credential file and 'aws' will write an AWS credentials file.

Resource Options

  • file: (filaname) by default all file are relative to the output directory specified and will have the name NAME.RESOURCE; the fn options allows you to switch names and paths to write the files
  • mode: (mode) overrides the default file permissions of the secret from 0664
  • create: (create) create the resource
  • update: (update) override the lease time of this resource and get/renew a secret on the specified duration e.g 1m, 2d, 5m10s
  • renew: (renewal) override the default behavour on this resource, renew the resource when coming close to expiration e.g true, TRUE
  • delay: (renewal-delay) delay the revoking the lease of a resource for x period once time e.g 1m, 1h20s
  • revoke: (revoke) revoke the old lease when you get retrieve a old one e.g. true, TRUE (default to allow the lease to expire and naturally revoke)
  • fmt: (format) allows you to specify the output format of the resource / secret, e.g json, yaml, ini, txt
  • exec (execute) execute's a command when resource is updated or changed
  • retries: (retries) the maximum number of times to retry retrieving a resource. If not set, resources will be retried indefinitely
  • jitter: (jitter) an optional maximum jitter duration. If specified, a random duration between 0 and jitter will be subtracted from the renewal time for the resource
  • ttl: (ttl) an optional ttl to use with the Vault PKI backend, should be specified as per the Vault PKI backend ttl resource (eg. 24h for one day). Hours are the largest suffix.

More Repositories

1

posters

Home Office Digital repository of posters covering different topics - research, access needs, accessibility, design.
2,897
star
2

keycloak-theme-govuk

A Keycloak theme using GOV.UK Template and Elements
FreeMarker
106
star
3

kd

Minimalistic kubernetes resources deployment tool with templating
Go
83
star
4

design-system

Home Office Design System
TypeScript
76
star
5

application-container-platform

Application Container Platform
HTML
63
star
6

docker-nginx-proxy

A generic WAF proxy layer
Shell
63
star
7

docker-clamav

ClamAV built on alpine
Shell
51
star
8

kb8or

CI Tool for deploying with kubernetes
Ruby
45
star
9

keycloak-utils

Shell
41
star
10

technical-service-requirements

Technical Service Requirements: This defines what a service needs i.e. "As a service: I need monitoring etc."
34
star
11

docker-logstash-kubernetes

Logstash container for pulling logs from Kubernetes
Shell
32
star
12

vaultctl

Hashcorp Vault provisoner
Go
30
star
13

smilodon

Manages attachment of EBS and ENI pairs in AWS EC2 auto scaling groups
Go
28
star
14

drone-trigger

Drone plugin for triggering downstream builds with custom parameters
Go
21
star
15

jenkins-docker-aws

Dockerised jenkins application which also includes:
Shell
19
star
16

vaultjks

Create JAVA keystore and truststore using certs from vault
Shell
19
star
17

docker-kafka

Kafka on Kubernetes
Shell
18
star
18

policy-admission

Kubernetes admission controller
Go
17
star
19

docker-zookeeper

Docker image for zookeeper to be run in Kubernetes
Shell
17
star
20

accessibility-posters

HTML
16
star
21

docker-landscaper-server

Docker landscape server container
Shell
14
star
22

s3secrets

Fetch encrypted files from S3 bucket and decrypt them using AWS KMS
Go
13
star
23

openvpn-authd

OpenVPN Access Service
Go
12
star
24

developer-induction

Contains inductions slides for developers
JavaScript
12
star
25

aws_usersync

Managing CoreOS Users by Using AWS Groups
Go
11
star
26

dq-packer-tableau-linux

Packer repo for Linux host for Tableau server
Shell
11
star
27

system-register

Java
10
star
28

docker-symmetricds

Docker image for one-way replication with SymmetricDS
Shell
9
star
29

clamav-http

Clamav instance with http api. To supersede https://github.com/UKHomeOffice/docker-clamav
Go
9
star
30

docker-elasticsearch

ElasticSearch with Kubernetes Discovery
Shell
9
star
31

api-guide-for-developers

8
star
32

cop-react-components

JavaScript
8
star
33

docker-vault

Vault in Kubernetes
Shell
8
star
34

vcloud_packer_images

A way of managing packer images for vcloud
Puppet
8
star
35

vaultconf

Configures vault with a given list of policies and users
Ruby
8
star
36

docker-filebeat-jenkins

Send build logs from Jenkins to Elasticsearch using Filebeat
Shell
7
star
37

drt-v2

Dynamic Response Tool v2
Scala
7
star
38

file-vault

JavaScript
6
star
39

lightweight-kube-etl

Simple start kubernetes job based on trigger from sns/sqs
JavaScript
6
star
40

dq-tf-ad

Terraform Module for painlessly having an AD provide authentication for instances
HCL
6
star
41

development_environment

Ansible provisioned hardened development environment for Ubuntu or CentOS
Shell
6
star
42

workflow-service

Workflow engine with Camunda and Cockpit
Java
6
star
43

dq-tf-internal-tableau

HCL
6
star
44

hocs-workflow

Java
6
star
45

tf-testrunner

tf-testrunner parses Terraform configuration to Python and then runs your tests.
Python
6
star
46

data-quality-profiler-and-rules-engine

Data Quality Profiler and Rules Engine
Scala
6
star
47

cop-react-form-renderer

A component library for rendering a Form from a JSON configuration
JavaScript
5
star
48

lev-api

Life Event Verification API
JavaScript
5
star
49

coe-infrastructure

to hold CoE infrastructure resources
5
star
50

kube-cfn-signal

Notify CloudFormation when Kubernetes node becomes ready
Go
5
star
51

platform-hub

Software powering the Application Container Platform Hub at the UK Home Office
Ruby
5
star
52

ho-style-prototype-kit

GOV.UK Prototype Kit - with Home Office styles
HTML
5
star
53

acp-tf-nlb

Terraform module to create network load balancers
HCL
5
star
54

email-api

An RESTful Email Templating API
Java
5
star
55

dq-tf-infra

HCL
5
star
56

hocs-serenity-automation

Automated Cucumber-JVM test pack for DECS built on the Serenity BDD framework
Java
5
star
57

passports-prototype-v2

Prototype for passport journeys. This is version 2 of the prototype.
HTML
4
star
58

acp-tf-s3

Terraform module to build S3 buckets
HCL
4
star
59

cerberus-service

Cerberus frontend service for cerberus-api
JavaScript
4
star
60

gds-react-components

React component ports of govuk-frontend nunjucks macros. Directly consuming govuk-frontend CSS & JS.
4
star
61

hocs-search

Hocs search microservice
Java
4
star
62

xgov-opensource-repo-scraper

JavaScript
4
star
63

hocs-info-service

Java
4
star
64

docker-neo4j-haproxy

Shell
4
star
65

vault-sidekick-pkcs12

sidekick to download certs from Vault and add the to a Java keystore
Shell
4
star
66

hocs-audit

Java
4
star
67

docker-centos-base

Dockerfile
4
star
68

hocs-casework

hocs caseworking service
Java
4
star
69

check_byod

Script to check OS is compliant with CESG guidelines
Shell
4
star
70

cfssl-sidekick

A sidekick used to request certificates from cfssl
Go
4
star
71

keycloak-knox

A Knox service that enables integration with keycloak, and also passing of JWTs to a backend server.
Java
4
star
72

taskflow

Proof of concept for workflow task management API - initially for ASL, but with a view towards re-use
JavaScript
4
star
73

rtp-akka-lib

Akka reusable functionality and Scala Spray functionality/template for general use
Scala
4
star
74

keycloakjs-redirect

JavaScript
4
star
75

hocs-frontend

JavaScript
4
star
76

egar-public-site-ui

Submit a GAR Public Site UI repository
JavaScript
4
star
77

AppealRightsExhausted

An online calculator to determine Appeal Rights Exhausted
JavaScript
4
star
78

OSX-Provisioning

For storing the scripts to provision new Macs
Shell
4
star
79

docker-mysql-maintenance

Shell
3
star
80

html-pdf-converter

Convert HTML to a PDF
JavaScript
3
star
81

passports-frontend-toolkit

Set of common UI patterns/styles for hmpo projects
JavaScript
3
star
82

passports-prototype

Passport journeys
HTML
3
star
83

ho-example-dashboard

Example smashing dashboard showcasing common widgets we use
Ruby
3
star
84

docker-confluence-server

Atlassian Confluence Server running as non-root
Dockerfile
3
star
85

formio-gds-template

Formio template for GDS
JavaScript
3
star
86

docker-nginx-tls

A sidekick container responsible for rotating certificates
Shell
3
star
87

git-workflow

A set of guidelines for using Git as a workflow tool
3
star
88

ubuntu-pxe

Ubuntu PXE deployment environment
Shell
3
star
89

hocs-notify

Hocs Notify Integration microservice
Java
3
star
90

workflow-ui

Front end for eForms application.
JavaScript
3
star
91

docker-nodejs-base

Base image for nodejs.
Shell
3
star
92

dronescheduler

Build Scheduler for drone
Go
3
star
93

docker-nginx

Minimal bare nginx docker image
Shell
3
star
94

modern-slavery

An app for reporting modern slavery
JavaScript
3
star
95

home-office-pattern-library

This pattern library contains patterns developed uniquely for Home Office services by researchers, designers and developers within Home Office Digital, Data and Technology.
JavaScript
3
star
96

cop-ui

Front end for main Border Force COP application.
JavaScript
3
star
97

lev-web

Life Event Verification Web-based User Interface
JavaScript
3
star
98

docker-solr

Dockerfile
3
star
99

cdp-nifi-processors

Custom NiFi processors for extracting data into a GDPR platform
Java
3
star
100

data-transfer

A python micro service for moving data between different locations
Python
3
star