• Stars
    star
    801
  • Rank 56,886 (Top 2 %)
  • Language
    Go
  • License
    Other
  • Created almost 10 years ago
  • Updated about 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A tool for securely storing secrets on S3 using Amazon KMS.

Sneaker

Build Status Apache V2 License

Setec Astronomy? Keynote Shogun.

sneaker is a utility for storing sensitive information on AWS using S3 and the Key Management Service (KMS) to provide durability, confidentiality, and integrity.

Secrets are stored on S3, encrypted with AES-256-GCM and single-use, KMS-generated data keys.

Table Of Contents

WARNING

This project has not been reviewed by security professionals. Its internals, data formats, and interfaces may change at any time in the future without warning.

Installing

go get -d -u github.com/codahale/sneaker
cd $GOPATH/src/github.com/codahale/sneaker
make install
sneaker version

Using

Configuring Access to AWS

sneaker requires access to AWS APIs, which means it needs a set of AWS credentials. It will look for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, the default credentials profile (e.g. ~/.aws/credentials), and finally any instance profile credentials for systems running on EC2 instances.

In general, if the aws command works, sneaker should work as well.

If you have multi-factor authentication enabled for your AWS account (and you should), you may need to provide a token via the AWS_SESSION_TOKEN environment variable.

If you're using IAM instance roles, you may need to set the AWS_REGION environment variable to the AWS region you're using (e.g. us-east-1).

Setting Up The Environment

Sneaker requires two things: a KMS master key and an S3 bucket.

You can create a KMS key via the AWS Console or using a recent version of aws. When you've created the key, store its ID (a UUID) in the SNEAKER_MASTER_KEY environment variable:

export SNEAKER_MASTER_KEY="9ed356fb-5f0f-4792-983d-91866faa3705"

As with the key, you can create an S3 bucket via the AWS Console or with the aws command. You can either use a dedicated bucket or use a directory in a common bucket, but we recommend you do two things:

  1. Use a Private ACL. In addition to the cryptographic controls of sneaker, access control is critical in preventing security breaches.

  2. Enable access logging, ideally to a tightly-controlled, secure bucket. While Amazon's CloudTrail provides audit logging for the vast majority of AWS services, it does not do so for S3 access.

Once you're done, set the SNEAKER_S3_PATH environment variable to the location where secrets should be stored (e.g. s3://bucket1/secrets/):

export SNEAKER_S3_PATH="s3://bucket1/secrets/"

(That will store the encrypted secrets in the bucket bucket1 prefixed with secrets/.)

Basic Operations

Once you've got sneaker configured, try listing the secrets:

sneaker ls

This will print out a table of all uploaded secrets. You haven't uploaded anything yet, so the table will be empty.

Let's create an example secret file and upload it:

echo "This is a secret!" > secret.txt
sneaker upload secret.txt example/secret.txt

This will use KMS to generate a random, 256-bit data key, encrypt the secret with AES-256-GCM, and upload the encrypted secret and an encrypted copy of the data key to S3. Running sneaker ls should display a table with the file in it.

If your file is so sensitive it shouldn't be stored on disk, using - instead of a filename will make sneaker read the data from STDIN.

You can download a single file:

sneaker download example/secret.txt secret.txt

Finally, you can delete the file:

sneaker rm example/secret.txt

Packing Secrets

To install a secret on a machine, you'll need to pack them into a tarball:

sneaker pack example/*,extra.txt example.tar.enc

This will perform the following steps:

  1. Download and decrypt all secrets matching any of the patterns: example/* or extra.txt.

  2. Package all the decrypted secrets into a TAR file in memory.

  3. Generate a new data key using KMS.

  4. Use the data key to encrypt the TAR file with AES-GCM.

  5. Write both the encrypted data key and the encrypted TAR file to example.tar.enc.

Using - as the output path will make sneaker write the data to STDOUT.

The result is safe to store and transmit -- only those with access to the Decrypt operation of the KMS key being used will be able to decrypt the data.

You can also use a different KMS key than your SNEAKER_MASTER_KEY when packing secrets by using the --key-id flag:

sneaker pack example/* example.tar.enc --key-id=deb207cd-d3a7-4777-aca0-01fbceb4c927

This allows you to unpack your secrets in environments which don't have access to the key used to store your secrets.

Unpacking Secrets

To unpack the secrets, run the following:

sneaker unpack example.tar.enc example.tar

This will perform the following steps:

  1. Read example.tar.enc.

  2. Extract the encrypted data key and encrypted TAR file.

  3. Use KMS to decrypt the data key.

  4. Decrypt the TAR file and write the result to example.tar.

Using - instead of a filename will make sneaker read the data from STDIN. Likewise, using - as the output path will make sneaker write the data to STDOUT. This allows you to pipe the output directly to a tar process, for example.

Encryption Contexts

KMS supports the notion of an Encryption Context: semi-structured data used in the encryption of data which is then required for resulting decryption operations to be successful.

sneaker uses the SNEAKER_MASTER_CONTEXT environment variable as the default encryption context for the secrets which are stored in S3. In addition, sneaker also includes the full S3 path, including bucket and prefix. Because of this, secrets in S3 cannot be renamed; they can only be deleted and re-uploaded.

Note: there is currently no way to change the contents of SNEAKER_MASTER_CONTEXT in place. If you want to change it, you'll need to download all your secrets and re-upload them with the new context.

For packing and unpacking secrets you can specify a different encryption context on the command line:

sneaker pack example/* secrets.tar.enc --context="hostname=web1.example.com,version=20"

That same context (hostname=web1.example.com,version=20) must be used to unpack those secrets:

sneaker unpack secrets.tar.enc secrets.tar --context="hostname=web1.example.com,version=20"

All data in the encryption contexts are logged via CloudTrail, which allows you to track when and where particular secrets are packed and unpacked.

Maintenance Operations

A common maintenance task is key rotation. To rotate the data keys used to encrypt the secrets, run sneaker rotate. It will download and decrypt each secret, generate a new data key, and upload a re-encrypted copy.

To rotate the KMS key used for each secret, simply specify a different SNEAKER_MASTER_KEY and run sneaker rotate.

Implementation Details

All data is encrypted with AES-256-GCM using random KMS data keys and random nonces. The ID of the KMS key is used as authenticated data.

The final result is the concatentation of the following:

  • A four-byte header of the length of the encrypted KMS data key, in bytes, in network order.

  • The encrypted KMS data key, verbatim. (This is an opaque Amazon format which includes the key ID.)

  • The AES-256-GCM ciphertext and tag of the secret.

Architecture

Sneaker Architecture

Threat Model

The threat model is defined in terms of what each possible attacker can achieve. The list is intended to be exhaustive, i.e. if an entity can do something that is not listed here then that should count as a break of Sneaker.

In broad strokes, the confidentiality and integrity of content stored with Sneaker is predicated on the integrity of Sneaker, the confidentiality and integrity of KMS, and the integrity of S3.

Assumptions

  • The user must act reasonably and in their best interest. They must not reveal secrets or allow unauthorized access to secrets.

  • The user must run a copy of Sneaker which has not been suborned.

  • The user's computer must function correctly and not be compromised by malware.

  • Communications with Amazon have confidentiality and integrity ensured by the use of TLS.

  • Key Management Service is reasonably secure: its keys will not be compromised, its random numbers are unguessable to adversaries, its cryptographic algorithms are correct.

  • The authentication and access control functionality of both KMS and S3 are secure.

  • AES-256 and GCM's security guarantees are valid.

Threats From A KMS Compromise

An attacker who suborns KMS can:

  • Create forged secret packages.
  • Decrypt packaged tarballs.
  • Deny the ability to decrypt secrets, either temporarily or permanently.

Threats From An S3 Compromise:

An attacker who suborns S3 can:

  • Delete or modify secrets such that they are no longer valid.

Threats From Seizure Or Compromise Of The User's Computer

An attacker who physically seizes the user's computer (or compromises the user's backups) or otherwise compromises it can:

  • Recover AWS credentials and pose as the user. If multi-factor authentication is not enabled, this would allow the attacker to extract all secrets, modify secrets, etc.

More Repositories

1

metrics

This is not the Java library.
Go
451
star
2

jerkson

[ABANDONED] The Scala applewood bacon to Jackson's chicken breast: JSON cordon bleu.
Scala
279
star
3

shamir

A Java implementation of Shamir's Secret Sharing algorithm over GF(256).
Java
208
star
4

usl4j

A reasonably complete implementation of the Universal Scalability Law model.
Java
201
star
5

lunk

lunk provides a set of tools for structured logging in the style of Google's Dapper or Twitter's Zipkin
Go
127
star
6

sss

A pure Go implementation of Shamir's Secret Sharing algorithm over GF(256)
Go
103
star
7

http-handlers

Some handy HTTP handlers for Go.
Go
85
star
8

ministat

A small tool to do the statistics legwork on benchmarks etc.
Shell
75
star
9

tinystat

A Go library and CLI tool for evaluating whether two or more sets of measurements are statistically different.
Go
69
star
10

sketchy

A small set of useful probabilistic data structures.
Rust
69
star
11

logula

[ABANDONED] A Scala library which provides a sane log output format and an easy-to-use mixin for adding logging to your code.
Scala
51
star
12

chacha20

A pure Go implementation of the ChaCha20 stream cipher.
Go
46
star
13

charlie

charlie provides a fast, safe, stateless mechanism for adding CSRF protection to web applications.
Go
45
star
14

buster

A Go library which provides a re-usable framework for load-testing things.
Go
43
star
15

fast-uuid

A fast random UUID generator.
Java
41
star
16

rfc6979

A Go implementation of RFC 6979's deterministic DSA/ECDSA signature scheme.
Go
39
star
17

blake2

A BLAKE2 wrapper for Go.
Go
37
star
18

xsalsa20poly1305

A pure Java implementation of XSalsa20Poly1305 authenticated encryption.
Java
34
star
19

aes-gcm-siv

A Java implementation of AES-GCM-SIV (RFC 8452).
Java
33
star
20

jersey-scala

[ABANDONED] A Scala library which adds support to native Scala types in Jersey applications.
Scala
32
star
21

guava-cache-clj

A Clojure wrapper for Guava caches.
Clojure
30
star
22

usl

A Go library and CLI tool which simplifies calculation of Universal Scalability Law parameters given system measurements.
Go
30
star
23

horcrux

A pure Go implementation of security question style password recovery that preserves end-to-end cryptographic security.
Go
28
star
24

soy-clj

An idiomatic Clojure wrapper for Google Closure templates.
Clojure
28
star
25

fig

[ABANDONED] A small Scala library for using JSON-based configuration files.
Scala
27
star
26

time-id

Generates 27-character, time-ordered, k-sortable, URL-safe, globally unique identifiers.
Java
26
star
27

grpc-proxy

A gRPC service which proxies requests to an HTTP server.
Java
25
star
28

chacha20poly1305

An implementation of the chacha20poly1305 AEAD construction.
Go
25
star
29

passpol

A Java library for validating passwords against NIST SP-800-63B requirements.
Java
25
star
30

sskg

A Go implementation of a fast, tree-based Seekable Sequential Key Generator.
Go
24
star
31

pcg.rs

A Rust implementation of the PCG PRNG.
Rust
23
star
32

veil

Veil is an incredibly experimental hybrid cryptosystem for sending and receiving confidential, authentic multi-recipient messages which are indistinguishable from random noise by an attacker.
Rust
21
star
33

ruby-gsl

[ABANDONED] New development on the Ruby bindings for the GNU Scientific Library
C
20
star
34

guild

[ABANDONED] Simple, fast actors for Scala.
Scala
20
star
35

xoroshiro.rs

A Rust implementation of the Xoroshiro128+ PRNG.
Rust
19
star
36

colorbot

A Go library which determines the dominant colors in an image.
Go
19
star
37

yellhole

Where else you gonna scream?
Rust
18
star
38

docker-rebase

A tool to rebase Docker images.
Go
18
star
39

og

Java
18
star
40

veil-go

Stupid crypto tricks
Go
18
star
41

nanostat

A Rust library and CLI tool for evaluating whether two or more sets of measurements are statistically different.
Rust
16
star
42

shore

[ABANDONED] What makes Jersey fun.
Java
16
star
43

yoink

[ABANDONED] A Scala library which wraps Clojure's immutable, persistent collections.
Scala
16
star
44

healthchecks

A Go package which exposes health check results as expvars.
Go
15
star
45

faster-builder

[ABANDONED] A drop-in replacement for Builder::XmlMarkup which uses libxml for speed and security.
Ruby
15
star
46

cassie

See https://github.com/twitter/cassie for an actual, maintained version.
Scala
15
star
47

emacs.d

My Emacs configuration.
Emacs Lisp
14
star
48

safecookie

A Go library which provides simple, strong protection for cookies.
Go
14
star
49

dotfiles

Shell
13
star
50

redact

A Go package which allows you to easily redact sensitive flag values to prevent them from being exposed as expvars or via debugging endpoints.
Go
12
star
51

etm

A Go implementation of AES/CBC/HMAC-based AEAD constructions.
Go
11
star
52

common-pom

A common Maven POM for my various bits of Java software.
11
star
53

aesnicheck

A Go library for checking AES-NI support.
Go
10
star
54

kmssig

A Go utility and library to sign and verify files using AWS’s Key Management Service.
Go
10
star
55

usl-rs

A Rust crate and CLI tool which simplifies calculation of Universal Scalability Law parameters given system measurements.
Rust
9
star
56

reynolds

[ABANDONED] Your serialization format sucks.
Scala
9
star
57

jumphash

A Rust implementation of the Jump Consistent Hash algorithm.
Rust
9
star
58

trove-scala

[ABANDONED] Scala wrappers for the Trove primitive collections.
Scala
9
star
59

hadoop-streaming

[ABANDONED] Support libraries for writing Hadoop Streaming-compatible map/reduce tasks.
Python
8
star
60

balloonhash

A Java implementation of the Balloon Hash algorithm.
Java
8
star
61

ropen

[ABANDONED] A process execution library which doesn't suck.
Ruby
8
star
62

veil-java

Stupid crypto tricks
Java
7
star
63

baconhand

[ABANDONED] A useful slap in the face with baconhand.
Ruby
7
star
64

vlock

[ABANDONED] Vector clocks in Scala
Scala
7
star
65

lockstitch

Lockstitch is an incremental, stateful cryptographic primitive for symmetric-key cryptographic operations (e.g. hashing, encryption, message authentication codes, and authenticated encryption) in complex protocols.
Rust
7
star
66

mocko

A simple mocking library for Clojure.
Clojure
6
star
67

wasp

[ABANDONED] An append-only transaction log for Scala
Scala
6
star
68

kht

A Go implementation of a keyed hash tree.
Go
6
star
69

hrw

A Go implementation of Highest Random Weight hashing.
Go
6
star
70

spritz

A pure Go implementation of the Spritz stream cipher.
Go
5
star
71

u2f-clj

A library for implementing FIDO U2F multi-factor authentication.
Clojure
5
star
72

totp-clj

A Clojure library for TOTP multi-factor authentication.
Clojure
5
star
73

geoip

A simple Go wrapper for libGeoIP.
Go
5
star
74

gimli

A Java implementation of the Gimli cryptographic permutation and hash algorithm.
Java
4
star
75

sss.rs

Shamir’s Secret Sharing for Rust
Rust
4
star
76

longmapper

A bijective mapping for 64-bit integers.
Java
4
star
77

simple_do

[ABANDONED] A simple wrapper around DataObjects.
Ruby
4
star
78

makwa-go

A Go implementation of the Makwa password hashing algorithm.
Go
4
star
79

deyaml

Life’s too short to be a YAML farmer.
Go
4
star
80

hs-shamir

A Haskell implementation of Shamir’s Secret Sharing Scheme over GF(256).
Haskell
4
star
81

jdbc-perf-wrapper

[ABANDONED] A JDBC performance tracking wrapper.
Java
4
star
82

zk-cds

Hypothetical design of a zero-knowledge contact discovery system
Rust
4
star
83

tmpmysqld

tmpmysqld allows you to spin up temporary instances of mysqld in Go for testing purposes.
Go
4
star
84

path_mapper

[ABANDONED] TOTALLY NOT EVEN CLOSE TO WORKING YET. An efficient URL-to-options mapper.
Python
3
star
85

areion

A Rust implementation of the Areion256 and Areion512 permutations.
Rust
3
star
86

testdb

Go
3
star
87

grump

do not use this
Go
3
star
88

sfearthquakes

@sfqearthquakes
Java
3
star
89

voldemort-sets

[ABANDONED] A collection of view classes for the Voldemort key/value store which allow for the storage of sets of numbers.
Java
3
star
90

cpseg

Go
3
star
91

merb_simple_do

[ABANDONED] A Merb plugin for using DataObjects::Simple (simple_do).
Ruby
3
star
92

yrgourd

yrgourd uses Lockstitch to establish mutually-authenticated, forward-secure, confidential, high-performance connections. Like a toy Wireguard.
Rust
3
star
93

aead-stream

Java
3
star
94

xoodoo-p

High-performance, pure-Rust implementation of the Xoodoo-p permutation.
Rust
2
star
95

hudson_deployer

[ABANDONED] A Capistrano-based solution for deploying Java projects from Hudson.
Ruby
2
star
96

gubbins

Go
2
star
97

esi-blog

[ABANDONED] A sample application for playing around with some ESI ideas I had.
Ruby
2
star
98

rsa_pprf

A puncturable PRF based on the RSA accumulator.
Rust
2
star
99

eventflow

A high-throughput, low-latency, GCP-native, analytical event pipeline.
Java
2
star
100

kms-pass

Java
2
star