• Stars
    star
    170
  • Rank 215,537 (Top 5 %)
  • Language
    Ruby
  • Created over 9 years ago
  • Updated about 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A practical workflow for AWS CloudFormation

cfn-flow

cfn-flow is a command-line tool for developing AWS CloudFormation templates and deploying stacks.

It provides a simple, standard, and flexible process for using CloudFormation, ideal for DevOps-style organizations.

Table of Contents

Opinions

cfn-flow introduces a consistent, convenient workflow that encourages good template organization and deploy practices.

  1. Optimize for happiness. The workflow should be easy & enjoyable to use.
  2. Optimize for onboarding. The workflow should be simple to learn, understand, & debug.
  3. Auditable changes. Know who changed what when & why. Leverage git history.
  4. Immutable releases. The code in a release never changes. To make a change, launch a new stack.

The features & implementation of cfn-flow itself must also be simple. This follows the Unix philosophy of "worse is better". cfn-flow values a simple design and implementation, and being composable with other workflows over handling every edge case out of the box.

See this introductory blog post for our motivation behind cfn-flow.

Installation

Via rubygems:

gem install cfn-flow

The git command is also needed.

Key concepts

cfn-flow works from a directory containing a cfn-flow.yml config file, and a CloudFormation template. Presumably your app code is in the same directory, but it doesn't have to be.

There are two key concepts for cfn-flow: services and environments.

Services

A service comprises a set of resources that change together. Each service has its own cfn-flow.yml config file. A service can be instantiated as several distinct environments.

For example, a WebApp service could have a CloudFormation template that creates an ELB, LaunchConfig, and AutoScalingGroup resources.

All the resources in a service change together. Deploying the WebApp service to an environment will create a new ELB, LaunchConfig, and AutoScalingGroup.

Resources that do not change across deploys are not part of the service (from cfn-flow's perspective). Say all WebApp EC2 servers connect to a long-running RDS database. That database is not part of the cfn-flow service because it is re-used across deploys. The database is a backing resource the service uses; not part of the service itself.

Environments

An environment is an particular instantiation of a service. For example, you could deploy your WebApp service to both a development and production environment.

cfn-flow is designed to support arbitrary environments like git supports arbitrary branches.

Pro tip: Use the CFN_FLOW_ENVIRONMENT environment variable in cfn-flow.yml config to use the environment in your template parameters. See Configuration for examples.

Deploying

Deployments consist of launching a new stack in a particular environment, then shutting down the old stack. For example:

cfn-flow deploy ENVIRONMENT --cleanup

This follows the red/black or blue/green deployment pattern.

After verifying the new stack is working correctly, the deployer is expected to delete the old stack.

To roll back a bad deploy, simply delete the new stack, while the old stack is running.

Although CloudFormation supports updating existing stacks, cfn-flow prefers launching immutable stacks. Stack updates are more difficult to test than new stacks; and there's less chance of a deployment error disrupting or breaking important resources.

AWS credentials

Set your AWS credentials so they can be found by the AWS SDK for Ruby (details here), e.g. using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

Configuration

cfn-flow looks for ./cfn-flow.yml for stack and template configuration. You can override this path by setting the CFN_FLOW_CONFIG_PATH environment variable to another path.

Here's a minimal cfn-flow.yml config file:

# Required service name
service: MyService

# Minimal configuration for launching the stack.
stack:
  # Stack name uses embedded ruby to support dynamic values
  stack_name: MyService-<%= Time.now.to_i %>
  # Required: *either* template_url or template_body
  # NB: template_body is a local path to the template
  template_body: path/to/template.json
  # Alternatively:
  # template_url: https://MyS3Bucket.s3.amazonaws.com/MyPrefix/release/abc123/template.json

And here's a maximal config file:

# Example cfn-flow.yml

service: MyService

# Set the AWS region here to override or avoid setting the AWS_REGION env var
region: us-east-1

##
# Templates
#
# These define where templates will get published.
#   $ cfn-flow publish --release my-cfn-template.json
#   Published url: https://MyS3Bucket.s3.amazonaws.com/My/S3/Prefix/<git sha>/my-cfn-template.json
templates:
  bucket: MyS3Bucket
  s3_prefix: 'My/S3/Prefix'

##
# Stacks
#
# These are the arguments passed when launching a new stack.
# It's nearly identical to the create_stack args in the ruby sdk, except
# parameters and tags are hashes. See http://amzn.to/1M0nBuq

stack:
  # Use the CFN_FLOW_ENVIRONMENT var & git sha in stack name
  stack_name: MyService-<%= ENV['CFN_FLOW_ENVIRONMENT'] %>-<%= `git rev-parse --short HEAD`.chomp %>
    # NB: template_body is a local path to the template
    template_body: path/to/template.yml
    template_url: http://...
    parameters:
      # Your parameters, e.g.:
      vpcid: vpc-1234
      ami: ami-abcd

      ##
      # Use outputs from other stacks

      # This set the `load_balancer` parameter to the value of the
      # `elbname` output of `my-elb-stack`
      load_balancer:
        stack: my-elb-stack
        output: elbname

      # If you don't specify the output name, it's assumed to be same
      # as the parameter key:
      ssh_security_group:
        stack: my-bastion-stack

    disable_rollback: true,
    timeout_in_minutes: 1,
    notification_arns: ["NotificationARN"],
    capabilities: ["CAPABILITY_IAM"], # This stack does IAM stuff
    on_failure: "DO_NOTHING", # either DO_NOTHING, ROLLBACK, DELETE
    stack_policy_body: "StackPolicyBody",
    stack_policy_url: "StackPolicyURL",
    tags:
      # Whatever you want.
      # Note that `cfn-flow` automatically adds two tags: `CfnFlowService` and `CfnFlowEnvironment`
      TagKey: TagValue
      # Who launched this stack
      Deployer: <%= ENV['USER'] %>
      # Tag production and development environments for accounting
      BillingType: <%= ENV['CFN_FLOW_ENVIRONMENT'] == 'production' ?  'production' : 'development' %>

UX improvements

cfn-flow includes a few developer-friendly features:

YAML > JSON

cfn-flow lets you write templates in either JSON or YAML. YAML is a superset of JSON that allows a terser, less cluttered syntax, inline comments, and code re-use with anchors (like variables). YAML templates are transparently converted to JSON when uploaded to S3 for use in CloudFormation stacks.

Note that you can use JSON snippets inside YAML templates. JSON is always valid YAML.

Embedded ruby in cfn-flow.yml

To allow dynamic/programmatic attributes, use ERB in cfn-flow.yml. For example:

stack:
  name: my-stack-<%= Time.now.to_i %>
  ...
  parameters:
    git_sha: <%= `git rev-parse --verify HEAD`.chomp %>

Use stack outputs as parameters

cfn-flow lets you easily reference stack outputs as parameters for new stacks.

# cfn-flow.yml
stack:
  parameters:
    # Set my-param to the `my-param` output of `another-stack`
    my-param:
      stack: another-stack

    # Set my-param to the `my-output` output of `another-stack`
    my-param:
      stack: another-stack
      output: my-output

Usage

Getting help:

# Get help
cfn-flow help

cfn-flow help COMMAND
# E.g.:
cfn-flow help deploy

Launch a CloudFormation stack:

cfn-flow deploy production

Working with stacks

cfn-flow automatically sets two tags on any stack it launches:

Name Example value
CfnFlowService myapp
CfnFlowEnvironment production

These tags let cfn-flow associate stacks back to services & environments.

Deploy (launch) a stack

cfn-flow deploy ENVIRONMENT

Launches a stack in ENVIRONMENT. E.g. cfn-flow deploy production

Add the --cleanup option to be prompted to shut down other stacks in the environment.

List stacks for your service or environment

cfn-flow list [ENVIRONMENT]

Show all stacks running in your service, or just in an ENVIRONMENT.

# For example:
$ cfn-flow list production

myapp-production-aaa (CREATE_COMPLETE)
myapp-production-bbb (CREATE_FAILED)

Inspect a stack

cfn-flow show STACK

Show the status of STACK.

Show stack events

cfn-flow events STACK

List events for STACK

Use the --poll option to poll for new events until the stack status is no longer *_IN_PROGRESS

Delete a stack

cfn-flow delete STACK

Deletes a stack.

# For example:
$ cfn-flow delete myapp-production-aaa

Common workflows

Deploying to production

# Launch a new stack for the current git commit
$ cfn-flow deploy production
Launching stack myapp-production-abc123
# ... wait for it to be ready

# See the other stacks
$ cfn-deploy list production

myapp-production-abc123 CREATE_COMPLETE
myapp-production-xyz987 CREATE_COMPLETE

# Shut down the old stack
$ cfn-flow delete myapp-production-xyz987

Launching a development environment

Launch a new stack for myenv environment

cfn-flow deploy myenv

Working with templates

Validate templates

cfn-flow validate TEMPLATE [...]

Validates CloudFormation templates; does not persist to S3.

# For example:
$ cfn-flow validate path/to/template.yml

Publish templates to S3

cfn-flow publish TEMPLATE [...]

Publish templates to S3 with immutable release names, or overwrite "dev names" for quicker testing.

Note: Publishing to S3 is only needed if you want to use nested stack resources, (that is, stacks that include other stacks).

# For example:
$ cfn-flow publish path/to/template.yml
# validates & uploads templates to dev path
# Env var CFN_FLOW_DEV_NAME=aaron
# E.g. https://mybucket.s3.amazonaws.com/myprefix/dev/aaron/mytemplate.yml

$ cfn-flow upload --release
# validates & uploads templates for current git sha
# E.g. https://mybucket.s3.amazonaws.com/myprefix/deadbeef/mytemplate.yml

$ cfn-flow upload --release=v1.0.0
# Upload templates for an arbitrary release name
# E.g. https://mybucket.s3.amazonaws.com/myprefix/v1.0.0/mytemplate.yml

License

Copyright Kickstarter, Inc.

Released under an MIT License.

More Repositories

1

ios-oss

Kickstarter for iOS. Bring new ideas to life, anywhere.
Swift
8,233
star
2

android-oss

Kickstarter for Android. Bring new ideas to life, anywhere.
Kotlin
5,723
star
3

kickstarter-autodesk-3d

Kickstarter Autodesk 3D Printer Evaluation
901
star
4

Kickstarter-Prelude

A collection of useful Swift tools that are used in the Kickstarter apps.
Swift
483
star
5

Kickstarter-ReactiveExtensions

A collection of extensions to the ReactiveSwift framework.
Swift
196
star
6

ipcat-ruby

Ruby port of client9/ipcat
Ruby
139
star
7

event-sourcing-rails-todo-app-demo

A demo app to illustrate Kickstarter's Event Sourcing experiment
Ruby
83
star
8

native-docs

Documentation for native application development at Kickstarter.
76
star
9

laptop

A shell script which turns your OS X laptop into an awesome web development machine
Shell
67
star
10

build-ubuntu-ami

Create custom Ubuntu AMIs the hard (secure) way
Ruby
66
star
11

amazon_flex_pay

API for Amazon's Flexible Payments Service
Ruby
50
star
12

kumquat

Kumquat is a Rails Engine for rendering RMarkdown files.
Ruby
43
star
13

fullscreen_slideshow

A generic version of the code that runs Kickstarter's 2013 Year In Review: https://www.kickstarter.com/year/2013
JavaScript
41
star
14

ruby-homograph-detector

🕵️‍♀️🕵️‍♂️ Ruby gem for determining whether a given URL is considered an IDN homograph attack
Ruby
28
star
15

replica_pools

A leader/replica setup with separate replica pools, originally forked from schoefmax/multi_db
Ruby
24
star
16

mail-x_smtpapi

Adds SendGrid X-SMTPAPI header support to Mail
Ruby
22
star
17

telekinesis

⚡ A JRuby Kinesis Client ⚡
Ruby
20
star
18

caption_crunch

Parsers for WebVTT and other subtitle files
Ruby
19
star
19

serverless-approle-manager

A lambda triggered at resource creation by a CloudFormation template that manages approle auth in Vault.
JavaScript
18
star
20

configs

Ruby library for managing YAML config
Ruby
17
star
21

dropwizard-influxdb-reporter

Dropwizard Integrations for InfluxDB.
Java
16
star
22

docker-s3-sync

Docker container to sync SSH authorized keys from S3
Ruby
16
star
23

egads

Extensible Git Archive Deploy Strategy
Ruby
11
star
24

ruby-maybe

A Ruby implementation of the Maybe type
Ruby
10
star
25

TCI-chrome-extension

A new tab Chrome extension that pulls quotes from The Creative Independent
HTML
10
star
26

geokit-geoip

Our GeoKit module for using a local (proprietary) Maxmind GeoIP database
Ruby
8
star
27

cog-consul-oss

HashiCorp Consul KV read/write bundle for Cog.
Ruby
7
star
28

jquery-clipster

JavaScript
7
star
29

ember-documentary

An ambitious documentation generator for ember-cli apps.
JavaScript
6
star
30

ios-ksapi

Moved! Incorporated into https://github.com/kickstarter/ios-oss
Swift
3
star
31

capistrano-hivequeen

Capistrano extensions for interacting with hivequeen
Ruby
2
star
32

hash_deep_reject

Microgem for a ruby Hash#deep_reject method
Ruby
2
star
33

homebrew-repo

Homebrew formulas
Ruby
1
star