cfn-flow
cfn-flow is a command-line tool for developing AWS CloudFormation templates and deploying stacks.
It provides a simple, standard, and flexible process for using CloudFormation, ideal for DevOps-style organizations.
Table of Contents
Opinions
cfn-flow introduces a consistent, convenient workflow that encourages good template organization
and deploy practices.
- Optimize for happiness. The workflow should be easy & enjoyable to use.
- Optimize for onboarding. The workflow should be simple to learn, understand, & debug.
- Auditable changes. Know who changed what when & why. Leverage git history.
- Immutable releases. The code in a release never changes. To make a change, launch a new stack.
The features & implementation of cfn-flow itself must also be simple. This follows the Unix philosophy of "worse is
better". cfn-flow values a simple design and implementation, and being composable with other workflows over handling every edge case out of the box.
See this introductory blog post for our motivation behind cfn-flow.
Installation
Via rubygems:
gem install cfn-flow
The git command is also needed.
Key concepts
cfn-flow works from a directory containing a cfn-flow.yml config file, and a CloudFormation template.
Presumably your app code is in the same directory, but it doesn't have to be.
There are two key concepts for cfn-flow: services and environments.
Services
A service comprises a set of resources that change together.
Each service has its own cfn-flow.yml config file. A service
can be instantiated as several distinct environments.
For example, a WebApp service could have a CloudFormation template that
creates an ELB, LaunchConfig, and AutoScalingGroup resources.
All the resources in a service change together. Deploying the WebApp
service to an environment will create a new ELB, LaunchConfig, and AutoScalingGroup.
Resources that do not change across deploys are not part of the service (from
cfn-flow's perspective).
Say all WebApp EC2 servers connect to a long-running RDS database. That
database is not part of the cfn-flow service because it is re-used across
deploys. The database is a backing resource the service uses; not part
of the service itself.
Environments
An environment is an particular instantiation of a service. For example, you
could deploy your WebApp service to both a development and production environment.
cfn-flow is designed to support arbitrary environments like git supports
arbitrary branches.
Pro tip: Use the CFN_FLOW_ENVIRONMENT environment variable in
cfn-flow.yml config to use the environment in your template parameters.
See Configuration for examples.
Deploying
Deployments consist of launching a new stack in a particular environment, then shutting down the old stack. For example:
cfn-flow deploy ENVIRONMENT --cleanup
This follows the red/black or blue/green deployment pattern.
After verifying the new stack is working correctly, the deployer is expected to delete the old stack.
To roll back a bad deploy, simply delete the new stack, while the old stack is running.
Although CloudFormation supports updating existing stacks, cfn-flow prefers
launching immutable stacks. Stack updates are more difficult to test than new stacks; and there's less chance of a deployment error disrupting or breaking important resources.
AWS credentials
Set your AWS credentials so they can be found by the AWS SDK for Ruby (details here), e.g. using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
Configuration
cfn-flow looks for ./cfn-flow.yml for stack and template configuration.
You can override this path by setting the CFN_FLOW_CONFIG_PATH environment
variable to another path.
Here's a minimal cfn-flow.yml config file:
# Required service name
service: MyService
# Minimal configuration for launching the stack.
stack:
# Stack name uses embedded ruby to support dynamic values
stack_name: MyService-<%= Time.now.to_i %>
# Required: *either* template_url or template_body
# NB: template_body is a local path to the template
template_body: path/to/template.json
# Alternatively:
# template_url: https://MyS3Bucket.s3.amazonaws.com/MyPrefix/release/abc123/template.jsonAnd here's a maximal config file:
# Example cfn-flow.yml
service: MyService
# Set the AWS region here to override or avoid setting the AWS_REGION env var
region: us-east-1
##
# Templates
#
# These define where templates will get published.
# $ cfn-flow publish --release my-cfn-template.json
# Published url: https://MyS3Bucket.s3.amazonaws.com/My/S3/Prefix/<git sha>/my-cfn-template.json
templates:
bucket: MyS3Bucket
s3_prefix: 'My/S3/Prefix'
##
# Stacks
#
# These are the arguments passed when launching a new stack.
# It's nearly identical to the create_stack args in the ruby sdk, except
# parameters and tags are hashes. See http://amzn.to/1M0nBuq
stack:
# Use the CFN_FLOW_ENVIRONMENT var & git sha in stack name
stack_name: MyService-<%= ENV['CFN_FLOW_ENVIRONMENT'] %>-<%= `git rev-parse --short HEAD`.chomp %>
# NB: template_body is a local path to the template
template_body: path/to/template.yml
template_url: http://...
parameters:
# Your parameters, e.g.:
vpcid: vpc-1234
ami: ami-abcd
##
# Use outputs from other stacks
# This set the `load_balancer` parameter to the value of the
# `elbname` output of `my-elb-stack`
load_balancer:
stack: my-elb-stack
output: elbname
# If you don't specify the output name, it's assumed to be same
# as the parameter key:
ssh_security_group:
stack: my-bastion-stack
disable_rollback: true,
timeout_in_minutes: 1,
notification_arns: ["NotificationARN"],
capabilities: ["CAPABILITY_IAM"], # This stack does IAM stuff
on_failure: "DO_NOTHING", # either DO_NOTHING, ROLLBACK, DELETE
stack_policy_body: "StackPolicyBody",
stack_policy_url: "StackPolicyURL",
tags:
# Whatever you want.
# Note that `cfn-flow` automatically adds two tags: `CfnFlowService` and `CfnFlowEnvironment`
TagKey: TagValue
# Who launched this stack
Deployer: <%= ENV['USER'] %>
# Tag production and development environments for accounting
BillingType: <%= ENV['CFN_FLOW_ENVIRONMENT'] == 'production' ? 'production' : 'development' %>UX improvements
cfn-flow includes a few developer-friendly features:
YAML > JSON
cfn-flow lets you write templates in either JSON or
YAML. YAML is a superset of JSON that allows a terser,
less cluttered syntax, inline comments, and code re-use with anchors (like
variables). YAML templates are transparently converted to JSON when uploaded to
S3 for use in CloudFormation stacks.
Note that you can use JSON snippets inside YAML templates. JSON is always valid YAML.
Embedded ruby in cfn-flow.yml
To allow dynamic/programmatic attributes, use
ERB in cfn-flow.yml. For example:
stack:
name: my-stack-<%= Time.now.to_i %>
...
parameters:
git_sha: <%= `git rev-parse --verify HEAD`.chomp %>Use stack outputs as parameters
cfn-flow lets you easily reference stack outputs as parameters for new stacks.
# cfn-flow.yml
stack:
parameters:
# Set my-param to the `my-param` output of `another-stack`
my-param:
stack: another-stack
# Set my-param to the `my-output` output of `another-stack`
my-param:
stack: another-stack
output: my-outputUsage
Getting help:
# Get help
cfn-flow help
cfn-flow help COMMAND
# E.g.:
cfn-flow help deploy
Launch a CloudFormation stack:
cfn-flow deploy production
Working with stacks
cfn-flow automatically sets two tags on any stack it launches:
| Name | Example value |
|---|---|
| CfnFlowService | myapp |
| CfnFlowEnvironment | production |
These tags let cfn-flow associate stacks back to services & environments.
Deploy (launch) a stack
cfn-flow deploy ENVIRONMENT
Launches a stack in ENVIRONMENT. E.g. cfn-flow deploy production
Add the --cleanup option to be prompted to shut down other stacks in the environment.
List stacks for your service or environment
cfn-flow list [ENVIRONMENT]
Show all stacks running in your service, or just in an ENVIRONMENT.
# For example:
$ cfn-flow list production
myapp-production-aaa (CREATE_COMPLETE)
myapp-production-bbb (CREATE_FAILED)
Inspect a stack
cfn-flow show STACK
Show the status of STACK.
Show stack events
cfn-flow events STACK
List events for STACK
Use the --poll option to poll for new events until the stack status is no
longer *_IN_PROGRESS
Delete a stack
cfn-flow delete STACK
Deletes a stack.
# For example:
$ cfn-flow delete myapp-production-aaa
Common workflows
Deploying to production
# Launch a new stack for the current git commit
$ cfn-flow deploy production
Launching stack myapp-production-abc123
# ... wait for it to be ready
# See the other stacks
$ cfn-deploy list production
myapp-production-abc123 CREATE_COMPLETE
myapp-production-xyz987 CREATE_COMPLETE
# Shut down the old stack
$ cfn-flow delete myapp-production-xyz987
Launching a development environment
Launch a new stack for myenv environment
cfn-flow deploy myenv
Working with templates
Validate templates
cfn-flow validate TEMPLATE [...]
Validates CloudFormation templates; does not persist to S3.
# For example:
$ cfn-flow validate path/to/template.yml
Publish templates to S3
cfn-flow publish TEMPLATE [...]
Publish templates to S3 with immutable release names, or overwrite "dev names" for quicker testing.
Note: Publishing to S3 is only needed if you want to use nested stack resources, (that is, stacks that include other stacks).
# For example:
$ cfn-flow publish path/to/template.yml
# validates & uploads templates to dev path
# Env var CFN_FLOW_DEV_NAME=aaron
# E.g. https://mybucket.s3.amazonaws.com/myprefix/dev/aaron/mytemplate.yml
$ cfn-flow upload --release
# validates & uploads templates for current git sha
# E.g. https://mybucket.s3.amazonaws.com/myprefix/deadbeef/mytemplate.yml
$ cfn-flow upload --release=v1.0.0
# Upload templates for an arbitrary release name
# E.g. https://mybucket.s3.amazonaws.com/myprefix/v1.0.0/mytemplate.yml
License
Copyright Kickstarter, Inc.
Released under an MIT License.