• Stars
    star
    57
  • Rank 522,947 (Top 11 %)
  • Language
    Ruby
  • License
    MIT License
  • Created over 8 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

CloudFormation made easy

Humidifier

Build Status Gem Version

Humidifier is a ruby tool for managing AWS CloudFormation stacks. You can use it to build and manage stacks programmatically or you can use it as a command line tool to manage stacks through configuration files.

Installation

Add this line to your application's Gemfile:

gem "humidifier"

And then execute:

$ bundle

Or install it yourself as:

$ gem install humidifier

Getting started

Stacks are represented by the Humidifier::Stack class. You can set any of the top-level JSON attributes (such as name and description) through the initializer.

Resources are represented by an exact mapping from AWS resource names to Humidifier resources names (e.g. AWS::EC2::Instance becomes Humidifier::EC2::Instance). Resources have accessors for each JSON attribute. Each attribute can also be set through the initialize, update, and update_attribute methods.

Example usage

The below example will create a stack with two resources, a loader balancer and an auto scaling group. It then deploys the new stack and pauses execution until the stack is finished being created.

stack = Humidifier::Stack.new(name: 'Example-Stack')

stack.add(
  'LoaderBalancer',
  Humidifier::ElasticLoadBalancing::LoadBalancer.new(
    scheme: 'internal',
    listeners: [
      {
        load_balancer_port: 80,
        protocol: 'http',
        instance_port: 80,
        instance_protocol: 'http'
      }
    ]
  )
)

stack.add(
  'AutoScalingGroup',
  Humidifier::AutoScaling::AutoScalingGroup.new(
    min_size: '1',
    max_size: '20',
    availability_zones: ['us-east-1a'],
    load_balancer_names: [Humidifier.ref('LoadBalancer')]
  )
)

stack.deploy_and_wait

Interfacing with AWS

Once stacks have the appropriate resources, you can query AWS to handle all stack CRUD operations. The operations themselves are intuitively named (i.e. #create, #update, #delete). There are also convenience methods for validating a stack body (#valid?), checking the existence of a stack (#exists?), and creating or updating based on existence (#deploy).

There are additionally four functions on Humidifier::Stack that support waiting for execution in AWS to finish. They all have non-blocking corollaries, and are named after them. They are: #create_and_wait, #update_and_wait, #delete_and_wait, and #deploy_and_wait.

CloudFormation functions

You can use CFN intrinsic functions and references using Humidifier.fn.[name] and Humidifier.ref. They will build appropriate structures that know how to be dumped to CFN syntax.

Change Sets

Instead of immediately pushing your changes to CloudFormation, Humidifier also supports change sets. Change sets are a powerful feature that allow you to see the changes that will be made before you make them. To read more about change sets see the announcement article. To use them in Humidifier, Humidifier::Stack has the #create_change_set and #deploy_change_set methods. The #create_change_set method will create a change set on the stack. The #deploy_change_set method will create a change set if the stack currently exists, and otherwise will create the stack.

Introspection

To see the template body, you can check the #to_cf method on stacks, resources, fns, and refs. All of them will output a hash of what will be uploaded (except the stack, which will output a string representation).

Humidifier itself contains a registry of all possible resources that it supports. You can access it with Humidifier::registry which is a hash of AWS resource name pointing to the class.

Resources have an ::aws_name method to see how AWS references them. They also contain a ::props method that contains a hash of the name that Humidifier uses to reference the prop pointing to the appropriate prop object.

Large templates

When templates are especially large (larger than 51,200 bytes), they cannot be uploaded directly through the AWS SDK. You can configure Humidifier to seamlessly upload the templates to S3 and reference them using an S3 URL instead by:

Humidifier.configure do |config|
  config.s3_bucket = 'my.s3.bucket'
  config.s3_prefix = 'my-prefix/' # optional
end

Forcing uploading

You can force a stack to upload its template to S3 regardless of the size of the template. This is a useful option if you're going to be deploying multiple copies of a template or if you want a backup. You can set this option on a per-stack basis:

stack.deploy(force_upload: true)

or globally, by setting the configuration option:

Humidifier.configure do |config|
  config.force_upload = true
end

CLI

Humidifier can also be used as a CLI for managing resources through configuration files. For a step-by-step guide, read on, but if you'd like to see a working example, check out the example directory.

To get started, build a ruby script (for example humidifier) that executes the Humidifier::CLI class, like so:

#!/usr/bin/env ruby
require 'humidifier'

Humidifier.configure do |config|
  # optional, defaults to the current working directory, so that all of the
  # directories from the location that you run the CLI are assumed to contain
  # resource specifications
  config.stack_path = 'stacks'

  # optional, a default prefix to use before deploying to AWS
  config.stack_prefix = 'humidifier-'

  # specifies that `users.yml` files contain specifications for `AWS::IAM::User`
  # resources
  config.map :users, to: 'IAM::User'
end

Humidifier::CLI.start(ARGV)

Resource files

Inside of the stacks directory configured above, create a subdirectory for each CloudFormation stack that you want to deploy. With the above configuration, we can create YAML files in the form of users.yml for each stack, which will specify IAM users to create. The file format looks like the below:

EngUser:
  path: /humidifier/
  user_name: EngUser
  groups:
  - Engineering
  - Testing
  - Deployment

AdminUser:
  path: /humidifier/
  user_name: AdminUser
  groups:
  - Management
  - Administration

The top-level keys are the logical resource names that will be displayed in the CloudFormation screen. They point to a map of key/value pairs that will be passed on to humidifier. Any humidifier (and therefore any CloudFormation) attribute may be specified. For more information on CloudFormation templates and which attributes may be specified, see both the humidifier docs and the CloudFormation docs.

Mappers

Oftentimes, specifying these attributes can become repetitive, e.g., each user should automatically receive the same "path" attribute. Other times, you may want custom logic to execute depending on which AWS environment you're running in. Finally, you may want to reference resources in the same or other stacks.

Humidifier's solution for this is to allow customized "mapper" classes to take the user-provided attributes and transform them into the attributes that CloudFormation expects. Consider the following example for mapping a user:

class UserMapper < Humidifier::Config::Mapper
  GROUPS = {
    'eng' => %w[Engineering Testing Deployment],
    'admin' => %w[Management Administration]
  }

  defaults do |logical_name|
    { path: '/humidifier/', user_name: logical_name }
  end

  attribute :group do |group|
    groups = GROUPS[group]
    groups.any? ? { groups: GROUPS[group] } : {}
  end
end

Humidifier.configure do |config|
  config.map :users, to: 'IAM::User', using: UserMapper
end

This means that by default, all entries in the users.yml files will get a /humidifier/ path, the user_name attribute will be set based on the logical name that was provided for the resource, and you can additionally specify a group attribute, even though it is not native to CloudFormation. With this group attribute, it will actually map to the groups attribute that CloudFormation expects.

With this new mapper in place, we can simplify our YAML file to:

EngUser:
  group: eng

AdminUser:
  group: admin

Using the CLI

Now that you've configured your CLI, your resources, and your mappers, you can use the CLI to display, validate, and deploy your infrastructure to CloudFormation. Run your script without any arguments to get the help message and explanations for each command.

Each command has an --aws-profile (or -p) option for specifying which profile to authenticate against when querying AWS. You should ensure that this profile has the correct permissions for creating whatever resources are going to part of your stack. You can also rely on the AWS_* environment variables, or the EC2 instance profile if you're deploying from an instance. For more information, see the AWS docs under the "Configuration" section.

Below are the list of commands and some of their options.

change [?stack]

Creates a change set for either the specified stack or all stacks in the repo. The change set represents the changes between what is currently deployed versus the resources represented by the configuration.

deploy [?stack] [*parameters]

Creates or updates (depending on if the stack already exists) one or all stacks in the repo.

The deploy command also allows a --prefix command line argument that will override the default prefix (if one is configured) for the stack that is being deployed. This is especially useful when you're deploying multiple copies of the same stack (for instance, multiple autoscaling groups) that have different purposes or semantically mean newer versions of resources.

display [stack] [?pattern]

Displays the specified stack in JSON format on the command line. If you optionally pass a pattern argument, it will filter the resources down to just ones whose names match the given pattern.

stacks

Displays the names of all of the stacks that humidifier is managing.

upgrade

Downloads the latest CloudFormation resource specification. Periodically AWS will update the file that humidifier is based on, in which case the attributes of the resources that were changed could change. This gem usually stays relatively in sync, but if you need to use the latest specs and this gem has not yet released a new version containing them, then you can run this command to download the latest specs onto your system.

upload [?stack]

Upload one or all stacks in the repo to S3 for reference later. Note that this must be combined with the humidifier s3_bucket configuration option.

validate [?stack]

Validate that one or all stacks in the repo are properly configured and using values that CloudFormation understands.

version

Output the version of Humidifier as well as the version of the CloudFormation resource specification that you are using.

Parameters

CloudFormation template parameters can be specified by having a special parameters.yml file in your stack directory. This file should contain a YAML-encoded object whose keys are the names of the parameters and whose values are the parameter configuration (using the same underscore paradigm as humidifier resources for specifying configuration).

You can pass values to the CLI deploy command after the stack name on the command line as in:

humidifier deploy foobar Param1=Foo Param2=Bar

Those parameters will get passed in as values when the stack is deployed.

Shortcuts

A couple of convenient shortcuts are built into humidifier so that writing templates and mappers both can be more concise.

Automatic id properties

There are a lot of properties in the AWS CloudFormation resource specification that are simply pointers to other entities within the AWS ecosystem. For example, an AWS::EC2::VPCGatewayAttachment entity has a VpcId property that represents the ID of the associated AWS::EC2::VPC.

Because this pattern is so common, humidifier detects all properties ending in Id and allows you to specify them without the suffix. If you choose to use this format, humidifier will automatically turn that value into a CloudFormation resource reference.

Anonymous mappers

A lot of the time, mappers that you create will not be overly complicated, especially if you're using automatic id properties. So, the config.map method optionally takes a block, and allows you to specify the mapper inline. This is recommended for mappers that aren't too complicated as to warrant their own class (for instance, for testing purposes). An example of this using the UserMapper from above is below:

Humidifier.configure do |config|
  config.map :users, to: 'IAM::User' do
    GROUPS = {
      'eng' => %w[Engineering Testing Deployment],
      'admin' => %w[Management Administration]
    }

    defaults do |logical_name|
      { path: '/humidifier/', user_name: logical_name }
    end

    attribute :group do |group|
      groups = GROUPS[group]
      groups.any? ? { groups: GROUPS[group] } : {}
    end
  end
end

Cross-stack references

AWS allows cross-stack references through the intrinsic Fn::ImportValue function. You can take advantage of this with humidifier by using the export: true option on resources in your stacks. For instance, if in one stack you have a subnet that you need to reference in another, you could (stacks/vpc/subnets.yml):

ProductionPrivateSubnet2a:
  vpc: ProductionVPC
  cidr_block: 10.0.0.0/19
  availability_zone: us-west-2a
  export: true

ProductionPrivateSubnet2b:
  vpc: ProductionVPC
  cidr_block: 10.0.64.0/19
  availability_zone: us-west-2b
  export: true

ProductionPrivateSubnet2c:
  vpc: ProductionVPC
  cidr_block: 10.0.128.0/19
  availability_zone: us-west-2c
  export: true

And then in another stack, you could reference those values (stacks/rds/db_subnets_groups.yml):

ProductionDBSubnetGroup:
  db_subnet_group_description: Production DB private subnet group
  subnets:
  - ProductionPrivateSubnet2a
  - ProductionPrivateSubnet2b
  - ProductionPrivateSubnet2c

Within the configuration, you would specify to use the Fn::ImportValue function like so:

Humidifier.configure do |config|
  config.stack_path = 'stacks'

  config.map :subnets, to: 'EC2::Subnet'

  config.map :db_subnet_groups, to: 'RDS::DBSubnetGroup' do
    attribute :subnets do |subnet_names|
      subnet_ids =
        subnet_names.map do |subnet_name|
          Humidifier.fn.import_value(subnet_name)
        end

      { subnet_ids: subnet_ids }
    end
  end
end

If you specify export: true it will by default export a reference to the resource listed in the stack. You can also choose to export a different attribute by specifying the attribute as the value to export. For example, if we were creating instance profiles and wanted to export the Arn so that it could be referenced by an instance later, we could:

APIRoleInstanceProfile:
  depends_on: APIRole
  roles:
  - APIRole
  export: Arn

Development

To get started, ensure you have ruby installed, version 2.4 or later. From there, install the bundler gem: gem install bundler and then bundle install in the root of the repository.

Testing

The default rake task runs the tests. Styling is governed by rubocop. The docs are generated with yard. To run all three of these, run:

$ bundle exec rake
$ bundle exec rubocop
$ bundle exec rake yard

Specs

The specs pulled from the CFN docs is saved to CloudFormationResourceSpecification.json. You can update it by running bundle exec rake specs. This script will pull down the latest resource specification to be used with Humidifier.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/kddnewton/humidifier.

License

The gem is available as open source under the terms of the MIT License.

More Repositories

1

tree

Implementations of the unix tree command
C
272
star
2

preval

Automatically optimizes your Ruby code
Ruby
98
star
3

snip_snip

Snip snip cuts the deadweight
Ruby
90
star
4

rails-pattern_matching

Pattern matching for Rails applications
Ruby
82
star
5

active_record-union_relation

Create ActiveRecord relations from UNIONs
Ruby
47
star
6

fast_underscore

Fast String#underscore implementation
C
33
star
7

regular_expression

A regular expression engine written in Ruby.
Ruby
29
star
8

gemfilelint

Lint your Gemfile
Ruby
29
star
9

exreg

A Ruby regular expression engine
Ruby
27
star
10

vernacular

Allows extending ruby's syntax and compilation process
Ruby
27
star
11

parser-prism

A prism backend for the whitequark/parser gem
Ruby
24
star
12

travis.ex

Simple Elixir wrapper for the Travis API
Elixir
21
star
13

minitest-keyword

Use Minitest assertions with keyword arguments
Ruby
20
star
14

compiling-ruby

Code for the "Compiling ruby" talk
Ruby
19
star
15

practical-debugging

Code for the "Practical debugging" talk
Ruby
18
star
16

sorbet-eraser

Erase all traces of sorbet-runtime code
Ruby
16
star
17

ripper-docs

Documentation for the ripper Ruby standard library
15
star
18

thor-hollaback

Adds callbacks to thor commands
Ruby
14
star
19

prettier-plugin-ini

A prettier plugin for INI files
JavaScript
11
star
20

mathlang

A small language for doing math
TypeScript
8
star
21

parsing-ruby

TypeScript
8
star
22

hollaback

Add callbacks to your methods
Ruby
8
star
23

attribute_extras

Extra macros for auto attribute manipulation.
Ruby
7
star
24

typescript-parse-math

Parse math expressions with TypeScript types
7
star
25

ruby-parser

A Ruby parser
C
7
star
26

iseq-rails-tools

Use AOT compilation within Rails
Ruby
7
star
27

bundler-console

A bundler plugin that starts a console session with your gem dependencies.
Ruby
6
star
28

yarv

A Ruby object layer on top of the YARV virtual machine
Ruby
6
star
29

wordle

A wordle solver
Ruby
5
star
30

ragel-bitmap

Use packed strings for ragel-generated code instead of integer arrays
Ruby
5
star
31

checks

Ruby
5
star
32

musicxml

Ruby bindings for musicxml
Ruby
5
star
33

ruby-reprs

Ruby representations
JavaScript
4
star
34

fast_camelize

Fast String#camelize implementation
Ruby
4
star
35

prettier-plugin-brainfuck

Prettier for Brainfuck
JavaScript
4
star
36

fast_parameterize

Fast String#parameterize implementation
Ruby
4
star
37

taxes

US federal income tax calculator
TypeScript
3
star
38

vernacular-ast

Extends Vernacular to support rewriting the AST
Ruby
3
star
39

sql-to-ruby

Parse SQL queries and turn them into equivalent Ruby
Ruby
2
star
40

lox

A Ruby implementation of the lox language
Ruby
2
star
41

bonsai

Code for the "Grow a bonsai, not a shrub" talk
Ruby
2
star
42

ripperjs

A wrapper for the ripper Ruby library
C++
2
star
43

yarp-identifiers

C
1
star
44

unicode-data

A Ruby wrapping for the unicode character data set
Ruby
1
star
45

pokerpg-builder

Webapp for building pokemon in PokeRPG
TypeScript
1
star
46

wm-memories

An app for storing memories from the College of William and Mary.
Ruby
1
star
47

kddnewton.github.io

Personal website
CSS
1
star
48

rubytoolbox-chrome

A Chrome extension for displaying rubytoolbox gem data on rubygems.org
TypeScript
1
star