• Stars
    star
    122
  • Rank 292,127 (Top 6 %)
  • Language
    Ruby
  • License
    Other
  • Created almost 15 years ago
  • Updated about 10 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Wise, discreet configuration for ruby scripts: integrate config files, environment variables and command line with no fuss

Configliere

This repository has moved: infodhimps-platform/configliere
All further code work and gems will take place in the above repo, not here.

Configliere provides discreet configuration for ruby scripts.

So, Consigliere of mine, I think you should tell your Don what everyone knows. ā€” Don Corleone

Youā€™ve got a script. Itā€™s got some settings. Some settings are for this module, some are for that module. Most of them donā€™t change. Except on your laptop, where the paths are different. Or when youā€™re in production mode. Or when youā€™re testing from the command line.

Configliere manages settings from many sources: static constants, simple config files, environment variables, commandline options, straight ruby. You donā€™t have to predefine anything, but you can ask configliere to type-convert, require, document or password-obscure any of its fields. Basically: Settings go in, the right thing happens.

Build Status

Example

Hereā€™s a simple example, using params from a config file and the command line. In the script:

    #/usr/bin/env ruby
    require 'configliere'
    Settings.use :commandline

    # Supply defaults inline.
    Settings({
      :dest_time => '11-05-1955',
      :delorean => {
        :power_source => 'plutonium',
        :roads_needed => true,
        },
      :username => 'marty',
    })
    
    # Pre-defining params isn't required, but it's easy and expressive to do so:
    Settings.define :dest_time, :type => DateTime, :description => "Target date"
    # This defines a 'deep key': it controls Settings[:delorean][:roads_needed]
    Settings.define 'delorean.roads_needed', :type => :boolean

    # The settings in this file will be merged with the above
    Settings.read './examples/simple_script.yaml'

    # resolve! the settings: load the commandline, do type conversion, etc.
    Settings.resolve!
    p Settings

Weā€™ll override some of the defaults with a config file, in this case ./examples/simple_script.yaml

    # Settings for return
    :dest_time:       11-05-1985
    :delorean:    
      :power_source:  1.21 jiggawatts

Output, when run with commandline parameters as shown:

    ./time_machine.rb --username=doc_brown --delorean.roads_needed="" --delorean.power_source="Mr. Fusion"

    {:delorean => {:power_source=>"Mr. Fusion", :roads_needed=>nil}, :username=>"doc_brown", :dest_time=>#<DateTime: 1985-05-11T00:00:00+00:00>}

For an extensive usage in production, see the wukong gem.

Notice

Configliere 4.x now has 100% spec coverage, more powerful commandline handling, zero required dependencies. However, it also strips out several obscure features and much magical code, which breaks said obscure features and magic-dependent code. See the CHANGELOG. for details as you upgrade.

Design goals:

  • Omerta (Code of Silence). Most commandline parsers force you to pre-define all your parameters in a centralized and wordy syntax. In configliere, you donā€™t have to pre-define anything ā€” commandline parameters map directly to values in the Configliere hash. Hereā€™s all you need to have full-fledged commandline params:
  $ cat ./shorty.rb
  require 'configliere'
  Settings.use(:commandline).resolve!
  p [Settings, Settings.rest]
  
  $ ./shorty.rb --foo=bar go
  [{:foo=>"bar"}, ["go"]]
  • Be willing to sit down with the Five Families. Takes settings from (at your option):
    • Pre-defined defaults from constants
    • Simple config files
    • Environment variables
    • Commandline options and git-style command runners
    • Ruby block (called when all other options are in place)
  • Donā€™t go outside the family. Has no dependencies and requires almost no code in your script. Configliere makes no changes to standard ruby classes.
  • Offer discreet counsel. Configliere offers many features, but only loads the code you request explicitly by calling use.
  • Donā€™t mess with my crew. Settings for a model over here can be done independently of settings for a model over there, and donā€™t require asking the boss to set something up. You centralize configuration values while distributing configuration definition:
    # In lib/handler/mysql.rb
    Settings.define :mysql_host, :type => String, :description => "MySQL db hostname", :default => 'localhost'

    # In app/routes/homepage.rb
    Settings.define :background_color, :description => "Homepage background color"

    # In config/app.yaml
    ---
    :background_color:  '#eee'
    :mysql_host:        'brains.infochimps.com'

You can decentralize even more by giving modules their own config files or separate Configliere::Param objects.

  • Can hide your assets. Rather than storing passwords and API keys in plain sight, configliere has a protection racket that can obscure values when stored to disk.

fuhgeddaboudit.

Settings structure

A Configliere settings object is just a (mostly-)normal hash.

You can define static defaults in your module

    Settings({
      :dest_time => '11-05-1955',
      :fluxcapacitor => {
        :speed => 88,
        },
      :delorean => {
        :power_source => 'plutonium',
        :roads_needed => true,
        },
      :username => 'marty',
      :password => '',
    })

All simple keys should be symbols. Retrieve the settings as:

    # hash keys
    Settings[:dest_time]                 #=> '11-05-1955'
    # deep keys
    Settings[:delorean][:power_source]   #=> 'plutonium'
    Settings[:delorean][:missing]        #=> nil
    Settings[:delorean][:missing][:fail] #=> raises an error
    # dotted keys resolve to deep keys
    Settings['delorean.power_source']    #=> 'plutonium'
    Settings['delorean.missing']         #=> nil
    Settings['delorean.missing.fail']    #=> nil
    # method-like (no deep keys tho, and you have to #define the param; see below)
    Settings.dest_time                   #=> '11-05-1955'

Configuration files

Call Settings.read(filename) to read a YAML config file.

    # Settings for version II.
    :dest_time:        11-05-2015
    :delorean:
      :power_source:    Mr. Fusion
      :roads_needed:    ~

If a bare filename (no ā€˜/ā€™) is given, configliere looks for the file in Configliere::DEFAULT_CONFIG_DIR (normally ~/.configliere). Otherwise it loads the given file.

    Settings.read('/etc/time_machine.yaml')  # looks in /etc/time_machine.yaml
    Settings.read('time_machine.yaml')       # looks in ~/.configliere/time_machine.yaml

As you can see, youā€™re free to use as many config files as you like. Loading a config file sets values immediately, so later-loaded files win out over earlier-loaded ones.

You can save configuration too:

    Settings.save!('/etc/time_machine.yaml') # overwrites /etc/time_machine.yaml
    Settings.save!('time_machine.yaml')      # overwrites ~/.configliere/time_machine.yaml

Command-line parameters

    # Head back
    time_machine --delorean.power_source='1.21 jiggawatt lightning strike' --dest_time=11-05-1985
    # (in the time_machine script:)
    Settings.use(:commandline)
    Settings.resolve!

Interpretation of command-line parameters:

  • name-val params: --param=val sets Configliere[:param] to val. You must use the ā€˜=ā€™ in there: ./my_cmd --filename=bar good, ./my_cmd --filename bar bad.
  • boolean params: --param sets Configliere[:param] to be true. --param="" sets Configliere[:param] to be nil.
  • single-char flags: Define a flag for a variable: Settings.define :filename, :flag => "f" allows you to say ./my_cmd -f=bar.
  • scoped params: A dot within a parameter name scopes that parameter: --group.sub_group.param=val sets Configliere[:group][:subgroup][:param] to val (and similarly for boolean parameters).
    • Only [\w\.]+ are accepted in parameter names. ā€˜-ā€™ is currently accepted but causes a warning.
  • Settings.rest: anything else is stored, in order, in Settings.rest.
  • stop marker: a -- alone stops parameter processing and tosses all remaining params (not including the --) into Settings.rest.

Here are some things you donā€™t get:

  • Configliere doesnā€™t complain about un-@define@ā€™d commandline argvs. However, it does store each undefineā€™d argv (when resolve! is called) into unknown_argvs, so you can decide what to do about it.
  • Apart from converting '' (an explicit blank string) to nil, no type coercion is performed on parameters unless requested explicitly (see below).
  • No validation is performed on parameters, but you can insert a middleware with a validate!() method, or use a :finally block.
  • No ordering or multiplicity is preserved: you canā€™t say --file=this --file=that. Instead, define the param as an array Settings.define :file, :type => Array and give a simple comma-separated list.

Commandline parameters are demonstrated in examples/simple_script.rb and examples/env_var_script.rb

Defined Parameters

You donā€™t have to pre-define parameters, but you can:

    Settings.define :dest_time, :type => DateTime, :description => 'Arrival time'
    Settings.define 'delorean.power_source', :env_var => 'POWER_SOURCE', :description => 'Delorean subsytem supplying power to the Flux Capacitor.'
    Settings.define :password, :required => true, :encrypted => true
  • :description: documents a param.
  • :type: converts params to a desired form.
  • :required: marks params required.
  • :encrypted: marks params to be obscured when saved to disk. See [#Encrypted Parameters] below for caveats.
  • :env_var: take param from given environment variable if set.

Defined parameters are demonstrated in most of the example scripts

Description

If you define a paramā€™s description, besides nicely documenting it within your code the description will be stuffed into the output when the ā€”help commandline option is invoked.

  $ ./examples/simple_script
  usage: simple_script.rb [...--param=val...]

  Params:
    --delorean.roads_needed   delorean.roads_needed
    --dest_time=DateTime      Date to travel to [Default: 11-05-1955]

Type Conversion

Parameters defined with a :type and will be type-converted when you call Settings.resolve!.

    Settings.define :dest_time,     :type => DateTime
    Settings.define :fugeddaboudit, :type => Array
    Settings :fugeddaboudit => 'badabing,badaboom,hey', :dest_time => '11-05-1955'
    Settings.resolve!
    Settings[:fugeddaboudit]   #=> ['badabing', 'badaboom', 'hey']
    Settings[:dest_time]       #=> #<DateTime: 4870833/2,0,2299161>

Configliere can coerce parameter values to Integer, Float, :boolean, Symbol, Array, Date and DateTime.

  • :boolean converts nil to nil ; false, ā€˜falseā€™, 0, ā€˜0ā€™ and ā€™ā€™ to false; and everything else to true.
  • Array just does a simple split on ā€œ,ā€. It doesnā€™t do any escaping or quoting.
  • Date and DateTime convert unparseable inputs to nil.
  • :filename calls File.expand_path() on the param.

Required Parameters

Any required parameter found to be nil raise an error (listing all missing params) when you call Settings.resolve! (See examples/env_var_script.rb)

Environment Variables

    Settings.define :dest_time,   :env_var => 'DEST_TIME'
    Settings.define :environment, :env_var => 'RACK_ENV'

Encrypted Parameters

Define a param to be encrypted and invoke Settings.save!. It will use Settings.encrypt_pass (or the ENCRYPT_PASS environment variable) to encrypt the data when it is saved to disk. (see examples/encrypted_script.rb)

    Settings.use :encrypted
    Settings.define 'amazon.api.key', :encrypted => true
    Settings 'amazon.api.key' => 'fnord'

In this example, the hash saved to disk will contain { :amazon => { :api => { :encrypted_key => "...encrypted val..." } } }. After reading from disk, #resolve! will recover its original value: { :amazon => { :api => { :key => "fnord" } } }.

There are two kinds of cryptography in this world: cryptography that will stop your kid sister from reading your files, and cryptography that will stop major governments from reading your files. This book is about the latter. ā€” Preface to Applied Cryptography by Bruce Schneier

Configliere provides the former.

Anyone with access to the script, its config files and its normal launch environment can recover the plaintext password; but it at least doesnā€™t appear when you cat the file while giving a presentation.

Ruby Block

    Settings.use :config_block
    Settings.finally do |c|
      c.dest_time = (Time.now + 60) if c.username == 'einstein'
      # you can use hash syntax too
      c[:dest_time] = (Time.now + 60) if c[:username] == 'einstein'
    end
    #
    # ... rest of setup ...
    #
    Settings.resolve!    # the finally blocks will be called in order

Configliere ā€˜finallyā€™ blocks are invoked when you call resolve!. Theyā€™re guaranteed to be called at the end of the resolve chain, and before the validate chain.

Config blocks are demonstrated in examples/config_block.rb

Shortcut syntax for deep keys

You can use a ā€˜dotted keyā€™ like ā€˜delorean.power_sourceā€™ as simple notation for a deep key: Settings['delorean.power_source'] is equivalent to Settings[:delorean][:power_source]. You can use a dotted key in any simple reference:

  Settings['delorean.power_source'] = "Mr. Fusion"
  Settings[:delorean][:power_source]
  #=> "Mr. Fusion"
  Settings.delete('delorean.power_source')
  #=> "Mr. Fusion"
  Settings
  #=> { :delorean => {} }

Intermediate keys ā€œauto-vivifyā€ (automatically create any intervening hashes):

  Settings['one.two.three'] = "To tha Fo'"
  # Settings is { :one => { :two => { :three => "To tha Fo'" } }, :delorean => { :power_source => "Mr. Fusion" }

Independent Settings

All of the above examples use the global variable Settings, defined in configliere.rb. It really works fine in practice, even where several systems intersect. Youā€™re free to define your own settings universe though:

    class Wolfman
      def config
        @config ||= Configliere::Param.new.use(:commandline).defaults({
          :moon    => 'full',
          :nards   => true,
          })
      end
    end

    teen_wolf = Wolfman.new
    teen_wolf.config.defaults(:give_me => 'keg of beer')
    
    teen_wolf.config #=> {:moon=>"full", :nards=>true, :give_me=>"keg of beer" }
    Settings         #=> {}

Values in here donā€™t overlap with the Settings object or any other settings universe. However, every one that pulls in commandline params gets a full copy of the commandline params.

Project info

Note on Patches/Pull Requests

  • Fork the project.
  • Make your feature addition or bug fix.
  • Add tests for it. This is important so I donā€™t break it in a future version unintentionally.
  • Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
  • Send a pull request to github.com/mrflip
  • Drop a line to the mailing list for infochimps open-source projects, [email protected]

Copyright

Copyright Ā© 2010 mrflip. See LICENSE for details.

More Repositories

1

ironfan

Chef orchestration layer -- your system diagram come to life. Provision EC2, OpenStack or Vagrant without changes to cookbooks or configuration
Ruby
501
star
2

wukong

Ruby on Hadoop: Efficient, effective Hadoop streaming & bulk data processing. Write micro scripts for terabyte-scale data
Ruby
497
star
3

wonderdog

Bulk loading for elastic search
Java
186
star
4

big_data_for_chimps

A Seriously Fun guide to Big Data Analytics in Practice
Ruby
169
star
5

ironfan-pantry

Battle-hardened Ironfan-ready big data chef cookbooks, laden with best practices and love from your friends at Infochimps
Python
99
star
6

ironfan-homebase

Skeleton homebase for Ironfan and Chef -- use this to hold your clusters, cookbooks and stacks
Ruby
34
star
7

data_science_fun_pack

Meta-repository of big data tools -- source and essential plugins for hadoop, pig, wukong, storm, kafka etc.
Java
29
star
8

cube

Cube: A system for time series visualization.
JavaScript
23
star
9

gorillib

Gorillib: infochimps lightweight subset of ruby convenience methods
Ruby
17
star
10

ironfan-ci

Continuous Integration testing of ironfan clusters and chef cookbooks. Pass your system diagram into iron law,
Ruby
17
star
11

wukong-hadoop

Execute Wukong code within the Hadoop framework.
Ruby
13
star
12

ironfan-repoman

Rake tasks to syndicate out 50 cookbooks from ironfan-pantry into distinct isolated repos. Don't look in the trunk, repo man.
Ruby
13
star
13

chimpstation-homebase

it's like rocket fuel for cookbook development
Ruby
10
star
14

icss

Infochimps Stupid Schema library: an avro-compatible data description standard. ICSS completely describes a collection of data (and associated assets) in a way that is expressive, scalable and sufficient to drive remarkably complex downstream processes.
Ruby
9
star
15

vayacondios

Data goes in. The right thing happens.
Ruby
8
star
16

swineherd-fs

Filesystem Abstraction for S3, HDFS and normal filesystem
Ruby
5
star
17

senor_armando

Skeleton Goliath (http://goliath.io) App layout
Ruby
5
star
18

wukong-storm

Storm plugin for Wukong
Java
5
star
19

iron_cuke

Integration tests for the cloud, done right.
Ruby
4
star
20

wukong-load

Plugin that makes it easy to load, dump, and sync data between Wukong and various data stores
Ruby
4
star
21

community-pantry

Ironfan cookbooks not currently maintained by Infochimps
Ruby
3
star
22

wukong-deploy

Deploy pack framework for the Infochimps Platform
Ruby
2
star
23

dotfiles

dotfiles (.zshrc, .emacs.d, etc) for an infochimps standard workstation
Ruby
2
star
24

pigsy

UDFs and Loaders for Apache Pig -- geodata and more on hadoop
Java
2
star
25

infochimps-labs.github.com

infochimps open source reference pages
JavaScript
2
star
26

dark_siphon

A Goliath web app for duplicating production traffic to another system.
Ruby
2
star
27

bare-pantry

An empty Ironfan pantry
Ruby
1
star
28

goliath-chimp

Collection of Chimp-inspired Goliath/Rack utility classes
Ruby
1
star
29

wukong_for_r

How to use Wukong to run R scripts in Hadoop as well as locally
R
1
star
30

kibana-es-plugin

CSS
1
star