Diplomat
A HTTP Ruby API for Consul
FAQ
What's Diplomat for?
Diplomat allows any ruby application to interact with Consul's distributed key value store, and also receive information about services currently available in the Consul cluster.
Does it work in rails?
Yup! In fact, we're using it in all of our rails production apps instead of any previous case where it'd be right to use environment variables according to 12Factor configuration principals. This gives us the ability to scale up without making any changes to the actual project codebase, and to move applications around the cluster with ease.
Here's what a production database.yml file might look like:
<% if Rails.env.production? %>
production:
adapter: postgresql
encoding: unicode
host: <%= Diplomat::Service.get('postgres').Address %>
database: <%= Diplomat::Kv.get('project/db/name') %>
pool: 5
username: <%= Diplomat::Kv.get('project/db/user') %>
password: <%= Diplomat::Kv.get('project/db/pass') %>
port: <%= Diplomat::Service.get('postgres').ServicePort %>
<% end %>Why would I use Consul over ZooKeeper, Doozerd, etcd, Nagios, Sensu, SmartStack, SkyDNS, Chef, Puppet, Ansible, etc?
Read up what makes Consul different here
How do I install Consul?
See here. I managed to roll it out on my production machines with the help of Ansible in one working day.
Which versions of Ruby does Diplomat support? Where did my ruby 1.9 compatibility go?
Check out GitHub Actions to see which versions of ruby we currently test when we're making builds.
We've dropped ruby 1.9 support. You can still depend on Diplomat by directly using the ruby-1.9-compatible branch on github, although be advised it's not actively maintained anymore.
ERB templating
It is possible to inject diplomat data into .erb files (such as in chef), but you could also have a look at
consul-templaterb that is highly optimized for ERB templating
with very hi parallelism and good optimized performance for large clusters.
Usage
The most up to date place to read about the API is here.
Here's a few examples of how diplomat works:
Key Values
Setting
Setting the value of a key is easy as pie:
foo = Diplomat::Kv.put('foo', 'bar')
# => "bar"Getting
Getting the value of a key is just as simple:
foo = Diplomat::Kv.get('foo')
# => "bar"Or retrieve a value from another datacenter:
foo = Diplomat::Kv.get('foo', :dc => 'dc-west')
# => "baz"You can also retrieve values recursively:
Diplomat::Kv.put('foo/a', 'lorem')
Diplomat::Kv.put('foo/b', 'ipsum')
Diplomat::Kv.put('foo/c', 'dolor')
Diplomat::Kv.get('foo/', recurse: true)
# => [{:key=>"foo/a", :value=>"lorem"}, {:key=>"foo/b", :value=>"ipsum"}, {:key=>"foo/c", :value=>"dolor"}]You can also use get_all to retrieve values recursively with a consistent return type:
Diplomat::Kv.put('foo/a', 'lorem')
Diplomat::Kv.put('foo/b', 'ipsum')
Diplomat::Kv.put('foo/c', 'dolor')
Diplomat::Kv.get('foo/', recurse: true)
# => [{:key=>"foo/a", :value=>"lorem"}, {:key=>"foo/b", :value=>"ipsum"}, {:key=>"foo/c", :value=>"dolor"}]
Diplomat::Kv.get_all('foo/')
# => [{:key=>"foo/a", :value=>"lorem"}, {:key=>"foo/b", :value=>"ipsum"}, {:key=>"foo/c", :value=>"dolor"}]
Diplomat::Kv.put('bar/a', 'lorem')
Diplomat::Kv.get('bar/', recurse: true)
# => "lorem"
Diplomat::Kv.get_all('bar/')
# => [{:key=>"bar/a", :value=>"lorem"}]Or list all available keys:
Diplomat::Kv.get('/', :keys => true) # => ['foo/a', 'foo/b']You can convert the consul data to a ruby hash
Diplomat::Kv.put('foo/a', 'lorem')
Diplomat::Kv.put('foo/b', 'ipsum')
Diplomat::Kv.put('foo/c', 'dolor')
Diplomat::Kv.get('foo/', recurse: true, convert_to_hash: true)
# => {"foo"=>{"a"=>"lorem", "b"=>"ipsum", "c"=>"dolor"}}Nodes
Getting
Look up a node:
foo_service = Diplomat::Node.get('foo')
# => {"Node"=>{"Node"=>"foobar", "Address"=>"10.1.10.12"}, "Services"=>{"consul"=>{"ID"=>"consul", "Service"=>"consul", "Tags"=>nil, "Port"=>8300}, "redis"=>{"ID"=>"redis", "Service"=>"redis", "Tags"=>["v1"], "Port"=>8000}}}Get all nodes:
nodes = Diplomat::Node.get_all
# => [#<OpenStruct Address="10.1.10.12", Node="foo">, #<OpenStruct Address="10.1.10.13", Node="bar">]Get all nodes for a particular datacenter
nodes = Diplomat::Node.get_all({ :dc => 'My_Datacenter' })
# => [#<OpenStruct Address="10.1.10.12", Node="foo">, #<OpenStruct Address="10.1.10.13", Node="bar">]Register a node:
Diplomat::Node.register({ :Node => "app1", :Address => "10.0.0.2" })
# => trueDe-register a node:
Diplomat::Node.deregister({ :Node => "app1", :Address => "10.0.0.2" })
# => trueServices
Getting
Looking up a service is easy as pie:
foo_service = Diplomat::Service.get('foo')
# => #<OpenStruct Node="hotel", Address="1.2.3.4", ServiceID="hotel_foo", ServiceName="foo", ServiceTags=["foo"], ServicePort=5432>Or if you have multiple nodes per service:
foo_service = Diplomat::Service.get('foo', :all)
# => [#<OpenStruct Node="hotel", Address="1.2.3.4", ServiceID="hotel_foo", ServiceName="foo", ServiceTags=["foo"], ServicePort=5432>,#<OpenStruct Node="indigo", Address="1.2.3.5", ServiceID="indigo_foo", ServiceName="foo", ServiceTags=["foo"], ServicePort=5432>]Or if you want to find services for a particular datacenter
foo_service = Diplomat::Service.get('foo', :all, { :dc => 'My_Datacenter'})
# => [#<OpenStruct Node="hotel", Address="1.2.3.4", ServiceID="hotel_foo", ServiceName="foo", ServiceTags=["foo"], ServicePort=5432>,#<OpenStruct Node="indigo", Address="1.2.3.5", ServiceID="indigo_foo", ServiceName="foo", ServiceTags=["foo"], ServicePort=5432>]Or if you want to filter services
foo_service = Diplomat::Service.get('foo', :all, { :filter => 'ServiceID == "indigo_foo"'})
# => [#<OpenStruct Node="indigo", Address="1.2.3.5", ServiceID="indigo_foo", ServiceName="foo", ServiceTags=["foo"], ServicePort=5432>]If you wish to list all the services on consul:
services = Diplomat::Service.get_all
# => #<OpenStruct consul=[], foo=[], bar=[]>If you wish to list all the services for a specific datacenter:
services = Diplomat::Service.get_all({ :dc => 'My_Datacenter' })
# => #<OpenStruct consul=[], foo=[], bar=[]>Datacenters
Getting a list of datacenters is quite simple and gives you the option to extract all services out of all accessible datacenters if you need to.
datacenters = Diplomat::Datacenter.get()
# => ["DC1", "DC2"]Sessions
Creating a session:
sessionid = Diplomat::Session.create({:Node => "server1", :Name => "my-lock"})
# => "fc5ca01a-c317-39ea-05e8-221da00d3a12"Or destroying a session:
Diplomat::Session.destroy("fc5ca01a-c317-39ea-05e8-221da00d3a12")Renew a session:
Diplomat::Session.renew(sessionid)List sessions:
Diplomat::Session.list.each {|session| puts "#{session["ID"]} #{session["Name"]}"}Locks
Acquire a lock:
sessionid = Diplomat::Session.create({:Node => "server1", :Name => "my-lock"})
lock_acquired = Diplomat::Lock.acquire("/key/to/lock", sessionid)
# => trueOr wait for a lock to be acquired:
sessionid = Diplomat::Session.create({:hostname => "server1", :ipaddress => "4.4.4.4"})
lock_acquired = Diplomat::Lock.wait_to_acquire("/key/to/lock", sessionid)Release a lock:
Diplomat::Lock.release("/key/to/lock", sessionid )Events
Fire an event:
Diplomat::Event.fire('do_something', 'payload')List all events with a certain name received by the local agent:
Diplomat::Event.get_all('do_something')Get the latest event with a certain name received by the local agent:
Diplomat::Event.get('do_something')Iterate through the events with a certain name received by the local agent:
events = Enumerator.new do |y|
ret = {token: :first}
while ret = begin Diplomat::Event.get('do_something', ret[:token], :reject) rescue nil end
y.yield(ret[:value])
end
end
events.each{ |e| puts e }Status
Returns information about the status of the Consul cluster.
Get the raft leader for the datacenter in which the local consul agent is running
Diplomat::Status.leader()Get an array of Raft peers for the datacenter in which the agent is running
Diplomat::Status.peers()Autopilot
Returns information about the autopilot configuration of the Consul cluster
Get the current autopilot configuration
Diplomat::Autopilot.get_configuration()Get the health status from autopilot
Diplomat::Autopilot.get_health()Health
Retrieve health of a node
Diplomat::Health.node('fooNode', :dc => 'abc')Retrieve health of a given check
Diplomat::Health.checks('fooCheck', :dc => 'abc')Retrieve health of a given service
Diplomat::Health.service('fooService', :dc => 'abc')Retrieve a list of anything that correspond to the state ("any", "passing", "warning", or "critical") You can use filters too !
Diplomat::Health.state("critical", {:dc => 'abc', :filter => 'Node==foo'})You also have some convenience method (any, passing, warning, critical)
That can be filtered
Diplomat::Health.critical({:dc => 'abc', :filter => 'ServiceName==foo'})Maintenance mode
Enable maintenance mode on a host, with optional reason and DC (requires access to local agent)
Diplomat::Maintenance.enable(true, 'doing stuff', :dc => 'abc')Determine if a host has maintenance mode enabled
Diplomat::Maintenance.enabled('foobar')
# => { :enabled => true, :reason => 'doing stuff' }Custom configuration
You can create a custom configuration using the following syntax:
Diplomat.configure do |config|
# Set up a custom Consul URL
config.url = "http://localhost:8888"
# Set up a custom Faraday Middleware
config.middleware = MyCustomMiddleware
# Set extra Faraday configuration options and custom access token (ACL)
config.options = {ssl: {version: :TLSv1_2}, headers: {"X-Consul-Token" => "xxxxxxxx-yyyy-zzzz-1111-222222222222"}}
endThis is traditionally kept inside the config/initializers directory if you're using rails. The middleware allows you to customise what happens when faraday sends and receives data. This can be useful if you want to instrument your use of diplomat, for example. You can read more about Faraday's custom middleware here.
Alternatively, configuration settings can be overriden at each method call allowing for instance to address different consul agents, with some other token.
Diplomat::Service.get('foo', { http_addr: 'http://consu01:8500' })
Diplomat::Service.get('foo', { http_addr: 'http://consu02:8500' })
Diplomat::Kv.put('key/path', 'value', { http_addr: 'http://localhost:8500', dc: 'dc1', token: '111-222-333-444-555' })Most common options are:
- dc: target datacenter
- token: identity used to perform the corresponding action
- http_addr: to target a remote consul node
- stale: use consistency mode that allows any server to service the read regardless of whether it is the leader
Todo
- Updating Docs with latest changes
- Using custom objects for response objects (instead of openStruct)
- PUTing and DELETEing services
- Custom SSL Cert Middleware for faraday
- Allowing the custom configuration of the consul url to connect to
- Deleting Keys
- Listing available services
- Health
- Members
- Status
- Datacenter support for services
- Ruby 1.8 support
- Events
