Blender
Blender is a modular remote command execution framework. Blender provides few basic primitives to automate across server workflows. Workflows can be expressed in plain ruby DSL and executed using the CLI.
Following is an example of a simple blender script that will update the package index of three ubuntu servers.
# example.rb
ssh_task 'update' do
execute 'sudo apt-get update -y'
members ['ubuntu01', 'ubuntu02', 'ubuntu03']
end
Which can execute it as:
blend -f example.rb
Output:
Run[example.rb] started
3 job(s) computed using 'Default' strategy
Job 1 [update on ubuntu01] finished
Job 2 [update on ubuntu02] finished
Job 3 [update on ubuntu03] finished
Run finished (42.228923876 s)
An workflow can have multiple tasks, individual tasks can have different members which can be run in parallel.
# example.rb
ssh_task 'update' do
execute 'sudo apt-get update -y'
members ['ubuntu01', 'ubuntu02', 'ubuntu03']
end
ssh_task 'install' do
execute 'sudo apt-get install screen -y'
members ['ubuntu01', 'ubuntu03']
end
concurrency 2
Output:
Run[blends/example.rb] started
5 job(s) computed using 'Default' strategy
Job 1 [update on ubuntu01] finished
Job 2 [update on ubuntu02] finished
Job 4 [install on ubuntu01] finished
Job 3 [update on ubuntu03] finished
Job 5 [install on ubuntu03] finished
Run finished (4.462043017 s)
Blender provides various types of task execution (like arbitrary ruby code, commands over ssh, serf handlers etc) which can ease automating large cluster maintenance, multi stage provisioning, establishing cross server feedback loops etc.
Installation
Blender is published as pd-blender
in rubygems. And you can install it as:
gem install pd-blender
Or declare it as a dependency in your Gemfile, if you are using bundler.
gem 'pd-blender'
Concepts
Blender is composed of two components:
-
Tasks and drivers - Tasks encapsulate commands (or equivalent abstraction). A blender script can have multiple tasks. Tasks are executed using drivers. Tasks can declare their target hosts.
-
Scheduling strategy - Determines the order of task execution across the hosts. Every blender scripts has one and only one scheduling strategy. Scheduling strategies uses the task list as input and produces a list of jobs, to be executed using drivers.
Tasks
Tasks and drivers compliment each other. Tasks act as front end, where we declare
what needs to be done, while drivers are used to interpret how those tasks can be done.
For example ssh_task
can be used to declare tasks, while ssh
and ssh_multi
driver
can execute ssh_task
s. Blender core ships with following tasks and drivers:
- shell_task: execute commands on current host. shell tasks can only have 'localhost' as its members. presence of any other hosts in members list will raise exception. shell_tasks are executed using shell_out driver. Example:
shell_task 'foo' do
execute 'sudo apt-get update -y'
end
- ruby_task: execute ruby blocks against current host. host names from members list is passed
to the block. ruby_tasks are executed using
Blender::Ruby
driver. Example:
ruby_task 'baz' do
execute do |host|
puts "Host name is: #{host}"
end
end
- ssh_task: execute commands against remote hosts using ssh. Blender ships with two ssh drivers,
one based on a vanilla Ruby
net-ssh
binding, another based onnet-ssh-multi
(which supports parallel execution) Example:
ssh_task 'bar' do
execute 'sudo apt-get update -y'
members ['host1', 'host2']
end
- scp_task: download or upload files using scp
members ['host1', 'host2', 'host3']
scp_upload '/foo/bar' do
from '/path/to/remote/file'
end
scp_download '/foo/bar' do
to '/local/path'
end
- blend_task: invoke a blender script as a task (nesting)
members ['host1', 'host2', 'host3']
blend_task 'test-task' do
file '/path/to/remote/file'
concurrency 10
strategy :per_host
end
As mentioned earlier tasks are executed using drivers. Tasks can declare their preferred driver or
Blender will assign a driver to them automatically. Blender will reuse the global driver if its
compatible, else it will create one. By default the global_driver
is a shell_out
driver.
Drivers can expose host concurrency, stdout/stderr streaming and various other customizations,
specific to their own implementations.
Scheduling strategies
Scheduling strategies are the most crucial part of a blender script. They decide the order of command execution across distributed nodes in blender. Each blender script is invoked using one strategy. Consider them as a transformation, where the input is tasks and ouput is jobs. Tasks and job are pretty similar in their structures (both holds command and hosts), except a jobs can hold multiple tasks within them. We'll come to this later, but first, let's see how the default strategy work.
- default strategy: the default strategy takes the list of declared tasks (and associated members in each tasks) breaks them up into per node jobs. For example:
members ['host1', 'host2', 'host3']
ruby_task 'test' do
execute do |host|
Blender::Log.info(host)
end
end
will result in 3 jobs. each with ruby_task[test]
on host1, ruby_task[test]
on host2 and
ruby_task[test]
on host3. And then these three tasks will be executed serially.
Following will create 6 jobs.
members ['host1', 'host2', 'host3']
ruby_task 'test 1' do
execute do |host|
Blender::Log.info("test 1 on #{host}")
end
end
ruby_task 'test 2' do
execute do |host|
Blender::Log.info("test 2 on #{host}")
end
end
While the next one will create 4 jobs (second task will give only one job).
members ['host1', 'host2', 'host3']
ruby_task 'test 1' do
execute do |host|
Blender::Log.info("test 1 on #{host}")
end
end
ruby_task 'test 2' do
execute do |host|
Blender::Log.info("test 2 on #{host}")
end
members ['host3']
end
The default strategy is conservative, and allows drivers that work against a single remote host to be integrated with blender. Also this allows the highest level of fine grain job control.
Apart from the default strategy, Blender ships with two more strategy, they are:
- per task strategy: this creates one job per task. Following example will
create 2 jobs, each with three hosts and one of the
ruby_task
in them.
members ['host1', 'host2', 'host3']
strategy :per_task
ruby_task 'test 1' do
execute do |host|
Blender::Log.info("test 1 on #{host}")
end
end
ruby_task 'test 2' do
execute do |host|
Blender::Log.info("test 2 on #{host}")
end
end
per task strategy allows drivers to optimize individual command execution across multiple hosts. For
example ssh_multi
driver allows parallel command execution across many hosts. And can be used
as:
strategy :per_task
global_driver(:ssh_multi, concurrency: 50)
ssh_task 'run chef' do
execute 'sudo chef-client --no-fork'
end
Note: if we use the default strategy, ssh_multi driver wont be able to leverage its concurrency features, as the resultant jobs (the driver will receive) will have only one host.
- per host strategy: it creates one job per host. Following example will create 3 jobs. each with one host and 2 ruby tasks. Thus two tasks will be executed in one host, then on the next one.. follow on. Think of deployments with rolling restart like scenarios. This also allows drivers to optimize multiple tasks/commandsi execution against individual hosts (session reuse etc).
strategy :per_host
members ['host1', 'host2', 'host3']
ruby_task 'test 1' do
execute do |host|
Blender::Log.info("test 1 on #{host}")
end
end
ruby_task 'test 2' do
execute do |host|
Blender::Log.info("test 2 on #{host}")
end
end
Note: this strategy does not work if you have different hosts per tasks.
Its fairly easy to write custom scheduling strategies and they can be used to rewrite or rearrange hosts/tasks as you wish. For example, null strategy that return 0 jobs irrespective of what tasks or members you pass, or a custome strategy that takes the hosts lists of every tasks and considers only one of them dynamically based on some metrics for jobs, etc.
Host discovery
For workflows that depends on dynamic infrastructure, where host names are changing, Blender provides abstractions that facilitate discovering them. blender-chef and blender-serf uses this and allows remote job orchestration for chef or serf managed infrastructure.
Following are some examples:
- serf: discover hosts using serf membership
require 'blender/serf'
ruby_task 'print host name' do
execute do |host|
Blender::Log.info("Host: #{host}")
end
members search(:serf, name: '^lt-.*$')
end
- chef: discover hosts using Chef search
require 'blender/discoveries/chef'
ruby_task 'print host name' do
execute do |host|
Blender::Log.info("Host: #{host}")
end
members search(:chef, 'roles:web')
end
Invoking blender periodially with Rufus scheduler
Blender is designed to be used as a standalone script that can be invoked on-demand or
consumed as a library, i.e. workflows are written in plain Ruby objects and invoked
from other tools or application. Apart from these, Blender can be use for periodic
job execution also. Underneath it uses Rufus::Scheduler
to trigger Blender run, after
a fixed interval (can be expressed via cron syntax as well, thanks to Rufus).
Following will run example.rb
blender script after every 4 hours.
schedule '/path/to/example.rb' do
cron '* */4 * * *'
end
Ignore failure
Blender will fail the execution immediately if any of the job fails. ignore_failure
attribute can be used to proceed execution even after failure. This can be declared
both per task level as well as globally.
shell_task 'fail' do
command 'ls /does/not/exists'
ignore_failure true
end
shell_task 'will be executed' do
command 'echo "Thrust is what we need"'
end
Event handlers
Blender provides an event dispatchment facility (inspired from Chef), where arbitrary logic can be hooked into the event system (e.g. HipChat notification handlers, statsd handlers, etc) and blender will automatically invoke them during key events. As of now, events are available before and after run and per job execution. Event dispatch system is likely to get more elaborate and blender might have few common event handlers (metric, notifications etc) in near future.
Ancillary projects
Blender has a few ancillary projects for integration with other systems, following are few of them:
- Zookeeper based locking for distributed blender deployments blender-zk
- Serf based host discovery and command dispatch blender-serf
- Chef based host discovery blender-chef
Supported ruby versions
Blender currently support the following Ruby implementations:
- Ruby 1.9.3
- Ruby 2.1.0
- Ruby 2.1.2
License
Contributing
- Fork it ( https://github.com/PagerDuty/blender/fork )
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request