• Stars
    star
    255
  • Rank 154,567 (Top 4 %)
  • Language
    Ruby
  • Created about 13 years ago
  • Updated about 7 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Vagrant for the cloud

Note: documentation is currently in flux

About

What the hell is mccloud?

Build Status

Over the years I fell in love with Vagrant and wanted to have the same workflow for ec2, kvm, internal clouds etc..

Therefore Mccloud aims to be the equivalent of vagrant but extending it to use providers:

  • aws/ec2
  • kvm
  • simple scripts
  • and of course vagrant itself.

I'm aware vagrant might extend it's providers in the future; still as they are currently not yet implemented I thought I'd share this code with you. As new provider will become available in vagrant they will also be available in mccloud through the vagrant provider

Kudos to great stuff

Without the following opensource software this would not be that awesome!

  • Vagrant is great for testing machines on your local machines
  • Fog is a great fog library for managing cloud systems
  • Fission is a gem to interact with vmware fusion machines

Kudos to the authors!

Why not in vagrant?

NOTE: mccloud is meant to be complementary to vagrant - we truely love @mitchellh

  • Main reason - this code has been around way before there was discussion on vagrant new providers
  • Companies are using it now as it supports EC2, KVM, scripts, NOW
  • Vagrant is (currently) focused on desktop vm types only - this extends it to server based cloud solution
  • Once providers are available in vagrant, you can easily switch: the effort is not in the Mccloudfile or Vagrantfile syntax but in the provisioning
  • Vagrant moves away from the 'gem ' support and targets fat installers - I need this code available as a library
  • Vagrant new setup requires root to be installed - not what I want
  • Vagrant builder (might) replace veewee with new setup - I want to continue working with it now

Bottom line - if vagrant has the new plugin architecture going and documented I'm happy to review again

Installation

Requirements

  • You currently need ruby installed. Either use the system ruby or install rvm
  • libxml, libxslt and nokogiri

Using the stable gem

$ gem install mccloud

Using the cutting edge github code

$ git clone [email protected]:jedi4ever/mccloud.git
$ cd mccloud
# Note when you use rvm , there is an .rvmrc that will set some aliases
$ bundle install
$ bundle exec mccloud

Configuration

Similar to a Vagrantfile, mccloud has a Mccloudfile where all is configured. TODO: there is currently no mccloud init as it's hard to guess your preferred options

Mccloudfile Skeleton

A mccloudfile is actually a ruby code file with a specific block

Mccloud::Config.run do |config|
end

Provider section

As mccloud supports multiple providers , the first part you need to do it define the providers you want to use

Provider AWS

You can use this provider to create/manage ec2 instances.

As this relies on fog, you first have to create a fog configuration file

$ cat $HOME/.fog
:default:
  :aws_access_key_id: <your id here>
  :aws_secret_access_key: <your acess key here>

The syntax to use for an ec2

Mccloud::Config.run do |config|

  # Define a :aws provider 'aws-us-east'
  config.provider.define "aws-us-east" do |provider_config|

    #Note: this are option provided to fog for creation
    provider_config.provider.options = { }

    provider_config.provider.flavor = :aws

    # Region in which to create the VM
    provider_config.provider.region = "us-east-1"

    ## Check if necessary keypairs exist
    ## To speed things up, set it to false
    provider_config.provider.check_keypairs = false

    ## Disable check if required security groups exist
    ## To speed things up, set it to false
    provider_config.provider.check_security_groups =  false

    ## If you share an amazon account with multiple people
    ## You can use namespaces to separate resources
    ## All resources will take this prefix
    provider_config.provider.namespace = ""

    ## Fog credential pair to use in .fog file
    provider_config.provider.credential = :default

  end
end

If using the aws/ec see also the section about defining keystores and keypairs

Provider host

Useful with machines that are only ssh-able and where you don't have create options

Mccloud::Config.run do |config|

  # Define a :host provider 'host-provider' that is ssh-able
  config.provider.define "host-provider" do |provider_config|
    provider_config.provider.flavor = :host
  end

end

Provider vagrant

Have mccloud pick up your Vagrantfile

Mccloud::Config.run do |config|

  # Define a :vagrant provider 'vagrant-provider'
  config.provider.define "vagrant-provider" do |provider_config|
    provider_config.provider.flavor = :vagrant
  end
end

Provider script

Usefull if your cloud doesn't have an ip, but you can create start,stop, etc... scripts to do the work

Mccloud::Config.run do |config|

  datacenter_settings = {
    :DATACENTER => 'belgium',
    :ENVIRONMENT => 'test'
  }

  # Define a :script provider 'script-provider'
  config.provider.define "script-provider" do |provider_config|
    provider_config.provider.flavor = :script


    # environment variables to pass to the scripts
    # these are passed as MCCLOUD_<varname>
    provider_config.provider.variables = datacenter_settings

    # No need for a namespace
    provider_config.provider.namespace = ""

    # location of the start, stop etc.. scripts
    provider_config.provider.script_dir = "myscript-provider"
  end

end

Provider kvm (might not work out of the box)

this works together with veewee that support creating kvm template machines. Like on vagrant, mccloud clones a veewee created vm

config.provider.define "kvm-libvirt" do |config|
  config.provider.flavor=:libvirt
  config.provider.options={ :libvirt_uri => "qemu+ssh://ubuntu@kvmbox/system" }
  config.provider.namespace="test"
end

Keypair section

Currently only used by aws provider. Allows you to define a re-usable name for keypairs for each aws region

config.keypair.define "mccloud" do |key_config|
  key_config.keypair.public_key_path = "#{File.join(ENV['HOME'],'.ssh','mccloud_rsa.pub')}"
  key_config.keypair.private_key_path = "#{File.join(ENV['HOME'],'.ssh','mccloud_rsa')}"
end

Keystore section

Currently only used by aws provider. Allows you to define multiple keystores for your aws keys

config.keystore.define "aws-us-east-key-store" do |keystore_config|
  keystore_config.keystore.provider = "aws-us-east"
  keystore_config.keystore.keypairs = [
    # :name is the name as it will be displayed on amazon
    # :keypair is the named as defined  in the mccloudfile
    { :name => "mccloud", :keypair => "mccloud"},
  ]
end

IP definitions

config.ip.define "ip-demo1" do |config|
  config.ip.provider="aws-eu-west"
  config.ip.address="46.137.72.170"
  config.ip.vmname = "aws-demo1"
end

LB definitions

config.lb.define "mccloud-development-patrick-lb" do |config|
 config.lb.provider="aws-eu-west"
 config.lb.members=["aws-demo2","aws-demo1"]
 config.lb.sorry_members=["aws-demo2"]
end

Template/definitions

TODO

VM definitions

Core vm

Sharing of files is done over rsync because cloud based architectures don't have the ability to mount local folders

vm_config.vm.share_folder("somename", "/source/inthemachinepath", "localmachinepath")

vm_config.vm.bootstrap = "somescript"
vm_config.vm.bootstrap_user = "root"
vm_config.vm.bootstrap_password = "blabla"
vm_config.vm.user = "ubuntu"

vm_config.vm.name
vm_config.vm.port

vm_config.vm.private_key_path
vm_config.vm.public_key_path
vm_config.vm.agent_forwarding
vm_config.vm.autoselection
vm_config.vm.bootstrap
vm_config.vm.bootstrap_user
vm_config.vm.bootstrap_password

vm_config.vm.forward_port

AWS vm

vm.ami
vm.key_name
vm.security_groups = Array
vm.user_data
vm.flavor
vm.user

config.vm.define "demo" do |config|
 config.vm.provider="aws-eu-west"
 config.vm.ami="ami-e59ca991"
 config.vm.flavor="t1.micro"
 config.vm.zone="eu-west-1a"
 config.vm.user="ubuntu"
 config.vm.security_groups=["thesecuritygroup"]
 config.vm.key_name="mccloud-key-patrick"
 config.vm.bootstrap="definitions/ubuntu/bootstrap-ubuntu-system.sh"
 config.vm.private_key_path="keys/mccloud_rsa"
 config.vm.public_key_path="keys/mccloud_rsa.pub"
end

this is the way we are currently mounting EBS Volumes with Mccloud. For attaching an EBS volume created from a Snaphot;

  # see http://fog.io/1.1.2/rdoc/Fog/Compute/AWS/Servers.html
  # and https://github.com/fog/fog/blob/v1.1.2/lib/fog/aws/requests/compute/run_instances.rb
  config.vm.create_options = {
    :block_device_mapping => [
      { "DeviceName" => "/dev/sdf", "Ebs.SnapshotId" =>

"snap-d056d786", "Ebs.DeleteOnTermination" => true } ] }

Or, for attaching a newly created EBS volume:

config.vm.create_options = {
:block_device_mapping => [
{ "DeviceName" => "/dev/sdf", "Ebs.VolumeSize" => "100",
"Ebs.DeleteOnTermination" => false }
]
}

The mounting we then do our provision.sh script:

  echo "/dev/sdf1 /mnt/ebs ext4 defaults 0 0" >> /etc/fstab
  mkdir -p /mnt/ebs
  mount -a

Fog vm

Vagrant vm

config.vm.define "compute1" do |vm_config|
  vm_config.vm.provider   = "vagrant"
end

Host vm

config.vm.define "mycoolhost.com" do |config|
  config.vm.provider=:hosts
  config.vm.ip_address="mycoolhost.com"
  config.vm.user="ubuntu"
  config.vm.port = "2222"
  config.vm.bootstrap  = "bootstrap/centos-603"
  config.vm.agent_forwarding = true
end

KVM vm

config.vm.define "backend" do |config|
 config.vm.provider="juno-libvirt"

 config.vm.create_options={
   :network_interface_type => "bridge",
   :volume_template_name => "test-baseimage.img",
   :cpus => "3",
   :memory_size => 2*1024*1024, #2 GB
 }
 config.vm.user="ubuntu"
 config.vm.bootstrap="definitions/ubuntu/bootstrap-ubuntu-system.sh"
 config.vm.private_key_path="keys/mccloud_rsa"
 config.vm.public_key_path="keys/mccloud_rsa.pub"
end

Vmfusion vm

need to check this

Provisioners

You can use multiple provisioners per vm

Puppet apply provisioner

manifest_file
manifest_path
module_paths = Array
pp_path
options

vm_config.vm.provision :puppet do |puppet|
  puppet_flags = "--verbose --show_diff"
  puppet.manifest_file = "site.pp"
  puppet.pp_path = "/var/tmp/puppet"
  puppet.manifests_path = "puppet/manifests"
  puppet.module_path = [ "puppet/modules" ,"puppet/my-modules"]
  puppet.options = puppet_flags
end

Chef-solo provisioner

cookbooks_path
roles_path
provisioning_path
data_bags_path
encrypted_data_bag_secret_key_path
json
json_erb
clean_after_run
roles

mccloud server is added to json

add_role(name)
add_recipe(name)

 # Read chef solo nodes files
 require 'chef'
 nodes = []
 Dir["data_bags/node/*.json"].each do |n|
       nodes << JSON.parse(IO.read(n))
 end

 nodes.each do |n|
   config.vm.define n.name do |vm_config|
     vm_config.vm.provider   = "host"
     vm_config.vm.ip_address = n.automatic_attrs[:ipaddress]
     vm_config.vm.user       = n.automatic_attrs[:sudo_user]

     vm_config.vm.bootstrap  = File.join("bootstrap","bootstrap-#{n.automatic_attrs[:platform]}.sh")
     vm_config.vm.bootstrap_user  = n.automatic_attrs[:bootstrap_user]
     vm_config.vm.bootstrap_password  = n.automatic_attrs[:bootstrap_password]

     vm_config.vm.provision :chef_solo do |chef|
      chef.cookbooks_path = [ "cookbooks", "site-cookbooks" ]
      chef.roles_path = "roles"
      chef.data_bags_path = "data_bags"
      chef.encrypted_data_bag_secret_key_path = "my_databag_secret"
      chef.clean_after_run = false

      chef.json.merge!(n.default_attrs)
      chef.json.merge!(n.automatic_attrs)
      chef.json.merge!(n.override_attrs)

      chef.add_role n.chef_environment
      chef.add_role n.automatic_attrs[:platform]

      n.run_list.run_list_items.each do |r|
        chef.add_role r.name if r.type == :role
        chef.add_recipe r.name if r.type == :recipe
      end
    end #end provisioner

  end #end vm define
end # nodes.each


 # Using ips in erb json
 chef.json.merge!({
   :logger => {
            :redis_host_ip => "<%= private_ips['frontend'] %>"
        }
 })

 # Defining a default node
  def default_node(chef)
    chef.add_recipe("ntp")
    chef.add_recipe("timezone")
    chef.json.merge!({
      :nagios_host_ip => "<%= private_ips['monitoring'] %>",
      :ruby => {
      :version => "1.9.2",
      :patch_level => "p180"}}
    })
  end

Shell provisioner

option command.sudo = true|false

  config.vm.provision :shell do |command|
    command.inline="uptime"
  end

  config.vm.provision :shell do |command|
    command.path="script.sh"
  end

Usage

Some functions are there in the CLI, but they are left overs from previous coding sessions.

Mostly working

Tasks:

mccloud version                      # Prints the Mccloud version information

mccloud bootstrap [NAME] [FILENAME]  # Executes the bootstrap sequence
mccloud destroy [NAME]               # Destroys the machine
mccloud forward [NAME]               # Forwards ports from a machine to localhost
mccloud halt [NAME]                  # Shutdown the machine
mccloud help [TASK]                  # Describe available tasks or one specific task
mccloud image                        # Subcommand to manage images
mccloud up [NAME]                    # Starts the machine and provisions it
mccloud provision [NAME]             # Provisions the machine
mccloud reload [NAME]                # Reboots the machine
mccloud ssh [NAME] [COMMAND]         # Ssh-shes into the box
mccloud status [name]                # Shows the status of the current Mccloud environment

mccloud lb                           # Subcommand to manage Loadbalancers
mccloud balance [LB-NAME]            # Balances loadbalancers
mccloud sorry [LB-NAME]              # Puts loadbalancers in a sorry state

mccloud ip                           # Subcommand to manage IP's
mccloud ips [NAME]                   # Associate IP addresses

Experimental/Not checked

mccloud define NAME TEMPLATE-NAME    # Creates a new definition based on a tempate
mccloud init                         # Initializes a new Mccloud project

mccloud keypair                      # Subcommand to manage keypairs
mccloud keystore                     # Subcommand to manage keystores

mccloud package [NAME]               # Packages the machine
mccloud template                     # Subcommand to manage templates
mccloud vm                           # Subcommand to manage vms

DISCLAIMER:

this is eternal beta sofware . Don't trust it:) And don't complain if it removes all your EC instances at once....

More Repositories

1

veewee

Easing the building of vagrant boxes
Shell
4,287
star
2

sahara

a plugin for vagrant that allows you manage a sandbox state
Ruby
732
star
3

markdown2confluence

Converting Markdown to Confluence Markup using Kramdown Gem
Ruby
256
star
4

stop-the-fork

Thoughts and Rules to Stop Configuration Management Code from Forking
73
star
5

logstash-cli

Commandline interface for logstash
Ruby
71
star
6

bashpack

turns nodejs projects into a single executable bash file
JavaScript
57
star
7

vagrant-guard-demo

Demo on how cucumber-puppet, rspec-puppet and guard and vagrant can work together
Ruby
54
star
8

devopsdays-webby

Website code for the devopsdays website
HTML
49
star
9

ttyrec.js

A javascript native encode/decoder for ttyrec files/records
JavaScript
36
star
10

gmond-zmq

Puts ganglia gmond information on a zeromq pub/sub
Ruby
34
star
11

puppet-homebrew

a puppet provider for the homebrew packaging system
Ruby
30
star
12

ecs-watch

Simple Service Discovery for use in an AWS ECS cluster
Go
29
star
13

nodejs-ops

my notes on operational things in nodejs
28
star
14

gditc

Gaming/Gpu Desktop in the Cloud
PowerShell
23
star
15

collectd-zmq

Collectd Listener that pushes the metrics onto zeromq pub/sub mechanism
Ruby
21
star
16

puppet-assert

Allows to excute assertion as part of your puppet code
Ruby
19
star
17

devicefarm-cli

A human friendly CLI interface to Amazon's DeviceFarm API - written in Golang
Go
18
star
18

puppet-cic

Puppet Changeset Impact Checker
Ruby
18
star
19

hetzner-cli

CLI to do more complicated Hetnzer tasks like re-installing a server and putting the keys on it
Ruby
17
star
20

veewee-old

Old masterbranch of Veewee project
Shell
17
star
21

slowlane

We are the @fastlane cli - without the magic
Ruby
15
star
22

omapi-dhcp

java interface to the omapi protocol (originally written by [email protected])
Java
13
star
23

jvspherecontrol

commandline tool to control various settings of a vsphere server
Java
13
star
24

gars

Google Analytics Realtime Scraper (using CasperJs)
JavaScript
9
star
25

devopsdays-checklist

Example trello sync for a checklist for devopsdays events
Ruby
9
star
26

ifplayer.js

Interactive Fiction player in nodejs
JavaScript
8
star
27

giraffe-web.js

Giraffe the graphite dashboard wrapped in a webserver and cli
CSS
8
star
28

test-sshd.js

launches an sshd server to test against that will echo commands back
JavaScript
7
star
29

theseboots

Bootstrapping code for getting ruby, chef, puppet, ... on a brand new server
Ruby
7
star
30

mcollective-cpp-agents

Sample code for writing c++ agent for mcollective
C++
7
star
31

cloudrunner

Run a command based on your cloud
JavaScript
7
star
32

veewee-demo

Shell
6
star
33

jvncsender

java library and commandline to send text over vnc
Java
6
star
34

puppet-codeblock

Execute abritrary codeblocks in Puppet
Ruby
5
star
35

mccloud-demo

Demo showing of Mccloud project
Ruby
5
star
36

cloudformation-textmate-bundle

4
star
37

webshooter

Create webshots using rubyCocoa Webkit
Ruby
4
star
38

phantom-check

Check your page for javascript errors, broken links using phantomjs
JavaScript
4
star
39

socialapp

an express/socket.IO/redisstore/terminated project to hook into
JavaScript
3
star
40

openstack-box

Ruby
3
star
41

chef-command.js

Generates the correct opscode chef command, dna.json and config.rb
JavaScript
3
star
42

eventbrite-invoicer

Simple Ruby script to generate invoices from Eventbrite Events
Ruby
2
star
43

chrome-pagecheck

Commandline to get information on page in Google Chrome - errors, timings, etc...
JavaScript
2
star
44

nagios2graphite

Forwards nagios internal metrics to graphite host (UDP)
C
2
star
45

hashmerge

Deep merges hash into a new object
JavaScript
2
star
46

dotfiles

collection of my vim, bash etc ... files in my homedirectory
Vim Script
2
star
47

sudofy.js

contructs the sudo command for a given command and options
JavaScript
2
star
48

terminal-parser.js

Native Javascript/Nodejs terminal escape codes Parser/Stream
JavaScript
2
star
49

grunt-bashpack

Grunt task to create bashpacks from your nodejs project (self-executable bash scripts)
JavaScript
2
star
50

jedi.be-nwebby

HTML
1
star
51

nwebby

Node webby like blog/website system (based on Mustache, Markdown, yaml config files)
JavaScript
1
star
52

mds-sinatra

sinatra demo
Ruby
1
star
53

node-fetcher.js

Fetches the nodejs binary from nodejs.org
JavaScript
1
star
54

livestatus

Livestatus is a simple Ruby library to control Nagios via MK Livestatus
Ruby
1
star