• This repository has been archived on 21/Jan/2020
  • Stars
    star
    2,249
  • Rank 20,314 (Top 0.5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 8 years ago
  • Updated almost 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A toolkit for creating and managing declarative, self-healing infrastructure.

InfraKit

CircleCI

Go Report Card

InfraKit is a toolkit for infrastructure orchestration. With an emphasis on immutable infrastructure, it breaks down infrastructure automation and management processes into small, pluggable components. These components work together to actively ensure the infrastructure state matches the user's specifications. InfraKit therefore provides infrastructure support for higher-level container orchestration systems and can make your infrastructure self-managing and self-healing.

To get started, try the tutorial, or check out the video below:

InfraKit + LinuxKit POC

infrakit+linuxkit

In this video, InfraKit was used to build a custom linux operating system (based on linuxkit). We then deployed a cluster of virtual machine instances on a local Mac laptop using the Mac Xhyve hypervisor (HyperKit). A cluster of 3 servers booted up in seconds. Later, after the custom OS image has been updated with a new public key, InfraKit detects the change and orchestrates a rolling update of the nodes. We then deploy the same OS image to a bare-metal ARM server running on Packet.net, where the server uses custom ipxe boot directly from the localhost. It demonstrates some of the key concepts and components in InfraKit and shows how InfraKit can be used to implement an integrated workflow from custom OS image creation to cluster deployment and Day N management. The entire demo is published as a playbook, and you can create your own playbooks too.

Use Cases

InfraKit is designed to automate setup and management of infrastructure in support of distributed systems and higher-level container orchestration systems. Some of the use cases we are working on include:

  • Bootstrap / installation of container orchestration systems like Docker Swarm and Kubernetes
  • Cluster autoscaler that can work across a variety of platforms from public clouds (like AWS autoscaling groups) to bare-metal hosts.
  • GPU cluster provisioning
  • Integration with LinuxKit for building and deploying immutable infrastructure from declarative specifications of the entire stack: from infrastructure resources to os / kernel and applications.
  • Day-N management and automation of infrastructure - from provisioning to rolling updates and capacity scaling.

InfraKit has a modular architecture with a set of interfaces which define the interactions of these 'plugin objects'. Plugins are active daemons that cooperate with one another to ensure the infrastructure state matches your specifications.

Plugins

InfraKit makes extensive use of Plugins to manage arbitrary systems in diverse environments, which can be composed to meet different needs. See the plugins documentation for more technical details.

Here is a list of plugins:

Core Implementations

plugin type description
group group core group controller for rolling updates, scale group, etc.
swarm flavor runs Docker in Swarm mode
kubernetes flavor bootstraps a single master kubernetes cluster
combo flavor combine multiple flavor plugins
vanilla flavor manual specification of instance fields
aws instance creates Amazon EC2 instances and other resource types
digitalocean instance creates DigitalOcean droplets
docker instance provisions container via Docker
google instance Google Cloud Platform compute instances
file instance useful for development and testing
hyperkit instance creates HyperKit VMs on Mac OSX
libvirt instance provisions KVM vms via libvirt
maas instance bare-metal provisioning using Ubuntu MAAS
packet instance provisions bare metal hosts on Packet
rackhd instance bare-metal server provisioning via RackHD
terraform instance creates resources using Terraform
vagrant instance creates Vagrant VMs
vsphere instance creates VMWare VMs

Community Implementations

plugin type description
HewlettPackard/infrakit-instance-oneview instance bare-metal server provisioning via HP-OneView
IBM Cloud instance Provisions instances on IBM Cloud via terraform
AliyunContainerService/infrakit.aliyun instance Provisions instances on Alibaba Cloud
1and1/infrakit-instance-oneandone instance Provisions instances on 1&1 Cloud Server
sacloud/infrakit-instance-sakuracloud instance Provisions instances on Sakura Cloud

Have a Plugin you'd like to share? Submit a Pull Request to add yourself to the list!

Building

Your Environment

Make sure you check out the project following a convention for building Go projects. For example,

# Install Go - https://golang.org/dl/
# Assuming your go compiler is in /usr/local/go
export PATH=/usr/local/go/bin:$PATH

# Your dev environment
mkdir -p ~/go
export GOPATH=!$
export PATH=$GOPATH/bin:$PATH

mkdir -p ~/go/src/github.com/docker
cd !$
git clone [email protected]:docker/infrakit.git
cd infrakit

We recommended go version 1.9 or greater for all platforms.

Also install a few build tools:

make get-tools

Running tests

$ make ci

Binaries

$ make binaries

Executables will be placed in the ./build directory. There is only one executable infrakit which can be used as CLI and as server, based on the CLI verbs and flags.

Design

Configuration

InfraKit uses JSON for configuration because it is composable and a widely accepted format for many infrastructure SDKs and tools. Since the system is highly component-driven, our JSON format follows simple patterns to support the composition of components.

A common pattern for a JSON object looks like this:

{
   "SomeKey": "ValueForTheKey",
   "Properties": {
   }
}

There is only one Properties field in this JSON and its value is a JSON object. The opaque JSON value for Properties is decoded via the Go Spec struct defined within the package of the plugin -- for example -- vanilla.Spec.

The JSON above is a value, but the type of the value belongs outside the structure. For example, the default Group Spec is composed of an Instance plugin, a Flavor plugin, and an Allocation:

{
  "ID": "name-of-the-group",
  "Properties": {
    "Allocation": {
    },
    "Instance": {
      "Plugin": "name-of-the-instance-plugin",
      "Properties": {
      }
    },
    "Flavor": {
      "Plugin": "name-of-the-flavor-plugin",
      "Properties": {
      }
    }
  }
}

The group's Spec has Instance and Flavor fields which are used to indicate the type, and the value of the fields follow the pattern of <some_key> and Properties as shown above.

The Allocation determines how the Group is managed. Allocation has two properties:

  • Size: an integer for the number of instances to maintain in the Group
  • LogicalIDs: a list of string identifiers, one will be associated with each Instance

Exactly one of these fields must be set, which defines whether the Group is treated as 'cattle' (Size) or 'pets' (LogicalIDs). It is up to the Instance and Flavor plugins to determine how to use LogicalID values.

As an example, if you wanted to manage a Group of NGINX servers, you could write a custom Group plugin for ultimate customization. The most concise configuration looks something like this:

{
  "ID": "nginx",
  "Plugin": "my-nginx-group-plugin",
  "Properties": {
    "port": 8080
  }
}

However, you would likely prefer to use the default Group plugin and implement a Flavor plugin to focus on application-specific behavior. This gives you immediate support for any infrastructure that has an Instance plugin. Your resulting configuration might look something like this:

{
  "ID": "nginx",
  "Plugin": "group",
  "Properties": {
    "Allocation": {
      "Size": 10
    },
    "Instance": {
      "Plugin": "aws",
      "Properties": {
        "region": "us-west-2",
        "ami": "ami-123456"
      }
    },
    "Flavor": {
      "Plugin": "nginx",
      "Properties": {
        "port": 8080
      }
    }
  }
}

Once the configuration is ready, you will tell a Group plugin to

  • watch it
  • update it
  • destroy it

Watching the group as specified in the configuration means that the Group plugin will create the instances if they don't already exist. New instances will be created if for any reason existing instances have disappeared such that the state doesn't match your specifications.

Updating the group tells the Group plugin that your configuration may have changed. It will then determine the changes necessary to ensure the state of the infrastructure matches the new specification.

Docs

Additional documentation can be found here.

Reporting security issues

The maintainers take security seriously. If you discover a security issue, please bring it to their attention right away!

Please DO NOT file a public issue, instead send your report privately to [email protected].

Security reports are greatly appreciated and we will publicly thank you for it. We also like to send giftsโ€”if you're into Docker schwag, make sure to let us know. We currently do not offer a paid security bounty program, but are not ruling it out in the future.

Design goals

InfraKit is currently focused on supporting setup and management of base infrastructure, such as a cluster orchestrator. The image below illustrates an architecture we are working towards supporting - a Docker cluster in Swarm mode.

arch image

This configuration co-locates InfraKit with Swarm manager nodes and offers high availability of InfraKit itself and Swarm managers (using attached storage). InfraKit is shown managing two groups - managers and workers that will be continuously monitored, and may be modified with rolling updates.

Countless configurations are possible with InfraKit, but we believe achieving support for this configuration will enable a large number of real-world use cases.

Copyright and license

Copyright ยฉ 2016 Docker, Inc. All rights reserved. Released under the Apache 2.0 license. See LICENSE for the full license text.

More Repositories

1

classicswarm

Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit
Go
5,759
star
2

docker-ce

โš ๏ธ This repository is deprecated and will be archived (Docker CE itself is NOT deprecated) see the https://github.com/docker/docker-ce/blob/master/README.md โš ๏ธ
Go
5,747
star
3

toolbox

The Docker Toolbox
Inno Setup
3,215
star
4

docker-registry

This is **DEPRECATED**! Please go to https://github.com/docker/distribution
Python
2,885
star
5

libcontainer

PROJECT MOVED TO RUNC
Go
1,189
star
6

compose-cli

Easily run your Compose application to the cloud with compose-cli
Go
955
star
7

dockercloud-haproxy

HAproxy image that autoreconfigures itself when used in Docker Cloud
Python
651
star
8

dockerlite

Lightweight virtualization system based on LXC and BTRFS. See dotcloud/docker.
Shell
496
star
9

docker-birthday-3

480
star
10

ecs-plugin

See http://github.com/docker/compose-cli
Go
411
star
11

go-redis-server

A Go implementation of the Redis server protocol
Go
404
star
12

communitytools-image2docker-win

PowerShell
304
star
13

openstack-docker

Nova driver and Glance backend to use Docker inside OpenStack
Python
293
star
14

go-p9p

A modern, performant 9P library for Go.
Go
206
star
15

jenkins-pipeline-scripts

Groovy
189
star
16

swarm-frontends

Deploy Kubernetes with Swarm
Shell
182
star
17

leeroy

Jenkins integration with GitHub pull requests
Go
177
star
18

migrator

Tool to migrate Docker images from Docker Hub or v1 registry to a v2 registry
Shell
160
star
19

leadership

Distributed Leader Election using docker/libkv
Go
160
star
20

go-healthcheck

Go
146
star
21

global-hack-day-3

Participant final submissions for the 3rd edition of the Docker Global Hack Day
Go
126
star
22

infra-container_exporter

Prometheus exporter exposing container metrics
Go
126
star
23

docker-snap

Shell
120
star
24

communitytools-image2docker-linux

Go
116
star
25

boatyard

[DEPRECATED] Web UI to docker build
Go
114
star
26

docker-tutorial

JavaScript
101
star
27

for-aws

92
star
28

dnsserver

A small DNS service for communicating A and SRV records.
Go
82
star
29

swarm-microservice-demo-v1

Swarm Web App Demo Cluster: Voting App
JavaScript
81
star
30

golem

Integration test framework using Docker
Go
76
star
31

dockercloud-cli

CLI for Docker Cloud
Python
74
star
32

dcus-hol-2017

Dockercon 2017 Hands-On Labs
JavaScript
61
star
33

infra-docker-lloyd

Shipment insurance for docker containers
Shell
55
star
34

docker-network

Experimental network control tool for Docker
Go
48
star
35

ucp_lab

Lab for UCP at DockerCon EU
48
star
36

ee-workshop

43
star
37

bender

IRC bot to manage scrum-like standup meetings
Python
43
star
38

dockercloud-authorizedkeys

Adds a user public SSH key to the host's ~/.ssh/authorized_keys via docker container
Shell
42
star
39

infra-reefer

Managing a stable environment in your container.
Go
41
star
40

dcus-hol-2016

Python
41
star
41

minecraft

Minecraft Dockerfile used in Kitematic
Dockerfile
39
star
42

oscalkit

NIST OSCAL SDK and CLI
Go
36
star
43

gordon-bot

Gordon the turtle IRC bot to rebuild Jenkins builds.
Go
34
star
44

dockercloud-hello-world

Hello world!
PHP
34
star
45

dockercloud-agent

Agent to manage docker in nodes controlled by Docker Cloud
34
star
46

openstack-heat-docker

Orchestrate Docker containers with OpenStack Heat
Python
32
star
47

cloud-integration-beta

Docker CLI with ACI integration (beta)
Shell
32
star
48

modernizing_apps_for_java_devs

Java
31
star
49

swarm-library-image

Official Image packaging for Classic Swarm, now archived
Shell
31
star
50

kitematic-html

Shell
30
star
51

serverlessbenchmark

Deploying and benchmarking open source Function-As-A-Service frameworks
HCL
29
star
52

infra-docker-proper

OS. Proper swabbing the deck - Cleanup of old Docker containers and images.
Go
29
star
53

dotnet-musicstore

Music Store web app written in .NET Core
C#
28
star
54

docker-bb

Docker binary builder, triggered from nsq messages.
Go
28
star
55

python-dockercloud

Python library for Docker Cloud
Python
28
star
56

for-azure

27
star
57

docker-status

Python
26
star
58

v1.10-migrator

Prepare docker graph directory for content addressability
Go
24
star
59

registration-docker

Tutorial for debugging Java applications in Docker
Java
22
star
60

infrakit.aws

Infrakit plugins for AWS.
Go
21
star
61

stacks

Go
21
star
62

hybrid-workshop

HTML
21
star
63

infra-prometheus-consul

Prometheus with dynamic configuration by consul and consul-template
Shell
20
star
64

dockercloud-node

Docker Cloud node as a docker-in-docker container
Shell
19
star
65

pulpo

Configuration mechanism for Node
TypeScript
18
star
66

mta-netfx-dev

Sample code from video series: Modernizing .NET Apps with Docker - for Developers
16
star
67

go-dockercloud

Go library for Docker Cloud API
Go
15
star
68

nginx-gohttp

Go
15
star
69

web-application-firewall

Sample cross-platform Web Application Firewall, used as a security proxy for existing web apps
C#
15
star
70

docker-for-oracle-weblogic

Shell
14
star
71

windows-container-utility

Windows Container Utility
C++
13
star
72

mta_java_workshop

MTA Java Workshop for Dockercon 2018
Java
12
star
73

docker-pets

Docker Pets is a simple application that's useful for testing out features of Docker Enterprise Edition.
Python
12
star
74

infrakit.gcp

Infrakit plugins for Google Cloud Platform.
Go
12
star
75

markdownlint

Docker projects markdown linter
Go
11
star
76

infrakit.digitalocean

DEPRECATED: Infrakit plugin(s) for Digitalocean
Go
11
star
77

varnish

Dockerized Varnish server image
Shell
10
star
78

birthdaysite

Docker's Birthday Site
CSS
10
star
79

dockercloud-network-daemon

[Docker Cloud System Image] Provides a secure overlay network between nodes
10
star
80

irc-minutes

Docker-dev irc meeting Organization and minutes.
10
star
81

buildx-packaging

Linux distro and static packaging for buildx
HCL
10
star
82

dnsutils

Docker image with dnsutils installed
9
star
83

tomcat_cluster

Java
8
star
84

dockercloud-quickstart-python

Getting started in Docker Cloud with Python
Python
8
star
85

ee-exercises

7
star
86

moby-extras

for stitching together docker-ce
Go
7
star
87

dockercloud-quickstart-go

Getting started in Docker Cloud with Go
Go
6
star
88

quickstart-node

Getting started in Tutum with Node.js
JavaScript
6
star
89

quickstart-go

Getting started in Tutum with Go
Go
6
star
90

dockercloud-events

[Docker Cloud System Image] Monitors docker events and sends updates to Docker Cloud's API
Go
6
star
91

infra-heka_dashboard_exporter

prometheus exporter for heka logging daemon
Go
5
star
92

vol-test

Go
5
star
93

licensing

Library for interacting with Docker issued product licenses
Go
5
star
94

homebrew-oscalkit

Homebrew recipe for oscalkit
Ruby
5
star
95

app-e2e

Go
4
star
96

mulesoft-docker-demo

C#
3
star
97

hybrid-app

Java
2
star
98

dctx

A context package for use across docker projects
Go
2
star
99

virtualbox

Mirror of VirtualBox
2
star
100

notary-signer-image

2
star