• Stars
    star
    117
  • Rank 301,828 (Top 6 %)
  • Language
    Go
  • License
    BSD 3-Clause "New...
  • Created almost 7 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

FaaS platform for running raw Go functions.

Sky Island

GoDoc License Version

Experimental / ALPHA stage

Sky Island is a FaaS platform built utilizing FreeBSD jails, on ZFS, for running raw Go functions, with interaction through a REST API and inspired by tidwall.

How It Works

A request comes in to run a function. The request contains a git URL to a Go repository containing the function. The request also contains the "call". The call is what will be run including any arguments.

Upon successfully accepting the inbound request, Sky Island will check if there's a binary already for that repo and if so, will move to executing it. If there isn't, Sky Island will check to see if the repo exists. If not, it clones the repo, however if it does, it'll move on to the compile step and generate a "main.go" file and compile a binary in the "build" jail. The "build" jail holds all of the cloned repositories and will be reused on each request unless otherwise told not to. Once a binary is created, an execution jail is created, the binary is copied into it, and executed. The binary's output is returned to the caller via an HTTP response to the original request.

Examples

Simple Call

curl --silent -XPOST http://demo.skyisland.io:3280/api/v1/function -d '{"url": "github.com/mmcloughlin/geohash", "call": "Encode(100.1, 80.9)"}'

Cache Bust Call

curl --silent -XPOST http://demo.skyisland.io:3280/api/v1/function -d '{"url": "github.com/mmcloughlin/geohash", "call": "Encode(100.1, 80.9)", "cache_bust": true}'

Result

{"timestamp":1513717061,"data":"jcc92ytsf8kn"}

Use Cases

  • Utilize existing Go code in any application
  • Build workflows and pipelines with Sky Island as the execution engine
  • Testing
  • Low overhead

Requirements

  • lib32.txz installed
  • ZFS
  • Go version >= 1.9
  • Make sure that jail_enabled="YES" is present in the "/etc/rc.conf" file

System Initialization

Initialzing the system does a number of things to make running Sky Island easier. Sky Island will check to see if the base system packages and Go tarball have already been downloaded and if they have, they'll use those.

  • Create a ZFS dataset to work from
  • Download the base package for the version of FreeBSD you have installed
  • Extract those packages to the dataset where the base jail will be kept
  • Update the base jail with freebsd-update
  • Set some basic jail configuration
  • Install Go and create a workspace
  • Create a ZFS snapshot of the base jail
  • Create build jail

This is accomplished by running:

sky-island -c config.json -i

Installation

go install will install the Sky Island binary into the Go bin directory in the GOPATH.

The above can be adequate however for some folks, you might want to have Sky Island controlled through the RC system. An RC script is included as well as a target in the Makefile to install it. make install

VirtualBox Appliance

A VirtualBox appliance is provided via a Packer build here to allow for easier testing.

Running Sky Island

To run Sky Island, run the command below.

sky-island -c config.json

IP Address Management

The Sky Island config file has an IP4 section to configure how it handles jails IP addressing. If a request is received that indicates a jail needs an IP address, Sky Island checks to see if there is an available address and returns one to be assigned to the execution jail. Use the admin API, described below, to manage the IP pool and to see which jail is associated with which IP and visa versa.

The subnet that Sky Island exists on should have DHCP turned off or at a minimum, make sure that the IP pools aren't overlapping.

There will be a future effort to support multiple IP4 pools.

Caching

Sky Island tries it's best to respond to API requests as quickly as possible. To achieve this, a number of caching mechanisms have been implemented for binaries and repositories. Upon receiving a request via the API, Sky Island will check to see if there's an associated binary that's already been compiled. If there is, that artifact is used. If there's no binary, Sky Island checks to see if the repository has been seen before and if so, uses the repo on disk and compiles a binary from there. The binary will be added to the binary cache for later use.

This cache can be busted however by including cache_bust=true in payload of a "function run" POST request. This will force Sky Island to clone the repo and build a new binary.

API

The Sky Island API provides insight into the Sky Island system. The healthcheck endpoint is not protected by header auth however the admin endpoints are. This can be configured by fields in the config.json file by setting the 'admin_api_token' and 'admin_token_header' fields.

Method Resource Description
GET /healthcheck Verifies the service is up and running
POST /api/v1/function Endpoint that receives function run requests
GET /api/v1/admin/api-stats API statistics
GET /api/v1/admin/jails Get a list of the running jails
GET /api/v1/admin/jail/{id} Get the details for the given jail
DELETE /api/v1/admin/jail/{id} Kill the jail with the given ID
DELETE /api/v1/admin/jails Kill all jails
GET /api/v1/admin/ips Get a list of IP's filtered by param. `?state={available
PUT /api/v1/admin/ips Update the state of a given IP

Metrics

By default, Sky Island uses StatsD to write out metrics. Jail created/removed counts, request times, etc are reported.

Contact

Brian Downs @bdowns328

License

Sky Island source code is available under the BSD 3 Clause License.

alt text

More Repositories

1

spinner

Go (golang) package with 90 configurable terminal spinner/progress indicators.
Go
2,150
star
2

simple-httpd

Drop-in replacement for Python SimpleHTTPServer. Provides TLS via Let's Encrypt over HTTP2, and auto generated self-signed certificates.
Go
230
star
3

openweathermap

Go (golang) package for use with openweathermap.org's API.
Go
224
star
4

jail

Package jail provides native FreeBSD Jail syscalls in Go
Go
52
star
5

super-hacker

SuperHacker is the ultimate utility to make you look like a hacker.
Go
50
star
6

libspinner

A pure C single-header library with 90 configurable terminal spinner/progress indicators.
C
29
star
7

remindme

Go
28
star
8

todo-view

Extract info from TODO's in your code from formatted TODO comments
Go
27
star
9

liblogger

A simple and performant, single-header JSON structured logger for C applications.
C
16
star
10

GoPasswordUtilities

Simple library for working with passwords in Go (golang).
Go
13
star
11

pass

Simple CLI Password Manager
C
12
star
12

flotsam

Project Generator and Dependency Manager for C.
C++
11
star
13

heads-up

A simple example of using Tile38 to track the ISS and send notifications when it's overhead.
Go
10
star
14

presentations

Public repo for my slides
Go
9
star
15

simple-map

A simple and dynamic hash map in C.
C++
9
star
16

stockwatcher

A CLI application to monitor one or more stocks.
Go
7
star
17

smile

Smile. Why not?
Go
7
star
18

libdotenv

libdotenv is a C single-headerlibrary providing the ability to load values from a .env file into environment variables.
C
6
star
19

recipes

A collection of recipes I've created, found, and have been given.
6
star
20

struCtured-logger

A simple JSON structured logger for C applications.
C++
6
star
21

raceway

OpenStack Rally Job Controller
Go
6
star
22

roam-client

Roam by Tile38 API Client
Go
6
star
23

neustar

Package neustar is a SDK to access and work with the Neustar API
Go
5
star
24

cgo_calc

IGNORE. Meant as a descripton of some tech for ${DAY_JOB}
Go
5
star
25

nlog

Structured Logging for Nim
Nim
5
star
26

libmap

A simple map implementation in C
C
5
star
27

libweyl

Middle Random Weyl Sequence
C++
4
star
28

aion

WIP - Job and Task Scheduler Utilizing Cron Syntax
Go
4
star
29

ip

Go
4
star
30

pyopenvswitch

Python module to control and interact with Open vSwitch switches.
Python
2
star
31

doom-fire-c

C
2
star
32

stock-exchange

Go
2
star
33

libbenchmark

loads is a simple library to benchmark C functions across a given number of threads.
C++
2
star
34

mapledb

2
star
35

down-low

down-low is a secure messaging system written in Go (golang) with the ability to send AES and RSA encrypted messages to a number of different endpoints.
Go
2
star
36

games

Terminal games written in Go (golang)
Go
2
star
37

wanbli

1
star
38

gen-release-notes

Go
1
star
39

vms

Shell
1
star
40

kahless

Makefile
1
star
41

gail

Go
1
star
42

jail-builder

Go
1
star
43

logcolor

Go
1
star
44

dictu_modules

C
1
star
45

formatifier

formatifier is a Go (golang) package to quickly format strings in common formats.
Go
1
star
46

daily_tracker

C
1
star
47

r_time

C library providing the ability to get time from remote servers.
C
1
star
48

pwc

Pascal
1
star
49

anagram-primes

C
1
star
50

dictum

Makefile
1
star
51

itsnotgolang

Go
1
star
52

micro-chat

C
1
star
53

dictu.vim

Vim Script
1
star
54

Felix

Felix is an application to be run on Asterisk servers and Kamalio servers when there's an environment needing a process to register in and out carriers and Asterisk servers from the cluster.
Java
1
star
55

fips_dockerfiles

Dockerfile
1
star