• Stars
    star
    445
  • Rank 98,085 (Top 2 %)
  • Language
    JavaScript
  • Created almost 13 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Task orchestration, creation and running using NodeJS

Tasks Workflows Orchestration API and Runners.

This repository is part of the Joyent Triton project. See the contribution guidelines and general documentation at the main Triton project page.

Installation

npm install wf

Overview

If you are building a completely new system composed of many discrete API applications, each of them with a clearly defined area of responsibility, or if you are trying to assemble a collaboration channel between a heterogeneous set of unrelated API applications, you need a means to orchestrate interactions between these applications.

A workflow is effectively an orchestration. It gives you a way to decompose a complex series of operations down to a sequence of discrete tasks within a state machine.

The sequence of tasks is more complex than just a series. Tasks can fail, and so you need to deal with timeouts, retries, "stuck" flows, and so forth.

One way to define a workflow and its tasks is using an arbitrarily-complex language. Or you can keep it simple by making some assumptions:

  • Code is the definition language.
  • Tasks are independent. Can be used into different workflows.
  • The only way to communicate between tasks is the workflow. Tasks can add, remove or modify properties of the workflow.
  • If a task requires a specific property of the workflow to be present, the task can fail, or re-schedule itself within the workflow, or ...

wf

This package provides a way to define re-usable workflows using JSON and run concrete jobs with specific targets and parameters based on such workflows.

Workflow Runners

In order to execute jobs, at least one WorkflowRunner must be up and ready to take jobs. An arbitrary number of runners can be used on any set of hosts; their configuration must match.

An example WorkflowRunner is provided with the package and can be started with:

./bin/workflow-runner path/to/config.json

(The test directory contains the file config.json.sample which can be used as reference).

You can create workflows and jobs either by using the provided REST API(s), or by embedding this module's API into your own system(s). The former will be easier to get up and running, but you should use the latter when:

  • You want to use the Workflow API in a (node.js) application that is not the bundled REST API.
  • You want to use a different backend storage system, or otherwise change the assumptions of the bundled REST API.

The package also provides a binary file to run the WorkflowAPI using the same configuration file we pass to our WorkflowRunner:

./bin/workflow-api path/to/config.json

See demo section below for more details about both approaches.

Development

Clone the repo and build the deps:

git clone git://github.com/joyent/node-workflow.git
cd node-workflow
make all

Note make all will setup all the required dependencies, node modules and run make check and make test. In order to just setup node modules, make setup is enough.

To run the Workflow API server:

./bin/workflow-api path/to/config.json

To run a Job Runner:

./bin/workflow-runner path/to/config.json

Note that it's fine to run more than one Runner, either on the same or different machines, so long as they have access to the same storage backend.

Testing

make test

Pre-commit git hook

In order to run make prepush before every commit, add the following to a file called .git/hooks/pre-commit and chmod +x it:

#!/bin/sh
# Run make prepush before allow commit

set -e

make prepush

exit 0

If you've made a change that does not affect source code files, but (for example) only docs, you can skip this hook by passing the option --no-verify to the git commit command.

Demo

The workflow-example repository contains everything needed to illustrate:

  • An example config file config.json.sample which should be renamed to config.json, and modified to properly match your local environment.

Remember that, in order to process any job the workflow-runner needs to be initialized pointing to the aforementioned configuration file:

./node_modules/.bin/workflow-runner config.json

Also, in order to be able to run the API-based example mentioned below, the workflow-api HTTP server needs to be up and running too:

./node_modules/.bin/workflow-api config.json

Contents for the other files within the workflow-example repository are:

  • An example of how to use node-workflow as a node module in order to create workflows, queue jobs and wait for the results. See module.js.
  • An example of how to achieve the same goal using the Workflow API instead of the node module. See api.js.
  • Both examples share the same workflow definition, contained in the file shared-workflow.js. The beginning of the aforementioned files can be useful to understand the differences when trying to create a workflow using these different approaches.
  • Finally, this directory also contains the file node.js, which does exactly the same thing as the workflow/job does -- create and star a gist using your github's username and password -- but straight from node.js. This file is useful in order to understand the differences between writing code to be executed by node.js directly, and using it to create workflows and the associated tasks. Remember, code within tasks runs sandboxed using Node's VM API and that tasks are totally independent.

See also example.js for more options when defining workflows and the different possibilities for task fallbacks, retries, timeouts, ...

Repository

deps/           Git submodules and/or commited 3rd-party deps.
                See "node_modules/" for node.js deps.
docs/           Project docs (restdown)
lib/            Source files.
node_modules/   Node.js deps, either populated at build time or commited.
pkg/            Package lifecycle scripts
test/           Test suite (using node-tap)
tools/          Miscellaneous dev/upgrade/deployment tools and data.
Makefile
package.json    npm module info
README.md

TODO

See https://github.com/joyent/node-workflow/issues.

LICENSE

The MIT License (MIT) Copyright (c) 2018 Joyent, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

More Repositories

1

libuv

Go to
C
3,271
star
2

smartos-live

For more information, please see http://smartos.org/ For any questions that aren't answered there, please join the SmartOS discussion list: http://smartos.org/smartos-mailing-list/
C
1,437
star
3

triton

Joyent Triton DataCenter: a cloud management platform with first class support for containers.
Shell
1,202
star
4

node-verror

Rich JavaScript errors
JavaScript
1,125
star
5

containerpilot

A service for autodiscovery and configuration of applications running in containers
Go
1,104
star
6

manta

Manta is a scalable HTTP-based object store
Makefile
565
star
7

node-http-signature

Reference implementation of Joyent's HTTP Signature Scheme
JavaScript
392
star
8

node-stackvis

Stacktrace visualization tools
JavaScript
340
star
9

node-vasync

utilities for observable asynchronous control flow
JavaScript
315
star
10

v8plus

Node.js native add-ons in C
C++
265
star
11

rfd

Requests for Discussion
Roff
251
star
12

mdb_v8

postmortem debugging for Node.js and other V8-based programs
C
235
star
13

manatee

Automated fault monitoring and leader-election system for strongly-consistent, highly-available writes to PostgreSQL (Joyent SDC, Manta).
JavaScript
228
star
14

statemap

Software for rendering statemaps
Rust
219
star
15

restdown

Pretty REST API docs authored in Markdown
Python
203
star
16

sdc-docker

Docker Engine for Triton
JavaScript
182
star
17

triton-kubernetes

Kubernetes on Triton
Go
174
star
18

node-sshpk

Parse, convert, fingerprint and use SSH keys in pure node.js
JavaScript
159
star
19

nodejs-advisory-board

Meeting Minutes and Working Group Discussions
158
star
20

nhttpsnoop

Trace Node.js HTTP server activity
Shell
138
star
21

pgsqlstat

report top-level postgres stats
Shell
129
star
22

node-panic

Postmortem debugging facility for Node.js
JavaScript
120
star
23

node-assert-plus

Extra assertions on top of node's assert module
JavaScript
119
star
24

illumos-kvm

KVM driver for illumos
C
117
star
25

node-snmpjs

SNMP toolkit for Node.js
JavaScript
111
star
26

node-ctype

Read and write binary structures with node
JavaScript
89
star
27

node-manta

Node.js SDK for Manta
JavaScript
75
star
28

node-bunyan-syslog

Syslog Stream for node-bunyan
JavaScript
68
star
29

illumos-kvm-cmd

qemu-kvm for illumos-kvm
C
65
star
30

node-watershed

Simple WebSockets Client/Server (RFC6455)
Makefile
65
star
31

node-smartdc

Client SDK and CLI for the Joyent SmartDataCenter API
JavaScript
63
star
32

mi-centos-7

Shell
63
star
33

node-asn1

Contains parsers and serializers for ASN.1 (currently BER only)
AGS Script
61
star
34

smartos_cookbooks

Chef Cookbooks for managing the SmartOS Global Zone
JavaScript
58
star
35

moray

Moray, the highly-available key/value store (Joyent Triton, Manta)
JavaScript
58
star
36

node-vstream

instrumented streams
JavaScript
56
star
37

node-triton

Triton client tool and node.js library
JavaScript
55
star
38

node-docker-registry-client

node.js client for the docker registry
JavaScript
55
star
39

kang

Introspection for distributed systems
JavaScript
49
star
40

smfgen

Generate SMF manifests from a JSON description
JavaScript
49
star
41

jsstyle

cstyle-based JavaScript style checker
Perl
49
star
42

node-debug-school

nodeschool curriculum for debugging Node.js
JavaScript
49
star
43

node-getopt

POSIX-style getopt() for Node.js
JavaScript
47
star
44

dtruss-osx

Shell
43
star
45

node-ip6addr

IPv6/IPv4 address parsing and manipulation for node.js
JavaScript
43
star
46

pg_prefaulter

Faults pages into PostgreSQL shared_buffers or filesystem caches in advance of WAL apply
Go
43
star
47

node-camp

Asynchronous IO ...camp
JavaScript
43
star
48

manatee-state-machine

design ideas for manatee
JavaScript
42
star
49

node-docker-file-parser

Parses a dockerfile contents string and returns the array of docker commands
JavaScript
42
star
50

smartos-vmtools

Shell
40
star
51

illumos-extra

Extra non-ON software required for Illumos
C
39
star
52

sdc-nfs

user-level NFS server written in node.js
JavaScript
35
star
53

node-extsprintf

Extended POSIX-style sprintf
JavaScript
34
star
54

node-kstat

A node.js addon for reading illumos kstats
Perl
32
star
55

node-jsprim

utilities for primitive JavaScript types
JavaScript
32
star
56

knife-joyent

Opscode Chef knife plug-in for Joyent CloudAPI
Ruby
32
star
57

eng

Joyent Engineering Guide
JavaScript
31
star
58

pkgsrc-joyent

Various pkgsrc packages used by Joyent, not committed upstream yet
Makefile
31
star
59

smartos-overlay

Overlay directory specific to open-source SmartOS
30
star
60

node-fast

streaming JSON RPC over TCP
JavaScript
29
star
61

convertvm

convert OVF vm packages to smartos compatible images
JavaScript
29
star
62

minecrab

Minecraft on Joyent's Cloud & Manta on Demand
Shell
28
star
63

cloud-perf-labs

Student labs for Cloud Performance training
C
28
star
64

node-consulite

Tiny consul Node.js module for client discovery
JavaScript
28
star
65

node-piloted

Service discovery in node using ContainerPilot
JavaScript
27
star
66

node-in-the-industry

This is the script that used to generate fresh "node in the industry" content. It is no longer being maintained. See: https://github.com/nodejs/nodejs.org.
HTML
27
star
67

mi-freebsd-10

Custom FreeBSD 10 ISO builder
Shell
26
star
68

javascriptlint

JavaScript Lint
C
25
star
69

binder

Triton/Manta DNS server over Apache Zookeeper
JavaScript
25
star
70

node-tracing

User definable tracing API
JavaScript
25
star
71

python-manta

Python SDK for Manta (community maintained)
Python
24
star
72

manufacturing

Manufacturing specifications
Python
24
star
73

pglockanalyze

analyze postgres locking behavior
Makefile
23
star
74

sdcboot

SDC FDUM environment
C
23
star
75

pkgsrc-wip

Conversion of the pkgsrc-wip CVS project
Makefile
23
star
76

conch-api

Datacenter build and management service
Perl
22
star
77

node-tab

Unix-style tables for command-line utilities
Makefile
22
star
78

triton-go

Go SDK for Joyent Triton (Compute) and Triton Object Storage (Manta)
Go
21
star
79

node-spawn-async

spawn child processes asynchronously
JavaScript
19
star
80

smartmachine_cookbooks

Chef Cookbooks for managing SmartOS SmartMachines
19
star
81

syslinux

replica of syslinux repo from git://git.kernel.org/pub/scm/boot/syslinux/syslinux.git
C
19
star
82

manta-nfs

NFSv3 Manta Storage Server Gateway
JavaScript
19
star
83

daggr

filter and aggregate numeric data in plaintext or json form
JavaScript
18
star
84

mod_usdt

DTrace provider for Apache
D
18
star
85

freebsd-vpc

Control plane for `projects/VPC` branch of `joyent/freebsd`
Go
18
star
86

mibe

Machine Image Build Environment
PHP
17
star
87

node-zfs

Node.js library to interface with ZFS utilities
JavaScript
17
star
88

ruby-manta

Ruby interface for Joyent's Manta service
Ruby
17
star
89

pgstatsmon

Node.js service for shoveling Postgres stats into Prometheus
JavaScript
17
star
90

tsg-infrastructure

Shell
17
star
91

java-manta

Java Manta Client SDK
Java
16
star
92

manta-thoth

Thoth is a Manta-based system for core and crash dump management
JavaScript
16
star
93

node-nfs

Node.js SDK for writing Portmap/Mount/NFS (v3) servers
JavaScript
16
star
94

triton-terraform

16
star
95

java-http-signature

Library for performing RSA signed HTTP requests in Java
Java
16
star
96

summit-workshop

Node.js Summit - Day Zero Workshop
JavaScript
16
star
97

sdc-adminui

Operator portal for SmartDataCenter
JavaScript
15
star
98

sdc-headnode

Responsible for building and setting up the Triton (formerly SmartDataCenter) headnode.
JavaScript
15
star
99

openbsd-kvm-image-builder

Scripts to create a custom OpenBSD install ISO and a KVM image for use in SmartOS and Triton.
Shell
15
star
100

ipxe

C
14
star