• This repository has been archived on 06/Jan/2022
  • Stars
    star
    191
  • Rank 202,877 (Top 4 %)
  • Language
    JavaScript
  • Created over 10 years ago
  • Updated almost 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Build cross platform data pipelines [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]

deprecated

More info on active projects and modules at dat-ecosystem.org


gasket

Preconfigured pipelines for node.js

logo

$ npm install -g gasket
$ gasket # prints help
$ gasket completion --save # install tab completion

Usage

To setup a pipeline add a gasket section to your package.json

{
  "name": "my-test-app",
  "dependencies" : {
    "transform-uppercase": "^1.0.0"
  },
  "gasket": {
    "example": [
      {
        "command": "echo hello world",
        "type": "pipe"
      },
      {
        "command": "transform-uppercase",
        "type": "pipe"
      }
    ]
  }
}

To run the above example pipeline simply to the repo and run

$ gasket run example # will print HELLO WORLD

gasket will spawn each command in the pipeline (it supports modules/commands installed via npm) and pipe them together (if the type is set to "pipe").

If you want to wait for the previous command to finish, set the type to "run" instead.

{
  "gasket": {
    "example": [
      {
        "command": "echo hello world",
        "type": "run"
      },
      {
        "command": "echo hello afterwards",
        "type": "run"
      }
    ]
  }
}

Running the above will print

hello world
hello afterwards

Modules in pipelines

In addition to commands it supports node modules that return streams

{
  "gasket": [
    {
      "command": "echo hello world",
      "type": "pipe"
    }
    {
      "command": {"module":"./uppercase.js"},
      "type": "pipe"
    }
  ]
}

Where uppercase.js is a file that looks like this

var through = require('through2')
module.exports = function() {
  return through(function(data, enc, cb) {
    cb(null, data.toString().toUpperCase())
  })
}

If your module reads/writes JSON object set json:true in the pipeline. That will make gasket parse newline separated JSON before parsing the objects to the stream and stringify the output.

Running gasket run main will produce HELLO WORLD

Using gasket.json

If you don't have a package.json file you can add the tasks to a gasket.json file instead

{
  "example": [
    {
      "command": "echo hello world",
      "type": "pipe"
    },
    {
      "command": "transform-uppercase",
      "type": "pipe"
    }
  ]
}

gasket as a module

You can use gasket as a module as well

var gasket = require('gasket')

var pipelines = gasket({
  example: [
    {
      "command": "echo hello world",
      "type": "pipe"
    },
    {
      "command": "transform-uppercase",
      "type": "pipe"
    }
  ]
})

pipelines.run('example').pipe(process.stdout)

More Repositories

1

dat-desktop

Peer to peer data syncronization [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
648
star
2

docs

Documentation resources for dat and the surrounding ecosystem [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
530
star
3

dat-node

Node module for creating dat compatible tools on file systems [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
505
star
4

discovery-channel

Search for a key across multiple discovery networks and find peers who answer. [ DEPRECATED - see https://github.com/hyperswarm/dht for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
299
star
5

datBase

Open data sharing powered by Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
248
star
6

awesome-dat

Community curated resources for Dat Project [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
225
star
7

how-dat-works

Dat documentation [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
HTML
169
star
8

DEPs

Dat Enhancement Proposals. Contains all specs for the Dat protocol, including drafts. [ DEPRECATED - see https://github.com/hypercore-protocol/hypercore-proposals for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
165
star
9

rabin

node native addon for rabin fingerprinting data streams [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
C++
146
star
10

dat-js

A pure JavaScript browser-friendly api for using dat [ DEPRECATED - see https://github.com/datproject/sdk for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
142
star
11

hyperdiscovery

[ DEPRECATED - see https://github.com/hyperswarm/replicator for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
135
star
12

hypercloud

A hosting server for Dat. [ DEPRECATED - see github.com/beakerbrowser/hashbase for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
97
star
13

hyperdrive-http

HTTP request handler for Hyperdrive and Hypercore [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
68
star
14

book

Documentation on how to implement the Dat Protocol [ DEPRECATED - see https://github.com/hypercore-protocol/new-website/tree/master/guides for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
Shell
68
star
15

datproject-discussions

a repo for discussions and other non-code organizing stuff [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
65
star
16

svalbard

A global metadata vault [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
62
star
17

whitepaper

2017 Dat Whitepaper - Added for Archival Purposes (see DEPs for latest spec) [ DEPRECATED - see https://github.com/dat-ecosystem-archive/DEPs for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
TeX
61
star
18

datasilo

Build a home server that adds high speed storage and upload capacity to a global volunteer science network [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
59
star
19

dat-dns

Issue DNS lookups for Dat archives using HTTPS requests to the target host. [ DEPRECATED - see https://github.com/martinheidegger/hyper-dns for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
57
star
20

node-dat-archive

A nodejs API for Dat which is compatible with Beaker's DatArchive API [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
50
star
21

dat.foundation

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
47
star
22

design

Design Resources and Assets [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
SCSS
43
star
23

datscript

A compile-to-JSON data pipeline scripting language [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
43
star
24

dat-storage

Dat specific storage provider for Hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
35
star
25

organization

Dat Project Organization Information (Grants, Accounting, Etc.) [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
28
star
26

dat.json

The WIP specification for the dat.json meta format [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
27
star
27

dat-swarm-defaults

Dat Defaults for Discovery Swarm [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
27
star
28

project-updates

updates from the various projects working on or with dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
27
star
29

dat-http

a http transport/storage provider for dats [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
26
star
30

planning

Dat planning documents [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
25
star
31

old-website

Dat Project Website [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
23
star
32

web-specs

Specifications for the Dat protocol in Web browsers [ DEPRECATED - see https://github.com/dat-ecosystem-archive/DEPs for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
22
star
33

multidat

Manage multiple dat instances [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
22
star
34

dat-ls

Small program that lists all the changes in a dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
22
star
35

hyperdrive-import-files

Import some files and folders into a hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
21
star
36

dat-secret-storage

hyperdrive storage module for dat secret keys [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
21
star
37

dat-link-resolve

Dat link resolver [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
19
star
38

dat-encoding

Dat's way of encoding and decoding dat links [ DEPRECATED - see https://github.com/mafintosh/abstract-encoding and https://github.com/compact-encoding for similar functionality. More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
19
star
39

hyperdrive-network-speed

track upload and download speed on a hyperdrive archive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
18
star
40

dat.land

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
17
star
41

multidrive

Manage multiple hyperdrive instances [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
13
star
42

dat-doctor

πŸ₯ extension for dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
13
star
43

projects

Dat Project Projects [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
13
star
44

dat-deep-link

[DEPRECATED] An experiment to define an URI scheme that implements 'hypercore-strong-link'
JavaScript
12
star
45

dat-log

Log History of Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
12
star
46

dat-foundation-website

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
Vue
11
star
47

datprotocol.org

Dat Protocol Website [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
HTML
10
star
48

dat-registry-api

Creates an api and database for tracking users and dats. [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
10
star
49

dat-json

manage dat.json file via toiletdb [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
10
star
50

dat-ignore

Check if you should ignore a file before adding to Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
9
star
51

anacapa-container

A containerized way to run the Anacapa eDNA processing toolkit on your own machine or server [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
Shell
9
star
52

getdat

Bulk data and/or metadata downloader for large lists of HTTP and FTP urls [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
9
star
53

dat-shaka-player-plugin

[DEPRECATED] A plugin for 'shaka-player' to load 'dat://' URI
JavaScript
8
star
54

dat-data.com

OLD website for the dat project [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
8
star
55

hyperdrive-staging-area

Staging area for local, uncommited writes that can sync to a hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
8
star
56

dat-registry-client

Node.js Client for datBase [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
6
star
57

dat-elements

Dat UI elements [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
5
star
58

hyperdrive-to-zip-stream

Create a zipfile from a hyperdrive [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
5
star
59

analytics-server

fair analytics server w/ docker deployment [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
4
star
60

dat.land-roadmap

[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
3
star
61

integration-tests

A mininet test suite for Dat implementations.[ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
Sass
3
star
62

Code-of-Conduct

Code of Conduct Discussions [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
3
star
63

discovery-deployment

WIP Deploy your own peer discovery infrastructure [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
2
star
64

datproject.org

JavaScript
1
star
65

desktop-website

πŸ–₯️ Dat Desktop project site [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star
66

dat-glob

Glob implementation for dat archives [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star
67

dat-walk

Recursive directory walker for dat archives [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star
68

dat-rm

Delete files from dat archives using glob patterns [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
JavaScript
1
star