• Stars
    star
    1,084
  • Rank 42,698 (Top 0.9 %)
  • Language
    JavaScript
  • License
    BSD 2-Clause "Sim...
  • Created over 9 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A collection of useful stream utility modules for writing better code using streams

mississippi

a collection of useful stream utility modules. learn how the modules work using this and then pick the ones you want and use them individually

the goal of the modules included in mississippi is to make working with streams easy without sacrificing speed, error handling or composability.

usage

var miss = require('mississippi')

methods

pipe

miss.pipe(stream1, stream2, stream3, ..., cb)

Pipes streams together and destroys all of them if one of them closes. Calls cb with (error) if there was an error in any of the streams.

When using standard source.pipe(destination) the source will not be destroyed if the destination emits close or error. You are also not able to provide a callback to tell when the pipe has finished.

miss.pipe does these two things for you, ensuring you handle stream errors 100% of the time (unhandled errors are probably the most common bug in most node streams code)

original module

miss.pipe is provided by require('pump')

example

// lets do a simple file copy
var fs = require('fs')

var read = fs.createReadStream('./original.zip')
var write = fs.createWriteStream('./copy.zip')

// use miss.pipe instead of read.pipe(write)
miss.pipe(read, write, function (err) {
  if (err) return console.error('Copy error!', err)
  console.log('Copied successfully')
})

each

miss.each(stream, each, [done])

Iterate the data in stream one chunk at a time. Your each function will be called with (data, next) where data is a data chunk and next is a callback. Call next when you are ready to consume the next chunk.

Optionally you can call next with an error to destroy the stream. You can also pass the optional third argument, done, which is a function that will be called with (err) when the stream ends. The err argument will be populated with an error if the stream emitted an error.

original module

miss.each is provided by require('stream-each')

example

var fs = require('fs')
var split = require('binary-split') // require('split2') would work here as well

var newLineSeparatedNumbers = fs.createReadStream('numbers.txt')

var pipeline = miss.pipeline(newLineSeparatedNumbers, split())
miss.each(pipeline, eachLine, done)
var sum = 0

function eachLine (line, next) {
  sum += parseInt(line.toString())
  next()
}

function done (err) {
  if (err) throw err
  console.log('sum is', sum)
}

pipeline

var pipeline = miss.pipeline(stream1, stream2, stream3, ...)

Builds a pipeline from all the transform streams passed in as arguments by piping them together and returning a single stream object that lets you write to the first stream and read from the last stream.

If you are pumping object streams together use pipeline = miss.pipeline.obj(s1, s2, ...).

If any of the streams in the pipeline emits an error or gets destroyed, or you destroy the stream it returns, all of the streams will be destroyed and cleaned up for you.

original module

miss.pipeline is provided by require('pumpify')

example

// first create some transform streams (note: these two modules are fictional)
var imageResize = require('image-resizer-stream')({width: 400})
var pngOptimizer = require('png-optimizer-stream')({quality: 60})

// instead of doing a.pipe(b), use pipeline
var resizeAndOptimize = miss.pipeline(imageResize, pngOptimizer)
// `resizeAndOptimize` is a transform stream. when you write to it, it writes
// to `imageResize`. when you read from it, it reads from `pngOptimizer`.
// it handles piping all the streams together for you

// use it like any other transform stream
var fs = require('fs')

var read = fs.createReadStream('./image.png')
var write = fs.createWriteStream('./resized-and-optimized.png')

miss.pipe(read, resizeAndOptimize, write, function (err) {
  if (err) return console.error('Image processing error!', err)
  console.log('Image processed successfully')
})

duplex

var duplex = miss.duplex([writable, readable, opts])

Take two separate streams, a writable and a readable, and turn them into a single duplex (readable and writable) stream.

The returned stream will emit data from the readable. When you write to it it writes to the writable.

You can either choose to supply the writable and the readable at the time you create the stream, or you can do it later using the .setWritable and .setReadable methods and data written to the stream in the meantime will be buffered for you.

original module

miss.duplex is provided by require('duplexify')

example

// lets spawn a process and take its stdout and stdin and combine them into 1 stream
var child = require('child_process')

// @- tells it to read from stdin, --data-binary sets 'raw' binary mode
var curl = child.spawn('curl -X POST --data-binary @- http://foo.com')

// duplexCurl will write to stdin and read from stdout
var duplexCurl = miss.duplex(curl.stdin, curl.stdout)

through

var transformer = miss.through([options, transformFunction, flushFunction])

Make a custom transform stream.

The options object is passed to the internal transform stream and can be used to create an objectMode stream (or use the shortcut miss.through.obj([...]))

The transformFunction is called when data is available for the writable side and has the signature (chunk, encoding, cb). Within the function, add data to the readable side any number of times with this.push(data). Call cb() to indicate processing of the chunk is complete. Or to easily emit a single error or chunk, call cb(err, chunk)

The flushFunction, with signature (cb), is called just before the stream is complete and should be used to wrap up stream processing.

original module

miss.through is provided by require('through2')

example

var fs = require('fs')

var read = fs.createReadStream('./boring_lowercase.txt')
var write = fs.createWriteStream('./AWESOMECASE.TXT')

// Leaving out the options object
var uppercaser = miss.through(
  function (chunk, enc, cb) {
    cb(null, chunk.toString().toUpperCase())
  },
  function (cb) {
    cb(null, 'ONE LAST BIT OF UPPERCASE')
  }
)

miss.pipe(read, uppercaser, write, function (err) {
  if (err) return console.error('Trouble uppercasing!')
  console.log('Splendid uppercasing!')
})

from

miss.from([opts], read)

Make a custom readable stream.

opts contains the options to pass on to the ReadableStream constructor e.g. for creating a readable object stream (or use the shortcut miss.from.obj([...])).

Returns a readable stream that calls read(size, next) when data is requested from the stream.

  • size is the recommended amount of data (in bytes) to retrieve.
  • next(err, chunk) should be called when you're ready to emit more data.

original module

miss.from is provided by require('from2')

example

function fromString(string) {
  return miss.from(function(size, next) {
    // if there's no more content
    // left in the string, close the stream.
    if (string.length <= 0) return next(null, null)

    // Pull in a new chunk of text,
    // removing it from the string.
    var chunk = string.slice(0, size)
    string = string.slice(size)

    // Emit "chunk" from the stream.
    next(null, chunk)
  })
}

// pipe "hello world" out
// to stdout.
fromString('hello world').pipe(process.stdout)

to

miss.to([options], write, [flush])

Make a custom writable stream.

opts contains the options to pass on to the WritableStream constructor e.g. for creating a writable object stream (or use the shortcut miss.to.obj([...])).

Returns a writable stream that calls write(data, enc, cb) when data is written to the stream.

  • data is the received data to write the destination.
  • enc encoding of the piece of data received.
  • cb(err, data) should be called when you're ready to write more data, or encountered an error.

flush(cb) is called before finish is emitted and allows for cleanup steps to occur.

original module

miss.to is provided by require('flush-write-stream')

example

var ws = miss.to(write, flush)

ws.on('finish', function () {
  console.log('finished')
})

ws.write('hello')
ws.write('world')
ws.end()

function write (data, enc, cb) {
  // i am your normal ._write method
  console.log('writing', data.toString())
  cb()
}

function flush (cb) {
  // i am called before finish is emitted
  setTimeout(cb, 1000) // wait 1 sec
}

If you run the above it will produce the following output

writing hello
writing world
(nothing happens for 1 sec)
finished

concat

var concat = miss.concat(cb)

Returns a writable stream that concatenates all data written to the stream and calls a callback with the single result.

Calling miss.concat(cb) returns a writable stream. cb is called when the writable stream is finished, e.g. when all data is done being written to it. cb is called with a single argument, (data), which will contain the result of concatenating all the data written to the stream.

Note that miss.concat will not handle stream errors for you. To handle errors, use miss.pipe or handle the error event manually.

original module

miss.concat is provided by require('concat-stream')

example

var fs = require('fs')

var readStream = fs.createReadStream('cat.png')
var concatStream = miss.concat(gotPicture)

function callback (err) {
  if (err) {
    console.error(err)
    process.exit(1)
  }
}

miss.pipe(readStream, concatStream, callback)

function gotPicture(imageBuffer) {
  // imageBuffer is all of `cat.png` as a node.js Buffer
}

function handleError(err) {
  // handle your error appropriately here, e.g.:
  console.error(err) // print the error to STDERR
  process.exit(1) // exit program with non-zero exit code
}

finished

miss.finished(stream, cb)

Waits for stream to finish or error and then calls cb with (err). cb will only be called once. err will be null if the stream finished without error, or else it will be populated with the error from the streams error event.

This function is useful for simplifying stream handling code as it lets you handle success or error conditions in a single code path. It's used internally miss.pipe.

original module

miss.finished is provided by require('end-of-stream')

example

var copySource = fs.createReadStream('./movie.mp4')
var copyDest = fs.createWriteStream('./movie-copy.mp4')

copySource.pipe(copyDest)

miss.finished(copyDest, function(err) {
  if (err) return console.log('write failed', err)
  console.log('write success')
})

parallel

miss.parallel(concurrency, each)

This works like through except you can process items in parallel, while still preserving the original input order.

This is handy if you wanna take advantage of node's async I/O and process streams of items in batches. With this module you can build your very own streaming parallel job queue.

Note that miss.parallel preserves input ordering. Passing the option {ordered:false} will output the data as soon as it's processed by a transform, without waiting to respect the order (this previously required a separate module through2-concurrent).

original module

miss.parallel is provided by require('parallel-transform')

example

This example fetches the GET HTTP headers for a stream of input URLs 5 at a time in parallel.

function getResponse (item, cb) {
  var r = request(item.url)
  r.on('error', function (err) {
    cb(err)
  })
  r.on('response', function (re) {
    cb(null, {url: item.url, date: new Date(), status: re.statusCode, headers: re.headers})
    r.abort()
  })
}

miss.pipe(
  fs.createReadStream('./urls.txt'), // one url per line
  split(),
  miss.parallel(5, getResponse),
  miss.through(function (row, enc, next) {
    console.log(JSON.stringify(row))
    next()
  })
)

see also

license

Licensed under the BSD 2-clause license.

More Repositories

1

art-of-node

❄️ a short introduction to node.js
JavaScript
9,640
star
2

menubar

➖ high level way to create menubar desktop applications with electron
TypeScript
6,673
star
3

screencat

🐈 webrtc screensharing electron app for mac os (Alpha)
CSS
3,014
star
4

cool-ascii-faces

ᕙ༼ຈل͜ຈ༽ᕗ
JavaScript
1,753
star
5

yo-yo

A tiny library for building modular UI components using DOM diffing and ES6 tagged template literals
JavaScript
1,326
star
6

voxel-engine

3D HTML5 voxel game engine
JavaScript
1,269
star
7

monu

menubar process monitor mac app [ALPHA]
CSS
1,111
star
8

callback-hell

information about async javascript programming
JavaScript
815
star
9

javascript-for-cats

an introduction to the javascript programming language. intended audience: cats
JavaScript
775
star
10

websocket-stream

websockets with the node stream API
JavaScript
665
star
11

torrent

download torrents with node from the CLI
JavaScript
637
star
12

concat-stream

writable stream that concatenates strings or data and calls a callback with the result
JavaScript
570
star
13

hexbin

community curated list of hexagon logos
JavaScript
526
star
14

linux

run Linux on Yosemite easily from the CLI
JavaScript
457
star
15

geojson-js-utils

JavaScript helper functions for manipulating GeoJSON
JavaScript
403
star
16

requirebin

write browser JavaScript programs using modules from NPM
JavaScript
391
star
17

extract-zip

Zip extraction written in pure JavaScript. Extracts a zip into a directory.
JavaScript
391
star
18

maintenance-modules

a list of modules that are useful for maintaining or developing modules
348
star
19

tabby

a browser with almost no UI
JavaScript
345
star
20

ndjson

streaming line delimited json parser + serializer
JavaScript
294
star
21

abstract-blob-store

A test suite and interface you can use to implement streaming file (blob) storage modules for various storage backends and platforms
JavaScript
266
star
22

standard-format

converts your code into Standard JavaScript Format
JavaScript
265
star
23

wzrd

Super minimal browserify development server
JavaScript
248
star
24

elementary-electron

NodeSchool workshop for learning Electron
JavaScript
228
star
25

gh-pages-template

free hosting on github! fork this to get a repo with only a gh-pages branch that is easy to edit
CSS
221
star
26

taco

a modular deployment system for unix
215
star
27

toiletdb

flushes an object to a JSON file. lets you do simple CRUD with async safely with the backend being a flat JSON file
JavaScript
215
star
28

bytewiser

a nodeschool workshop that teaches you the fundamentals of working with binary data in node.js and HTML5 browsers
HTML
208
star
29

csv-write-stream

A CSV encoder stream that produces properly escaped CSVs
JavaScript
204
star
30

electron-spawn

easy way to run code inside of a headless electron window from the CLI
JavaScript
198
star
31

voxel

tools to work with voxel generation and meshing in javascript
JavaScript
186
star
32

monocles

[NOT MAINTAINED] diaspora... as a couchapp! in pure javascript and fully OStatus compliant (almost)
JavaScript
180
star
33

simplify-geojson

apply the ramer-douglas-peucker line simplification to geojson features or feature collections in JS or on the CLI
JavaScript
170
star
34

nugget

minimalist wget clone written in node. HTTP GET files and downloads them into the current directory
JavaScript
162
star
35

electron-microscope

use electron-microscope to inspect websites and extract data
JavaScript
157
star
36

domnode

node style streams for HTML5 APIs
JavaScript
154
star
37

voxel-builder

build stuff with blocks in the browser, export for papercraft or 3d printing
CSS
149
star
38

multiplex

A binary stream multiplexer
JavaScript
141
star
39

gifify-docker

docker container for the gifify utility
135
star
40

csv-spectrum

A variety of CSV files to serve as an acid test for CSV parsing libraries
JavaScript
134
star
41

async-team

Documentation about how to run an async team (e.g. a remote team in different places)
132
star
42

node-repl

run a node program but also attach a repl to the same context that your code runs in so you can inspect + mess with stuff as your program is running. node 0.12/iojs and above only
JavaScript
129
star
43

datacouch

[ON HIATUS] distributed, collaborative dataset sharing
JavaScript
122
star
44

HyperOS

A 50MB linux distribution that has dat-container for booting live containers on mac OS
Shell
119
star
45

couchpubtato

use Node.js to make CouchDB eat feeds like potato chips
JavaScript
116
star
46

voxel-mesh

generate a three.js mesh from voxel data
JavaScript
109
star
47

browser-locale

normalizes weird cross browser issues and tries to return the users selected language in 100% client side JS by looking at various properties on the `window.navigator` object
JavaScript
106
star
48

binary-csv

A fast, streaming CSV binary parser written in javascript
JavaScript
105
star
49

cats

BSD licensed cat photos that I've taken
103
star
50

filereader-stream

Read an HTML5 File object (from e.g. HTML5 drag and drops) as a stream.
JavaScript
102
star
51

nets

nothing but nets. http client that works in node and browsers
JavaScript
101
star
52

javascript-editor

codemirror + esprima powered html5 javascript editor component
JavaScript
99
star
53

adventure-time

a web based environment for doing nodeschool adventures
JavaScript
97
star
54

workerstream

use HTML5 web workers with the node stream API
JavaScript
92
star
55

tree-view

tree viewer UI widget made with react
JavaScript
91
star
56

packify

packs up browserify apps by inlining all assets into one html file
JavaScript
89
star
57

ViewKit

UI library designed for WebKit/Mobile Safari/Android WebViews
JavaScript
87
star
58

gut

hosted open data filet knives
JavaScript
86
star
59

conversationThreading-js

javascript port of JWZ email conversation threading
JavaScript
83
star
60

refine-python

Python client library for controlling Google Refine
Python
83
star
61

require-times

find out how long require calls take in your program. this is a debugging tool for figuring out why apps load slowly
JavaScript
77
star
62

binary-split

a fast newline (or any delimiter) splitter stream - like require('split') but specific for binary data
JavaScript
77
star
63

commonjs-html-prettyprinter

easy HTML pretty printing in commonJS
JavaScript
76
star
64

voxel-server

multiplayer server for voxel-engine
JavaScript
74
star
65

nginx-vhosts

Programmatically add or remove vhosts to a running Nginx instance
JavaScript
71
star
66

github-oauth

simple node.js functions for doing oauth login with github
JavaScript
71
star
67

get-dat

A command line tutorial to learn dat
HTML
69
star
68

docker-stream

CLI tool for automating the use of docker containers in streaming data processing pipelines. Works on Windows, Mac and Linux.
JavaScript
68
star
69

ogmail

minimalist gmail cli client
JavaScript
63
star
70

dhtkv

CLI for storing arbitrary key/value data in the bittorrent mainline DHT
JavaScript
63
star
71

voxel-hello-world

a template voxel game repo you can use to build your own voxel games
JavaScript
62
star
72

browser-module-sandbox

browser editor for code that gets 'compiled' on the server with node and run on the client
JavaScript
61
star
73

minecraft-skin

load minecraft skins as meshes in three.js applications
JavaScript
58
star
74

atomic-queue

a crash friendly queue that persists queue state and can restart. uses a worker pool and has configurable concurrency
JavaScript
57
star
75

PDXAPI

JSON API for CivicApps.org datasets for Portland, OR
JavaScript
53
star
76

googleauth

Create and load persistent Google authentication tokens for command-line apps
JavaScript
52
star
77

joinopenwifi

automatically join open and internet connect wireless networks on linux
JavaScript
51
star
78

superlevel

a minimalist cli utility for leveldb databases
JavaScript
51
star
79

ble-stream

experimental duplex stream api over bluetooth low energy connections (BLE)
JavaScript
50
star
80

json-merge

given two streams of newline delimited JSON data perform a merge/extend on each object in the stream
JavaScript
49
star
81

haraka-couchdb

a real time email server using nodejs, haraka and couchdb
JavaScript
49
star
82

multirepo

a power tool for batch processing multiple github repositories
JavaScript
49
star
83

mount-url

mount a http file as if it was a local file using fuse
JavaScript
48
star
84

dat-editor

web app console/dashboard/spreadsheet thingy for dat
CSS
47
star
85

csv2html

CSV to HTML command line utility
JavaScript
47
star
86

collaborator

easily add new collaborators to your github repos from the CLI
JavaScript
46
star
87

subcommand

Create CLI tools with subcommands. A minimalist CLI router
JavaScript
44
star
88

blockplot

[alpha] explore minecraft worlds in your browser
JavaScript
44
star
89

refine-ruby

Ruby client library for controlling Google Refine
Ruby
43
star
90

element-class

exactly like .addClass and .removeClass from jquery but without dependencies
JavaScript
42
star
91

biofabric

a client side module for generating biofabric graphs in svg using d3
JavaScript
42
star
92

dat-core

low level implementation of the dat data version graph
JavaScript
42
star
93

kawaii

kawaii face detection
JavaScript
41
star
94

doorknob

convenience module for adding Mozilla Persona user login + LevelDB based session storage to node web apps
JavaScript
38
star
95

stl-obj-viewer

super simple viewer for .stl or .obj files powered by three.js
JavaScript
38
star
96

ftpfs

an ftp client that expose the node fs API
JavaScript
38
star
97

xml-json

convert xml to json on the command line. not streaming, pure javascript
JavaScript
37
star
98

ndarray-stl

convert voxels into 3D printable .stl files
JavaScript
37
star
99

masseuse.js

a (now deprecated) library for fast taps on mobile browsers
JavaScript
37
star
100

current-location

Get your current location (latitude, longitude) on the command line as JSON
JavaScript
37
star