• Stars
    star
    146
  • Rank 246,897 (Top 5 %)
  • Language
    JavaScript
  • Created over 9 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Pipeline pattern implementation with the support for sync and async stages

pipeline-js

Build Status Package Version

Hassle free pipeline pattern implementation with the support for sync and async stages

Introduction

Pipeline JS allows you to implement the pipeline pattern while creating reusable pipelines in your Javascript applications. You can create pipelines consisting of one or more stages once and then process them using different payloads. Pipeline processing is initiated by some payload and this payload will be passed from stage to stage in order to complete the required process.

General Pipeline Example

To demonstrate it using an example, consider a request made to access user by id; the pipeline may consist of stages including getUserById, transformUser, convertToJson and return. And for each next stage, the input comes from the last stage i.e.

->(User ID)>>getUserById()
    ->(User Detail)>>transformUser()
          ->(Transformed Detail)>>convertToJson()
                  ->(User Detail.json)>>return

Sample Programmatic implementation

While using Pipeline JS, it can be written programmatically as

// First syntax
var userPipeline = new Pipeline([
    getUserById,
    transformUser,
    convertToJson
]);

// Alternatively, you may write the same using the pipe method
var userPipeline = (new Pipeline()).pipe(getUserById)
                                   .pipe(transformUser)
                                   .pipe(convertToJson);

// Then this pipeline can be used with any payload i.e.
var userJson = userPipeline.process(10);  // JSON detail for the user with ID 10
var userJson = userPipeline.process(23);  // JSON detail for the user with ID 23

Where stages shown above can be anything invokable. The sample implementation for the stages may be something like below

// For example, methods from some objects
var getUserById   = UserModel.getUserById,
    transformUser = Transformers.transformUser,
    convertToJson = Utility.convertToJson;

// Or functions
var getUserById = function (userId) {
    //..
    return promise;
};

var transformUser = function (userDetail) {
    // ..
    return transformedObject;
};

var convertToJson = function(userDetail) {
    // ..
    return jsonString;
};

Using pipeline will not only allow constructing reusable pipelines but also result in comparatively cleaner and readable code i.e. consider the following example

  var output = JSON.stringify(transformFilters(getSelectedFilters()));
  
  // It can be transformed to
  var pipeline = new Pipeline([
    getSelectedFilters,
    transformFilters,
    JSON.stringify
  ]);
  
  // Or maybe use the alternate syntax
  var pipeline = (new Pipeline()).pipe(getSelectedFilters)
                                 .pipe(transformFilters)
                                 .pipe(JSON.stringify);

  // Get the output by processing the pipeline 
  var output = pipeline.process();

Installation

Run the below command to install using NPM

npm install --save pipeline-js

Usage

Operations in a pipeline i.e. stages can be anything that is callable i.e. closures and anything that's invokable is good.

In order to create a pipeline, you can either pass the stages as an array parameter to constructor

var pipeline = require('pipeline-js');

// Creating pipeline using constructor parameterå
var pipeline = new Pipeline([
  invokableStage1,
  invokableStage2,
  invokableStage3
]);

// Process pipeline with payload1
var output1 = pipeline.process(payload1);
var output2 = pipeline.process(payload2);

Or you can create a pipeline object and call the pipe method on it to successively add the invokable stages

var pipeline = require('pipeline-js');

// Creating pipeline using pipe method
var pipeline = (new Pipeline()).pipe(invokableStage1)
                               .pipe(invokableStage2)
                               .pipe(invokableStage3

// Process pipeline with payload1
var output1 = pipeline.process(payload1);
var output2 = pipeline.process(payload2);

Sync/Async Usage

The only difference between the synchronous usage and asynchronous usage is how the output is handled. For both types of usages, pipelines are created the same way. The difference is when you call the process() method, if the pipeline has all the stages returning concrete output the process method returns concrete value, however if any of the stages returns a promise then the process method returns promise and you will have to use .then() to get the output.

Examples for both the sync and async usage are given below

Sync Example

How to use when all the stages return concrete values?

If none of the stages return promise then process(payload) will return concrete value

var addOne = function (x) {
  return x + 1;
};

var square = function (x) {
  return x * x;
};

var minusTwo = function (x) {
  return x - 2;
};

// Without pipeline
// - Not reusable
// - Not that clean
var output1 = minusTwo(square(addOne(10)));
var output2 = minusTwo(square(addOne(10)));

// With Pipeline
// Reusable with different payload
// Cleaner
var someFormula = new Pipeline([
  addOne,
  square,
  minusTwo
]);

var result = someFormula.process(10);   // 10 + 1  => 11
                                        // 11 * 11 => 121
                                        // 121 - 2 => 119
console.log(result);  // (int) 119

var result = someFormula.process(20);   // 20 + 1  => 21
                                        // 21 * 21 => 441
                                        // 441 - 2 => 339
console.log(result); // (int) 339

Or maybe you can write the same example as

var someFormula = (new Pipeline()).pipe(addOne)
                                  .pipe(square)
                                  .pipe(minusTwo);

var output1 = someFormula.process(20);
var output2 = someFormula.process(20);

Async Example

How to use when one or all of the stages return promise?

If any single of the stages returns a promise, process(payload) will return a promise

var Pipeline = require('pipeline-js');

// Gets the user by ID and returns promise
var getUserById = function (userId) {
  var q = q.defer();
  // ..
  return q.promise;
};

// Transforms the user
var transformUser = function (userDetail) {
  return {
    name: userDetail.name,
    email: userDetail.email,
    password: '*****'
  };
};

// Converts to JSON
var createJson = function (object) {
  return JSON.stringify(object);
};

var pipeline = new Pipeline([
    getUserById,    // Returns promise
    transformUser,
    createJson
]);

// process() will return promise; since one of the stages returns a promise
var output = pipeline.process(142)
                     .then(function(userJson){
                        console.log(userJson);    // (string) {"name": "John Doe", "email": "[email protected]", "password": "****"}
                     })
                     .catch(function(error) {
                        console.log(error);
                     });

var output = pipeline.process(263)  // promise will be returned
                     .then(function(userJson){
                        console.log(userJson);    // (string) {"name": "Jane Doe", "email": "[email protected]", "password": "****"}
                     })
                     .catch(function(error) {
                        console.log(error);
                     });

Altneratively,

// Same pipeline using `pipe` method
var pipeline = (new Pipeline()).pipe(getUserById)     // Returns promise
                               .pipe(transformUser)
                               .pipe(createJson);

var output = pipeline.process(142)  // promise will be returned; since one of the stages returns a promise
                     .then(function(userJson){
                        console.log(userJson);    // (string) {"name": "John Doe", "email": "[email protected]", "password": "****"}
                     })
                     .catch(function(error) {
                        console.log(error);
                     });

Sidenote

You may also want to check this pipeline proposal

Contribution

Feel free to fork, extend, create issues, create PRs or spread the word.

License

MIT © Kamran Ahmed

More Repositories

1

developer-roadmap

Interactive roadmaps, guides and other educational content to help developers grow in their careers.
TypeScript
273,916
star
2

design-patterns-for-humans

An ultra-simplified explanation to design patterns
43,344
star
3

driver.js

A light-weight, no-dependency, vanilla JavaScript engine to drive the user's focus across the page
TypeScript
20,517
star
4

git-standup

Recall what you did on the last working day. Psst! or be nosy and find what someone else in your team did ;-)
Shell
7,534
star
5

pennywise

Cross-platform application to open any website or media in a floating window
JavaScript
3,710
star
6

githunt

Hunt the most starred projects on any date on GitHub
JavaScript
2,751
star
7

roadmap.sh

Community driven roadmaps, articles and resources for developers
Nunjucks
2,196
star
8

jquery-toast-plugin

Highly customizable jquery plugin to show toast messages
JavaScript
1,488
star
9

brusher

Create beautiful webpage backgrounds
JavaScript
745
star
10

tab-switcher

Chrome Extension - Switch between the opened tabs in the blink of an eye
JavaScript
444
star
11

aws-cost-cli

CLI tool to perform cost analysis on your AWS account with Slack integration
TypeScript
386
star
12

itomate

Automate your iTerm layouts and session setup
Python
333
star
13

datastructures-in-javascript

Illustrated Data Structures — Video Series
JavaScript
161
star
14

kamranahmedse.github.io

Blog created using github-pages-blog-action
HTML
148
star
15

local-ses

Trap and test AWS SES emails locally
TypeScript
147
star
16

awesome-minimal-sites

An opinionated collection of minimal yet beautiful websites.
123
star
17

github-pages-blog-action

Create good looking blog from your markdown files in a GitHub repository
CSS
81
star
18

redux-persist-expire

Expiring transformer for redux-persist
JavaScript
78
star
19

css-tailor

✂ Automatically generate CSS from your HTML classes
JavaScript
66
star
20

express-api-problem

Express package to automatically turn your exceptions to the API Problem JSON response
TypeScript
65
star
21

stylos

Webpack plugin to automatically generate and inject CSS utilities to your application
JavaScript
65
star
22

git-first

Chrome Extension – Takes you to the first commit of a GitHub repository
JavaScript
64
star
23

laravel-faulty

Automatically turn your thrown exceptions to JSON response while conforming to API problem specification
PHP
63
star
24

smasher

Smash your directories to get JSON or Array representation and vice versa.
PHP
60
star
25

laravel-modular-boilerplate

A boilerplate to create modular application in laravel 5.1
PHP
59
star
26

copy-marker

Chrome extension — Create URLs with highlighted page selection
JavaScript
52
star
27

jumper-bot

A game developed using HTML-5 canvas and Javascript
JavaScript
50
star
28

beetle

Abuse unicode to incite mayhem
Shell
49
star
29

laravel-censor

A middleware for Laravel 5.* to easily redact or replace the words from the pages you want.
PHP
42
star
30

db-playground

Easily create a sandbox environment for your database exploration.
Shell
39
star
31

github-notable-comments

Chrome Extension – Navigate through the most reacted comments in github issues and PRs
JavaScript
39
star
32

gulp-css-tailor

✂ Gulp plugin to automatically generate CSS from your HTML classes
JavaScript
38
star
33

github-diffs

Easier code reviews by collapse/expand diffs in pull requests
JavaScript
35
star
34

laraformer

Laravel 5.* package to easily introduce a transformation layer for your data
PHP
32
star
35

makers.ae

Community of makers in UAE
TypeScript
30
star
36

brainza-fps-unity3d-game

First person shooter 3d game.
ASP
29
star
37

github-actions-youtube

Codebase for my youtube video on GitHub actions
HTML
24
star
38

php-shorthand

Calculate unique shorthands for a given set of strings
PHP
24
star
39

promises-examples

Codebase accompanying my YouTube video: All about Asynchronous JavaScript
21
star
40

node-basic-auth-example

Sample implementation of Basic Authentication in Node.js
JavaScript
20
star
41

tweet-counter

Chrome Extension - Bring back the character counters in tweet boxes
JavaScript
19
star
42

mongo-playground

Single command to setup MongoDB playground with different datasets
Shell
19
star
43

node-session-auth-example

Sample implementation of Session Authentication in Node.js
JavaScript
18
star
44

gcuf-news-caster

Google Chrome Extension that shows the latest news from GCUF Website
JavaScript
16
star
45

slack-msg

Dead simple CLI tool to send messages to Slack
JavaScript
16
star
46

mongodumper

Docker image to back up MongoDB database and upload to S3
Shell
16
star
47

mondex

CLI tool to create and manage MongoDB indexes using code
TypeScript
14
star
48

walkers

A console based fan fiction RPG for The Walking Dead TV Series
PHP
13
star
49

ng-atlas

An Angular JS application that lets you tour around the globe ..free of cost ;-)
JavaScript
12
star
50

markdown-it-class

Plugin for markdown-it to allow adding classes to HTML tags.
JavaScript
8
star
51

yaml-sample

JavaScript
8
star
52

egghead-mongo-aggregation

Codebase for my "Aggregation in MongoDB" videos on Egghead
JavaScript
7
star
53

notable

The Markdown-based note-taking app that doesn't suck.
7
star
54

sendy

Dockerized Sendy application with one-click deployment setup for Railway.
Shell
5
star
55

mulk

Gets the country details by country name, ITU or ISO codes
JavaScript
4
star
56

astro-view-transitions-bug

JavaScript
2
star