• Stars
    star
    103
  • Rank 325,389 (Top 7 %)
  • Language
    JavaScript
  • License
    Other
  • Created over 11 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

📃 Streaming CSV Parser for Node. Small and made entirely out of streams.

csv-streamify Build Status Greenkeeper badge

NPM

Parses csv files. Accepts options. No coffee script, no weird APIs. Just streams. Tested against csv-spectrum and used in production. It is also "fast enough" (around 60,000 rows per second, but that varies with data obviously).

Works in node 4, 6, 8 and 9. Might work in earlier versions, but is not tested in it.

Installation

npm install csv-streamify

Usage

This module implements a simple node stream.Transform stream. You can write to it, read from it and use .pipe as you would expect.

const csv = require('csv-streamify')
const fs = require('fs')

const parser = csv()

// emits each line as a buffer or as a string representing an array of fields
parser.on('data', function (line) {
  console.log(line)
})

// now pipe some data into it
fs.createReadStream('/path/to/file.csv').pipe(parser)

with options and callback

The first argument can either be an options object (see below) or a callback function.

Note: If you pass a callback to csv-streamify it will buffer the parsed data for you and pass it to the callback when it's done. This behaviour can obviously lead to out of memory errors with very large csv files.

const csv = require('csv-streamify')
const fs = require('fs')

const parser = csv({ objectMode: true }, function (err, result) {
  if (err) throw err
  // our csv has been parsed succesfully
  result.forEach(function (line) { console.log(line) })
})

// now pipe some data into it
fs.createReadStream('/path/to/file.csv').pipe(parser)

Options

You can pass some options to the parser. All of them are optional.

The options are also passed to the underlying transform stream, so you can pass in any standard node core stream options.

{
  delimiter: ',', // comma, semicolon, whatever
  newline: '\n', // newline character (use \r\n for CRLF files)
  quote: '"', // what's considered a quote
  empty: '', // empty fields are replaced by this,

  // if true, emit arrays instead of stringified arrays or buffers
  objectMode: false,

  // if set to true, uses first row as keys -> [ { column1: value1, column2: value2 }, ...]
  columns: false
}

Also, take a look at iconv-lite (npm install iconv-lite --save), it provides pure javascript streaming character encoding conversion.

CLI

To use on the command line install it globally:

$ npm install csv-streamify -g

This should add the csv-streamify command to your $PATH.

Then, you either pipe data into it or give it a filename:

# pipe data in
$ cat some_data.csv | csv-streamify
# pass a filename
$ csv-streamify some_data.csv > output.json
# tell csv-streamify to read from + wait on stdin
$ csv-streamify -

Wishlist

  • browser support
  • better CLI

If you would like to contribute either of those just open an issue so we can discuss it further. :)

Contributors

Nicolas Hery (objectMode)

More Repositories

1

docker-couchdb

🐳 Source of the official Apache CouchDB Docker image ⚠️ NOTICE ⚠️: moved to the CouchDB org
Dockerfile
154
star
2

s3-website

☁️ Easily publish static websites to Amazon S3.
JavaScript
104
star
3

react-es6

React's official tutorial with ES6 and Browserify
JavaScript
78
star
4

docker-couchdb-ssl

🚨 Deprecated 🚨
Nginx
19
star
5

nova-standardjs

standardJS extension for Nova.app
JavaScript
14
star
6

popover

Small, composable popover component.
JavaScript
13
star
7

docker-couchdb-lucene

Dockerized Full Text Search for CouchDB
Shell
12
star
8

couch-proxy-auth

Creates CouchDB Proxy Authentication Headers
JavaScript
10
star
9

postgres-triggers

Setup changes-feed-like trigger functions in PostgreSQL
JavaScript
8
star
10

couch-surfer

🏄 Rebuild those views like a pro
JavaScript
8
star
11

postgres-watcher

👓 Watch your PostgreSQL tables for changes
JavaScript
7
star
12

cloudfront-tls

🔐 Sets up a cloudfront distribution with custom TLS certificates.
JavaScript
7
star
13

lazy-fb

Load the Facebook SDK lazily
JavaScript
7
star
14

snowball-german

JS implementation of the German Snowball Stemmer by Martin Porter
JavaScript
5
star
15

backbone-pouch-collection

Backbone Collections with PouchDB as the data source
JavaScript
3
star
16

node-container

Put a node app into a docker container and run it.
JavaScript
3
star
17

deep-set

Sets a value of a property in an object tree in Node and the browser.
JavaScript
3
star
18

dotfiles

🔧 .files, including osxdefaults — sensible hacker defaults for OS X
Shell
2
star
19

docker-wrk

Wrk (HTTP benchmarking tool) Dockerfile
Shell
1
star
20

slubnav

🚀 Komplexpraktikum InfoVis - Virtueller Rundgang
CSS
1
star
21

cloudflare-functions-status-bug

Bug repro
TypeScript
1
star
22

web-boilerplate

Opinionated boilerplate for simple frontend projects
CSS
1
star
23

tuio-server

JavaScript server implementation of the TUIO library
JavaScript
1
star