• Stars
    star
    311
  • Rank 134,521 (Top 3 %)
  • Language
    JavaScript
  • License
    Other
  • Created almost 12 years ago
  • Updated about 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

MessagePack implementation for Node.js

node-msgpack is an addon for NodeJS that provides an API for serializing and de-serializing JavaScript objects using the MessagePack library. The performance of this addon compared to the native JSON object isn't too bad, and the space required for serialized data is far less than JSON.

Performance

node-msgpack is currently slower than the built-in JSON.stringify() and JSON.parse() methods. In recent versions of node.js, the JSON functions have been heavily optimized. node-msgpack is still more compact, and we are currently working performance improvements. Testing shows that, over 500k iterations, msgpack.pack() is about 5x slower than JSON.stringify(), and msgpack.unpack() is about 3.5x slower than JSON.parse().

Old performance numbers are below.

The following tests were performed with 500,000 instances of the JavaScript object {'abcdef' : 1, 'qqq' : 13, '19' : [1, 2, 3, 4]}:

  • JSON.stringify() 7.17 seconds
  • JSON.parse(JSON.stringify()) 22.18 seconds
  • msgpack.pack() 5.80 seconds
  • msgpack.unpack(msgpack.pack()) 8.62 seconds

Note that node-msgpack produces and consumes Buffer objects, and a such does not incur encoding/decoding overhead when performing I/O with native strings.

Usage

This module provides two methods: pack(), which consumes a JavaScript object and produces a node Buffer object; and unpack(), which consumes a node Buffer object and produces a JavaScript object. Packing of all native JavaScript types (undefined, boolean, numbers, strings, arrays and objects) is supported, as is the node Buffer type.

The below code snippet packs and then unpacks a JavaScript object, verifying the resulting object at the end using assert.deepEqual().

    var assert = require('assert');
    var msgpack = require('msgpack');

    var o = {"a" : 1, "b" : 2, "c" : [1, 2, 3]};
    var b = msgpack.pack(o);
    var oo = msgpack.unpack(b);

    assert.deepEqual(oo, o);

As a convenience, a higher level streaming API is provided in the msgpack.Stream class, which can be constructed around a net.Stream instance. This object emits msg events when an object has been received.

    var msgpack = require('msgpack');

    // ... get a net.Stream instance, s, from somewhere
    
    var ms = new msgpack.Stream(s);
    ms.addListener('msg', function(m) {
        sys.debug('received message: ' + sys.inspect(m));
    });

Type Mapping

The JavaScript type system does not map cleanly on to the MsgPack type system, though it's pretty close.

When packing, JavaScript values are mapped to MsgPack types as follows

  • undefined and null values map to MSGPACK_OBJECT_NIL
  • boolean values map to MSGPACK_OBJECT_BOOLEAN
  • number values map differently depending on their value
    • Floating point values map to MSGPACK_OBJECT_DOUBLE
    • Positive values map to MSGPACK_OBJECT_POSITIVE_INTEGER
    • Negative values map to MSGPACK_OBJECT_NEGATIVE_INTEGER
  • string values map to MSGPACK_OBJECT_RAW; all strings are serialized with UTF-8 encoding
  • Array values (as defined by Array.isArray()) map to MSGPACK_OBJECT_ARRAY; each element in the array is packed individually the rules in this list
  • NodeJS Buffer values map to MSGPACK_OBJECT_RAW
  • Everything else maps to MSGPACK_OBJECT_MAP, where we iterate over the object's properties and pack them and their values as per the mappings in this list

When unpacking, MsgPack types are mapped to JavaScript values as follows

  • MSGPACK_OBJECT_NIL values map to the null value
  • MSGPACK_OBJECT_BOOLEAN values map to boolean values
  • MSGPACK_OBJECT_POSITIVE_INTEGER, MSGPACK_OBJECT_NEGATIVE_INTEGER and MSGPACK_OBJECT_DOUBLE values map to number values
  • MSGPACK_OBJECT_ARRAY values map to arrays; each object in the array is packed individually using the rules in this list
  • MSGPACK_OBJECT_RAW values are mapped to string values; these values are unpacked using either UTF-8 or ASCII encoding, depending on the contents of the raw buffer
  • MSGPACK_OBJECT_MAP values are mapped to JavaScript objects; keys and values are unpacked individually using the rules in this list

Strings are particularly problematic here, as it's difficult to get hints down into the packing and unpacking codepaths about how to interpret a particular string or MSGPACK_OBJECT_RAW. If you have strict requirements about the encoding of your strings, it's recommended that you populate a Buffer object yourself (e.g. using Buffer.write()) and pack that buffer rather than the string. This will ensure that you can control what gets packed.

When unpacking, things are trickier as there is no way to know the encoding used when a string was packed. There is an an open ticket for the MsgPack format to address this.

Command Line Utilities

As a convenience and for debugging, bin/json2msgpack and bin/msgpack2json are provided to convert JSON data to and from MessagePack data, reading from stdin and writing to stdout.

% echo '[1, 2, 3]' | ./bin/json2msgpack | xxd -
0000000: 9301 0203                                ....
% echo '[1, 2, 3]' | ./bin/json2msgpack | ./bin/msgpack2json 
[1,2,3]

Building, Installation, Testing

There are two ways to install msgpack.

NPM

	npm install msgpack

This should build and install msgpack for you. Then just require('msgpack').

Manually

You will need node-gyp:

npm install -g node-gyp

Then from the root of the msgpack repo, you can run:

node-gyp rebuild
NOTE:
node-gyp attempts to contact the Internet and download the target version of node.js source and store it locally. This will only happen once for each time it sees a new node.js version. If you're on a host with no direct Internet access, you may need to shuffle this source over from another box or sneaker net. Good luck!

Testing

To run all tests use:

./run_tests

To run a specific test:

./run_tests test/lib/msgpack.js
NOTE:
Tests are based on a modified version of nodeunit (https://github.com/godsflaw/nodeunit). Follow ./run_tests instructions if you run into problems.

Benchmarks

To run benchmarks:

./run_tests test/benchmark/benchmark.js

More Repositories

1

msgpack

MessagePack is an extremely efficient object serialization library. It's like JSON, but very fast and small.
6,905
star
2

msgpack-c

MessagePack implementation for C and C++ / msgpack.org[C/C++]
2,991
star
3

msgpack-python

MessagePack serializer implementation for Python msgpack.org[Python]
Python
1,883
star
4

msgpack-java

MessagePack serializer implementation for Java / msgpack.org[Java]
Java
1,403
star
5

msgpack-javascript

@msgpack/msgpack - MessagePack for JavaScript / msgpack.org[JavaScript/TypeScript/ECMA-262]
TypeScript
1,256
star
6

msgpack-cli

MessagePack implementation for Common Language Infrastructure / msgpack.org[C#]
C#
828
star
7

msgpack-php

msgpack.org[PHP]
PHP
774
star
8

msgpack-ruby

MessagePack implementation for Ruby / msgpack.org[Ruby]
Ruby
759
star
9

msgpack-objectivec

MessagePack serializer implementation for Objective-C / msgpack.org[Objective-C]
C
300
star
10

msgpack-erlang

MessagePack (de)serializer implementation for Erlang / msgpack.org[Erlang]
Erlang
210
star
11

msgpack-go

Go
149
star
12

msgpack-haskell

Haskell implementation of MessagePack / msgpack.org[Haskell]
Haskell
136
star
13

msgpack-d

MessagePack for D / msgpack.org[D]
D
116
star
14

msgpack-scala

MessagePack serializer implementation for Scala / msgpack.org[Scala]
Shell
94
star
15

website

http://msgpack.org/
HTML
57
star
16

msgpack-perl

MessagePack serializer implementation for Perl / msgpack.org[Perl]
C
51
star
17

msgpack-ocaml

MessagePack implementation for OCaml / msgpack.org[OCaml]
OCaml
45
star
18

msgpack-hadoop

MessagePack-Hadoop integration provides an efficient schema-free data representation for Hadoop and Hive.
Java
34
star
19

msgpack-smalltalk

MessagePack serialization library for various Smalltalk dialects / msgpack.org[Smalltalk]
Smalltalk
22
star
20

pickling-msgpack

Msgpack support for Scala pickling object serialization framework.
Scala
7
star
21

custom-types

See this thread: https://github.com/msgpack/msgpack/issues/121
1
star