• Stars
    star
    759
  • Rank 59,846 (Top 2 %)
  • Language
    Ruby
  • License
    Apache License 2.0
  • Created about 13 years ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

MessagePack implementation for Ruby / msgpack.org[Ruby]

MessagePack

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON but it's faster and smaller. For example, small integers (like flags or error code) are encoded into a single byte, and typical short strings only require an extra byte in addition to the strings themselves.

If you ever wished to use JSON for convenience (storing an image with metadata) but could not for technical reasons (binary data, size, speed...), MessagePack is a perfect replacement.

require 'msgpack'
msg = [1,2,3].to_msgpack  #=> "\x93\x01\x02\x03"
MessagePack.unpack(msg)   #=> [1,2,3]

Use RubyGems to install:

gem install msgpack

or build msgpack-ruby and install:

bundle
rake
gem install --local pkg/msgpack

Use cases

  • Create REST API returing MessagePack using Rails + RABL
  • Store objects efficiently serialized by msgpack on memcached or Redis
  • Upload data in efficient format from mobile devices such as smartphones
    • MessagePack works on iPhone/iPad and Android. See also Objective-C and Java implementations
  • Design a portable protocol to communicate with embedded devices
    • Check also Fluentd which is a log collector which uses msgpack for the log format (they say it uses JSON but actually it's msgpack, which is compatible with JSON)
  • Exchange objects between software components written in different languages
    • You'll need a flexible but efficient format so that components exchange objects while keeping compatibility

Portability

MessagePack for Ruby should run on x86, ARM, PowerPC, SPARC and other CPU architectures.

And it works with MRI (CRuby) and Rubinius. Patches to improve portability are highly welcomed.

Serializing objects

Use MessagePack.pack or to_msgpack:

require 'msgpack'
msg = MessagePack.pack(obj)  # or
msg = obj.to_msgpack
File.binwrite('mydata.msgpack', msg)

Streaming serialization

Packer provides advanced API to serialize objects in streaming style:

# serialize a 2-element array [e1, e2]
pk = MessagePack::Packer.new(io)
pk.write_array_header(2).write(e1).write(e2).flush

See API reference for details.

Deserializing objects

Use MessagePack.unpack:

require 'msgpack'
msg = File.binread('mydata.msgpack')
obj = MessagePack.unpack(msg)

Streaming deserialization

Unpacker provides advanced API to deserialize objects in streaming style:

# deserialize objects from an IO
u = MessagePack::Unpacker.new(io)
u.each do |obj|
  # ...
end

or event-driven style which works well with EventMachine:

# event-driven deserialization
def on_read(data)
  @u ||= MessagePack::Unpacker.new
  @u.feed_each(data) {|obj|
     # ...
  }
end

See API reference for details.

Serializing and deserializing symbols

By default, symbols are serialized as strings:

packed = :symbol.to_msgpack     # => "\xA6symbol"
MessagePack.unpack(packed)      # => "symbol"

This can be customized by registering an extension type for them:

MessagePack::DefaultFactory.register_type(0x00, Symbol)

# symbols now survive round trips
packed = :symbol.to_msgpack     # => "\xc7\x06\x00symbol"
MessagePack.unpack(packed)      # => :symbol

The extension type for symbols is configurable like any other extension type. For example, to customize how symbols are packed you can just redefine Symbol#to_msgpack_ext. Doing this gives you an option to prevent symbols from being serialized altogether by throwing an exception:

class Symbol
    def to_msgpack_ext
        raise "Serialization of symbols prohibited"
    end
end

MessagePack::DefaultFactory.register_type(0x00, Symbol)

[1, :symbol, 'string'].to_msgpack  # => RuntimeError: Serialization of symbols prohibited

Serializing and deserializing Time instances

There are the timestamp extension type in MessagePack, but it is not registered by default.

To map Ruby's Time to MessagePack's timestamp for the default factory:

MessagePack::DefaultFactory.register_type(
  MessagePack::Timestamp::TYPE, # or just -1
  Time,
  packer: MessagePack::Time::Packer,
  unpacker: MessagePack::Time::Unpacker
)

See API reference for details.

Extension Types

Packer and Unpacker support Extension types of MessagePack.

# register how to serialize custom class at first
pk = MessagePack::Packer.new(io)
pk.register_type(0x01, MyClass1, :to_msgpack_ext) # equal to pk.register_type(0x01, MyClass)
pk.register_type(0x02, MyClass2){|obj| obj.how_to_serialize() } # blocks also available

# almost same API for unpacker
uk = MessagePack::Unpacker.new()
uk.register_type(0x01, MyClass1, :from_msgpack_ext)
uk.register_type(0x02){|data| MyClass2.create_from_serialized_data(data) }

MessagePack::Factory is to create packer and unpacker which have same extension types.

factory = MessagePack::Factory.new
factory.register_type(0x01, MyClass1) # same with next line
factory.register_type(0x01, MyClass1, packer: :to_msgpack_ext, unpacker: :from_msgpack_ext)
pk = factory.packer(options_for_packer)
uk = factory.unpacker(options_for_unpacker)

For MessagePack.pack and MessagePack.unpack, default packer/unpacker refer MessagePack::DefaultFactory. Call MessagePack::DefaultFactory.register_type to enable types process globally.

MessagePack::DefaultFactory.register_type(0x03, MyClass3)
MessagePack.unpack(data_with_ext_typeid_03) #=> MyClass3 instance

Alternatively, extension types can call the packer or unpacker recursively to generate the extension data:

Point = Struct.new(:x, :y)
factory = MessagePack::Factory.new
factory.register_type(
  0x01,
  Point,
  packer: ->(point, packer) {
    packer.write(point.x)
    packer.write(point.y)
  },
  unpacker: ->(unpacker) {
    x = unpacker.read
    y = unpacker.read
    Point.new(x, y)
  },
  recursive: true,
)
factory.load(factory.dump(Point.new(12, 34))) # => #<struct Point x=12, y=34>

Pooling

Creating Packer and Unpacker objects is expensive. For best performance it is preferable to re-use these objects.

MessagePack::Factory#pool makes that easier:

factory = MessagePack::Factory.new
factory.register_type(
  0x01,
  Point,
  packer: ->(point, packer) {
    packer.write(point.x)
    packer.write(point.y)
  },
  unpacker: ->(unpacker) {
    x = unpacker.read
    y = unpacker.read
    Point.new(x, y)
  },
  recursive: true,
)
pool = factory.pool(5) # The pool size should match the number of threads expected to use the factory concurrently.

pool.load(pool.dump(Point.new(12, 34))) # => #<struct Point x=12, y=34>

Buffer API

MessagePack for Ruby provides a buffer API so that you can read or write data by hand, not via Packer or Unpacker API.

This MessagePack::Buffer is backed with a fixed-length shared memory pool which is very fast for small data (<= 4KB), and has zero-copy capability which significantly affects performance to handle large binary data.

How to build and run tests

Before building msgpack, you need to install bundler and dependencies.

gem install bundler
bundle install

Then, you can run the tasks as follows:

Build

bundle exec rake build

Run tests

bundle exec rake spec

Generating docs

bundle exec rake doc

How to build -java rubygems

To build -java gems for JRuby, run:

rake build:java

If this directory has Gemfile.lock (generated with MRI), remove it beforehand.

Updating documents

Online documents (http://ruby.msgpack.org) is generated from gh-pages branch. Following commands update documents in gh-pages branch:

bundle exec rake doc
git checkout gh-pages
cp doc/* ./ -a

Copyright

  • Author
  • Copyright
    • Copyright (c) 2008-2015 Sadayuki Furuhashi
  • License
    • Apache License, Version 2.0

More Repositories

1

msgpack

MessagePack is an extremely efficient object serialization library. It's like JSON, but very fast and small.
6,905
star
2

msgpack-c

MessagePack implementation for C and C++ / msgpack.org[C/C++]
2,991
star
3

msgpack-python

MessagePack serializer implementation for Python msgpack.org[Python]
Python
1,883
star
4

msgpack-java

MessagePack serializer implementation for Java / msgpack.org[Java]
Java
1,403
star
5

msgpack-javascript

@msgpack/msgpack - MessagePack for JavaScript / msgpack.org[JavaScript/TypeScript/ECMA-262]
TypeScript
1,256
star
6

msgpack-cli

MessagePack implementation for Common Language Infrastructure / msgpack.org[C#]
C#
828
star
7

msgpack-php

msgpack.org[PHP]
PHP
774
star
8

msgpack-node

MessagePack implementation for Node.js
JavaScript
311
star
9

msgpack-objectivec

MessagePack serializer implementation for Objective-C / msgpack.org[Objective-C]
C
300
star
10

msgpack-erlang

MessagePack (de)serializer implementation for Erlang / msgpack.org[Erlang]
Erlang
210
star
11

msgpack-go

Go
149
star
12

msgpack-haskell

Haskell implementation of MessagePack / msgpack.org[Haskell]
Haskell
136
star
13

msgpack-d

MessagePack for D / msgpack.org[D]
D
116
star
14

msgpack-scala

MessagePack serializer implementation for Scala / msgpack.org[Scala]
Shell
94
star
15

website

http://msgpack.org/
HTML
57
star
16

msgpack-perl

MessagePack serializer implementation for Perl / msgpack.org[Perl]
C
51
star
17

msgpack-ocaml

MessagePack implementation for OCaml / msgpack.org[OCaml]
OCaml
45
star
18

msgpack-hadoop

MessagePack-Hadoop integration provides an efficient schema-free data representation for Hadoop and Hive.
Java
34
star
19

msgpack-smalltalk

MessagePack serialization library for various Smalltalk dialects / msgpack.org[Smalltalk]
Smalltalk
22
star
20

pickling-msgpack

Msgpack support for Scala pickling object serialization framework.
Scala
7
star
21

custom-types

See this thread: https://github.com/msgpack/msgpack/issues/121
1
star