• Stars
    star
    209
  • Rank 181,327 (Top 4 %)
  • Language
    Erlang
  • Created over 12 years ago
  • Updated 7 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

MessagePack (de)serializer implementation for Erlang / msgpack.org[Erlang]

MessagePack Erlang

hex.pm version

Prerequisites for runtime

Erlang/OTP, >= 22.0 Also based on the new msgpack spec 0b8f5a.

edit rebar.config to use in your application

{deps, [
  {msgpack, ".*",
    {git, "git://github.com/msgpack/msgpack-erlang.git", {branch, "master"}}}
]}.

Or as it is now published at hex.pm, just

{deps, [msgpack]}.

might work.

Simple deserialization

Ham = msgpack:pack(Spam),
{ok, Spam} = msgpack:unpack(Ham).

Stream deserialization

{Term0, Rest0} = msgpack:unpack_stream(Binary),
{Term1, Rest1} = msgpack:unpack_stream(Rest0),
...

Options, for packing and unpacking

{spec, new|old}

Both for packing and unpacking. Default is new. Major difference between old and new spec is:

  • raw family (0xa0~0xbf, 0xda, 0xdb) becomes new str family
  • 0xd9 is new as str8
  • new bin space (0xc4, 0xc5, 0xc6 as bin8, bin16, bin32)
  • new ext space (0xc7, 0xc8, 0xc9 as ext8, ext16, ext32)
  • new fixext space (0xd4, 0xd5, 0xd6, 0xd7, 0xd8 as fixext1, fixext2, fixext4, fixext8, fixext16),

The default is new spec. Old spec mode does not handle these new types but returns error. To use old spec mode, this option is explicitly added.

OldHam = msgpack:pack(Spam, [{spec, old}]),
{ok, Spam} = msgpack:unpack(OldHam, [{spec, old}]).

{allow_atom, none|pack}

Only in packing. Atoms are packed as binaries. Default value is pack. Otherwise, any term including atoms throws badarg.

{known_atoms, [atom()]}

Both in packing and unpacking. In packing, if an atom is in this list a binary is encoded as a binary. In unpacking, msgpacked binaries are decoded as atoms with erlang:binary_to_existing_atom/2 with encoding utf8. Default value is an empty list.

Even if allow_atom is none, known atoms are packed.

{unpack_str, as_binary|as_list}

A switch to choose decoded term style of str type when unpacking. Only available at new spec. Default is as_list.

mode        as_binary    as_list
-----------+------------+-------
bin         binary()     binary()
str         binary()     string()

{use_nil, boolean()}

Handles Elixir nil as null Erlang atom. Default value is false.

{validate_string, boolean()}

Only in unpacking, UTF-8 validation at unpacking from str type will be enabled. Default value is false.

{pack_str, from_binary|from_list|none}

A switch to choose packing of string() when packing. Only available at new spec. Default is from_list for symmetry with unpack_str option.

mode        from_list    from_binary    none
-----------+------------+--------------+-----------------
binary()    bin          str*/bin       bin
string()    str*/array   array of int   array of int
list()      array        array          array

But the default option pays the cost of performance for symmetry. If the overhead of UTF-8 validation is unacceptable, choosing none as the option would be the best.

  • * Tries to pack as str if it is a valid string().

{map_format, map|jiffy|jsx}

Both at packing and unpacking. Default value is map.

msgpack:pack(#{ <<"key">> => <<"value">> }, []).
msgpack:pack(#{ <<"key">> => <<"value">> }, [{map_format, map}]).
msgpack:pack({[{<<"key">>, <<"value">>}]}, [{map_format, jiffy}]),
msgpack:pack([{<<"key">>, <<"value">>}], [{map_format, jsx}]).

{ext, {msgpack_ext_packer(), msgpack_ext_unpacker()}|module()}

At both. The default behaviour in case of facing ext data at decoding is to ignore them as its length is known.

Now msgpack-erlang supports ext type. Now you can serialize everything with your original (de)serializer. That will enable us to handle erlang- native types like pid(), ref() contained in tuple(). See test/msgpack_ext_example_tests.erl for example code.

Packer = fun({ref, Ref}, Opt) when is_reference(Ref) -> {ok, {12, term_to_binary(Ref)}} end,
Unpacker = fun(12, Bin) -> {ok, {ref, binary_to_term(Bin)}} end,
Ref = make_ref(),
Opt = [{ext,{Packer,Unpacker}}],
{ok, {ref, Ref}} = msgpack:unpack(msgpack:pack({ref, Ref}, Opt), Opt).

Misc

Float type

The Float type of Message Pack represents IEEE 754 floating point number, so it includes Nan and Infinity. In unpacking, msgpack-erlang returns nan, positive_infinity and negative_infinity.

License

Apache License 2.0

Release Notes

0.7.0

  • Support nan, positive_infinity and negative_infinity

0.6.0

  • Support OTP 19.0

0.5.0

  • Renewed optional arguments to pack/unpack interface. This is incompatible change from 0.4 series.

0.4.0

  • Deprecate nil
  • Moved to rebar3
  • Promote default map unpacker as default format when OTP is >= 17
  • Added QuickCheck tests
  • Since this version OTP older than R16B03-1 are no more supported

0.3.5 / 0.3.4

  • 0.3 series will be the last versions that supports R16B or older versions of OTP.
  • OTP 18.0 support
  • Promote default map unpacker as default format when OTP is >= 18

0.3.3

  • Add OTP 17 series to Travis-CI tests
  • Fix wrong numbering for ext types
  • Allow packing maps even when {format,map} is not set
  • Fix Dialyzer invalid contract warning
  • Proper use of null for jiffy-style encoding/decoding

0.3.2

  • set back default style as jiffy
  • fix bugs around nil/null handling

0.3.0

  • supports map new in 17.0
  • jiffy-style maps will be deprecated in near future
  • set default style as map

0.2.8

0.2 series works with OTP 17.0, R16, R15, and with MessagePack's new and old format. But does not support map type introduced in OTP 17.0.

It also supports JSX-compatible mode.

More Repositories

1

msgpack

MessagePack is an extremely efficient object serialization library. It's like JSON, but very fast and small.
6,744
star
2

msgpack-c

MessagePack implementation for C and C++ / msgpack.org[C/C++]
2,871
star
3

msgpack-python

MessagePack serializer implementation for Python msgpack.org[Python]
Python
1,817
star
4

msgpack-java

MessagePack serializer implementation for Java / msgpack.org[Java]
Java
1,371
star
5

msgpack-javascript

@msgpack/msgpack - MessagePack for JavaScript / msgpack.org[JavaScript/TypeScript/ECMA-262]
TypeScript
1,197
star
6

msgpack-cli

MessagePack implementation for Common Language Infrastructure / msgpack.org[C#]
C#
817
star
7

msgpack-php

msgpack.org[PHP]
PHP
773
star
8

msgpack-ruby

MessagePack implementation for Ruby / msgpack.org[Ruby]
Ruby
751
star
9

msgpack-node

MessagePack implementation for Node.js
JavaScript
309
star
10

msgpack-objectivec

MessagePack serializer implementation for Objective-C / msgpack.org[Objective-C]
C
298
star
11

msgpack-go

Go
149
star
12

msgpack-haskell

Haskell implementation of MessagePack / msgpack.org[Haskell]
Haskell
135
star
13

msgpack-d

MessagePack for D / msgpack.org[D]
D
116
star
14

msgpack-scala

MessagePack serializer implementation for Scala / msgpack.org[Scala]
Shell
94
star
15

website

http://msgpack.org/
HTML
55
star
16

msgpack-perl

MessagePack serializer implementation for Perl / msgpack.org[Perl]
C
50
star
17

msgpack-ocaml

MessagePack implementation for OCaml / msgpack.org[OCaml]
OCaml
45
star
18

msgpack-hadoop

MessagePack-Hadoop integration provides an efficient schema-free data representation for Hadoop and Hive.
Java
34
star
19

msgpack-smalltalk

MessagePack serialization library for various Smalltalk dialects / msgpack.org[Smalltalk]
Smalltalk
22
star
20

pickling-msgpack

Msgpack support for Scala pickling object serialization framework.
Scala
7
star
21

custom-types

See this thread: https://github.com/msgpack/msgpack/issues/121
1
star