bundle 📦

Bundle is an embeddable compression library that supports 23 compression algorithms and 2 archive formats.

Distributed in two files.

Features

• Archive support: .bun, .zip
• Stream support: DEFLATE, LZMA, LZIP, ZPAQ, LZ4, ZSTD, BROTLI, BSC, CSC, BCM, MCM, ZMOLLY, ZLING, TANGELO, SHRINKER, CRUSH, LZJB, BZIP2 and SHOCO
• Optimized for highest compression ratios on each compressor, where possible.
• Optimized for fastest decompression times on each decompressor, where possible.
• Configurable, redistributable, self-contained, amalgamated and cross-platform (C++03).
• Optional benchmark infrastructure (C++11).
• ZLIB/LibPNG licensed.

Bundle stream format

[0x00  ...]          Optional zero padding (N bits)
[0x70 0x??]          Header (8 bits). De/compression algorithm (8 bits)
                     enum { RAW, SHOCO, LZ4F, MINIZ, LZIP, LZMA20, ZPAQ, LZ4,      //  0..7
                            BROTLI9, ZSTD, LZMA25, BSC, BROTLI11, SHRINKER, CSC20, //  7..14
                            ZSTDF, BCM, ZLING, MCM, TANGELO, ZMOLLY, CRUSH, LZJB,  // 15..22
                            BZIP2                                                  // 23..
                     };
[vle_unpacked_size]  Unpacked size of the stream (N bytes). Data is stored in a variable
                     length encoding value, where bytes are just shifted and added into a
                     big accumulator until MSB is found.
[vle_packed_size]    Packed size of the stream (N bytes). Data is stored in a variable
                     length encoding value, where bytes are just shifted and added into a
                     big accumulator until MSB is found.
[bitstream]          Compressed bitstream (N bytes). As returned by compressor.
                     If possible, header-less bitstreams are preferred.

Bundle .bun archive format

- Files/datas are packed into streams by using any compression method (see above)
- Streams are archived into a standard ZIP file:
  - ZIP entry compression is (0) for packed streams and (1-9) for unpacked streams.
  - ZIP entry comment is a serialized JSON of (file) meta-datas (@todo).
- Note: you can mix streams of different algorithms into the very same ZIP archive.

Showcase

#include <cassert>
#include "bundle.h"

int main() {
    using namespace bundle;
    using namespace std;

    // 23 mb dataset
    string original( "There's a lady who's sure all that glitters is gold" );
    for (int i = 0; i < 18; ++i) original += original + string( i + 1, 32 + i );

    // pack, unpack & verify all encoders
    vector<unsigned> libs { 
        RAW, SHOCO, LZ4F, MINIZ, LZIP, LZMA20,
        ZPAQ, LZ4, BROTLI9, ZSTD, LZMA25,
        BSC, BROTLI11, SHRINKER, CSC20, BCM,
        ZLING, MCM, TANGELO, ZMOLLY, CRUSH, LZJB
    };
    for( auto &lib : libs ) {
        string packed = pack(lib, original);
        string unpacked = unpack(packed);
        cout << original.size() << " -> " << packed.size() << " bytes (" << name_of(lib) << ")" << endl;
        assert( original == unpacked );
    }

    cout << "All ok." << endl;
}

On choosing compressors (on a regular basis)

• Note: SHOCO is a text compressor intended to be used for plain ascii IDs only.

Charts

@mavam has an awesome R script that plots some fancy graphics in his compbench repository. The following CC images are a few of his own showcasing an invocation for a 10,000 packet PCAP trace:

API - data

namespace bundle
{
  // low level API (raw pointers)
  bool is_packed( *ptr, len );
  bool is_unpacked( *ptr, len );
  unsigned type_of( *ptr, len );
  size_t len( *ptr, len );
  size_t zlen( *ptr, len );
  const void *zptr( *ptr, len );
  bool pack( unsigned Q, *in, len, *out, &zlen );
  bool unpack( unsigned Q, *in, len, *out, &zlen );

  // medium level API, templates (in-place)
  bool is_packed( T );
  bool is_unpacked( T );
  unsigned type_of( T );
  size_t len( T );
  size_t zlen( T );
  const void *zptr( T );
  bool unpack( T &, T );
  bool pack( unsigned Q, T &, T );

  // high level API, templates (copy)
  T pack( unsigned Q, T );
  T unpack( T );
}

For a complete review check bundle.hpp header

API - archives

namespace bundle
{
  struct file : map<string,string> { // ~map of properties
    bool    has(property);           // property check
    string &get(property);           // property access
    string  toc() const;             // inspection (json)
  };
  struct archive : vector<file>    { // ~sequence of files
    void   bun(string);              // .bun serialization
    string bun() const;              // .bun serialization
    void   zip(string);              // .zip serialization
    string zip() const;              // .zip serialization
    string toc() const;              // inspection (json)
  };
}

Build Directives (Licensing)

(*): will disable .bun and .zip archive support as well.

Build Directives (Libraries)

(*): will disable .bun and .zip archive support as well.

Build Directives (Other)

Licensing table

Evaluated alternatives

FastLZ, FLZP, LibLZF, LZFX, LZHAM, LZLIB, LZO, LZP, SMAZ, Snappy, ZLIB, bzip2, Yappy, CMix, M1

Creating DLLs

cl bundle.cpp -DBUNDLE_API=BUNDLE_API_EXPORT /LD
cl demo.cc -DBUNDLE_API=BUNDLE_API_IMPORT bundle.lib

Changelog

• v2.1.0 (2017/06/10): C API and DLL bindings; Pump up libcsc; Rename .bnd to .bun
• v2.0.5 (2016/02/06): Bring back BZIP2 support
• v2.0.4 (2015/12/04): Add padding support; Fix reentrant CRUSH; Optimizations & fixes
• v2.0.3 (2015/12/02): Add LZJB and CRUSH; Add BUNDLE_NO_CDDL directive
• v2.0.2 (2015/11/07): Fix ZMolly segmentation fault (OSX)
• v2.0.1 (2015/11/04): Fix clang warnings and compilation errors
• v2.0.0 (2015/11/03): Add BCM, ZLING, MCM, Tangelo, ZMolly, ZSTDf support
• v2.0.0 (2015/11/03): Change archive format (break change)
• v2.0.0 (2015/11/03): Disambiguate .bnd/.zip archive handling
• v2.0.0 (2015/11/03): Fix compilation errors (C++03)
• v2.0.0 (2015/11/03): Fix missing entries in benchmarks
• v2.0.0 (2015/11/03): Improve runtime C++ de/initialization stability
• v2.0.0 (2015/11/03): Optimize archive decompression
• v2.0.0 (2015/11/03): Remove obsolete enums
• v2.0.0 (2015/11/03): Upgrade Brotli, ZPAQ, LZ4, ZSTD and Shoco
• v1.0.2 (2015/10/29): Skip extra copy during archive decompression
• v1.0.2 (2015/10/29): Add extra archive meta-info
• v1.0.1 (2015/10/10): Shrink to fit during measures() function
• v1.0.0 (2015/10/09): Change benchmark API to sort multiples values as well (API break change)
• v0.9.8 (2015/10/07): Remove confusing bundle::string variant class from API
• v0.9.7 (2015/10/07): Add license configuration directives
• v0.9.6 (2015/10/03): Add library configuration directives
• v0.9.5 (2015/09/28): Add missing prototypes
• v0.9.5 (2015/09/28): Bugfix helper function
• v0.9.4 (2015/09/26): Add CSC20 + Shrinker support
• v0.9.4 (2015/09/26): Rename enums LZ4->LZ4F/LZ4HC->LZ4
• v0.9.3 (2015/09/25): Add a few missing API calls
• v0.9.2 (2015/09/22): Pump up Brotli
• v0.9.2 (2015/09/22): Split BROTLI enum into BROTLI9/11 pair
• v0.9.1 (2015/05/10): Switch to ZLIB/LibPNG license
• v0.9.0 (2015/04/08): BSC support
• v0.8.1 (2015/04/07): Pump up Brotli+ZSTD
• v0.8.1 (2015/04/07): LZMA20/25 dict
• v0.8.1 (2015/04/07): Unify FOURCCs
• v0.8.0 (2015/01/27): ZSTD support
• v0.8.0 (2015/01/27): Reorder enums
• v0.8.0 (2015/01/27): Simplify API
• v0.7.1 (2015/01/26): Fix LZMA
• v0.7.1 (2015/01/26): Verify DEFLATEs
• v0.7.1 (2015/01/26): New AUTO enum
• v0.7.0 (2014/10/22): Brotli support
• v0.7.0 (2014/10/22): Pump up LZ4
• v0.6.3 (2014/09/27): Switch to BOOST license
• v0.6.2 (2014/09/02): Fix 0-byte streams
• v0.6.2 (2014/09/02): Deflate alignment
• v0.6.1 (2014/06/30): Safer LZ4 decompression
• v0.6.1 (2014/06/30): Pump up LZ4 + ZPAQ
• v0.6.0 (2014/06/26): LZ4HC support
• v0.6.0 (2014/06/26): Optimize in-place decompression
• v0.5.0 (2014/06/09): ZPAQ support
• v0.5.0 (2014/06/09): UBER encoding
• v0.5.0 (2014/06/09): Fixes
• v0.4.1 (2014/06/05): Switch to lzmasdk
• v0.4.0 (2014/05/30): Maximize compression (lzma)
• v0.3.0 (2014/05/28): Fix alignment (deflate)
• v0.3.0 (2014/05/28): Change stream header
• v0.2.1 (2014/05/23): Fix overflow bug
• v0.2.0 (2014/05/14): Add VLE header
• v0.2.0 (2014/05/14): Fix vs201x compilation errors
• v0.1.0 (2014/05/13): Add high-level API
• v0.1.0 (2014/05/13): Add iOS support
• v0.0.0 (2014/05/09): Initial commit