• Stars
    star
    194
  • Rank 200,186 (Top 4 %)
  • Language
    Clojure
  • Created almost 11 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Buffy The ByteBuffer Slayer, Clojure library for working with binary data.

Buffy, The Byte Buffer Slayer

Buffy is a Clojure library for working with binary data, writing complete binary protocol implementations in Clojure, storing complex data structures in an off-heap cache, reading binary files and doing everything you would usually do with ByteBuffer.

Main features

  • partial deserialization (read and deserialize parts of a byte buffer)
  • named access (access parts of your buffer by names)
  • composing/decomposing from key/value pairs
  • pretty hexdump
  • many useful default types that you can combine and extend easily

Project Maturity

Buffy is a young project. The API is fairly stable and the project has reached 1.0 in December 2014.

Installation

Artefacts

Latest artifacts are published to Clojars If you are using Maven, add the following repository definition to your pom.xml:

<repository>
  <id>clojars.org</id>
  <url>http://clojars.org/repo</url>
</repository>

The Most Recent Release

With Leiningen:

[clojurewerkz/buffy "1.1.0"]

With Maven:

<dependency>
  <groupId>clojurewerkz</groupId>
  <artifactId>buffy</artifactId>
  <version>1.1.0</version>
</dependency>

Usage

Require Buffy's main namespace:

(ns my.app
  (:require [clojurewerkz.buffy.core :refer :all]))

Buffy creates buffers from a spec you specify. The spec consists of one or more fields of known data types, for example:

(spec :my-field-1 (int32-type)
      :my-field-2 (string-type 10))

The spec can be a map (e.g. array-map) or a vector of vectors. Avoid using hash maps, since they are unordered.

Below is a specification for a buffer containing 2 fields, one 4 bytes long and second one 10:

0            4                         14
+------------+-------------------------+
| my-field-1 |         my-field-2      |
|    (int)   |        (10 string)      |
+------------+-------------------------+

Now you can use this specification to create a byte buffer:

(compose-buffer (spec :my-field-1 (int32-type) :my-field-2 (string-type 10)))
;= a byte buffer

Note that they keys (:my-field-1, :my-field-2) are not part of the byte buffer (not serialized in the data). If you're transferring a byte buffer over a network, the receiving end should be able to deserialize it.

Accessing Fields In The Payload

Use get-field and set-field to access individual fields of the payload.

Here's an example:

(ns my-binary-project.core
  (:require [clojurewerkz.buffy.core :refer :all]))

(let [s    (spec :int-field (int32-type)
                 :string-field (string-type 10))
      buf  (compose-buffer s)]

  (set-field buf :int-field 101)
  (get-field buf :int-field)
  ;; => 101

  (set-field buf :string-field "stringie")
  (get-field buf :string-field)
  ;; => "stringie"
  )

Deserializing complete buffer

You can also serialize and deserialize a complete buffer:

(let [s     (spec :first-field (int32-type)
                  :second-field (string-type 10)
                  :third-field (boolean-type))
      buf  (compose-buffer spec)]

  (compose buf {:first-field 101
                :second-field "string"
                :third-field true})

  (decompose buf)
  ;; => {:third-field true :second-field "string" :first-field 101}
)

Data Types

Built-in data types are:

Primitive types

  • int32-type: 32 bit integer
  • boolean-type: boolean (1 byte)
  • byte-type: a single byte
  • short-type: 16 bit integer
  • medium-type: 24 bit integer
  • float-type: 32 bit floating point
  • long-type: 64 bit integer

Arbitrary-length types

  • string-type - arbitrary size string (you define the length).

In order to construct a string-type, specify its length:

(string-type 15)
  • bytes-type - arbitrary size byte-array (you define length yourself)

Same is true for BytesType, when constructing it, just pass a number of bytes it should contain:

(bytes-type 25)

Bit type

Bit type is n bits long sequence of bits that are turned either on or off, for example,

[true true false false
 false false false false
 false false false false
 false false false false
 false false false false
 false false false false
 false false false false
 false false false false]

Translates to binary

0000 0000 0000 00011

Which translates to decimal 3, that is stored in a 4-bits integer field.

There are some helper functions, such as:

(clojurewerkz.buffy.util/bits-on-at [0 1 2])

[true true true false
 false false false false
 false false false false
 false false false false
 false false false false
 false false false false
 false false false false
 false false false false]

Or an inverse of it, clojurewerkz.buffy.util/bits-off-at.

Also, on-bits-indexes that returns positions at which bits are set, and off-bits-indexes that returns positions at which bits are cleared.

In order to use bit type, you need to give it a 32-items long sequence of truthy or falsy falues:

(let [s (spec :first-field (bit-type 4) ;; Bit field that fills 4 bytes
              :second-field (string-type 10))
      buf (compose-buffer s)]
  (set-field b :first-field [true  true  false false
                             false false false false
                             false false false false
                             false false false false
                             false false false false
                             false false false false
                             false false false false
                             false false false false]))

Complex (Composite) Types

Composite types combine multiple primitive types.

composite-type produces a slice of a buffer. In the byte representation, no paddings or offsets are added. All parts are written to the buffer sequentially:

Here's what composite type consisting of int and 10 characters long string would look like:

(composite-type (int32-type) (string-type 10))

repeated-type repeats a type one or more times. Repeated types are used when you need to have many fields of the same size:

(repeated-type (string-type 10) 5)

will produce a type consisting of 5 strings of length 10.

It's possible to combine repeated-type and composite-type:

(repeated-type (composite-type (int32-type) (string-type 10)) 5)

Which will construct a type consisting of int/string chunks repeated 5 times.

enum-type produces a mapping between human-readable values and their internal binary representation.

Consider a binary protocol where the STARTUP verb is encoded as a long value of 0x01 and the QUERY verb is encoded as 0x07:

(enum-type (long-type) {:STARTUP 0x02 :QUERY 0x07})

With this enum type, it is possible to set a field using :STARTUP and :QUERY keywords:

(set-field buffer :payload-type :STARTUP)

When reading a field, its symbolic representation is returned:

(get-field buffer :payload-type)
;; => :QUERY

Buffer types

Currently, Buffy supports direct, heap and wrapped buffers. In order to create a heap buffer:

(def my-spec (spec :first-field (int32-type)
                   :second-field (string-type 10)))
(compose-buffer my-spec :buffer-type :heap)

For off-heap (direct) buffer:

(def my-spec (spec :first-field (int32-type)
                   :second-field (string-type 10)))
(compose-buffer my-spec :buffer-type :direct)

And for wrapped buffer (that wraps the given byte array, j.nio.ByteBuffer or netty ByteBuf):

(def my-spec (spec :first-field (int32-type)
                   :second-field (string-type 10)))
(compose-buffer my-spec :orig-buffer (java.nio.ByteBuffer/allocate 14))

Dynamic Frames

If you're working with sophisticated protocols, more often than not you can't know the buffer size before you construct an entire type. One of the most primitive examples is the netstrings protocol, that consists of

(short-type) ;; Identifies the length of string
(string-type 10) ;; Identifies the string itself

Problem with construction of such type lays in the fact that you can't construct a buffer before you know the value of the string itself. Buffy helps you here, too. This feature is called dynamic frame. In order to construct a dynamic frame, you should create an encoder and decoder. Let's take a closer look at netstrings protocol implementation:

First, encoder:

(frame-encoder [value]
               ;; Name     ;; Child frame or type      ;; Dynamic value
               length      (short-type)                (count value)
               string      (string-type (count value)) value)

Here, in a binding you have a value. length part of the frame is a short-type that holds a length of the string, you specify this value through (count value).

Next off, the string itself, that is a string-type and holds a value itself.

Decoder is written in a same manner:

(frame-decoder [buffer offset]
               length (short-type)
               string (string-type (read length buffer offset)))

Since values are not decoded by that time just yet, and you may need access to an entire buffer in order to read a certain field's value, you specify only types and have a possibility of "look-behind", using already constructed types.

So, string type is constructed by reading off the length as a first field of the frame.

An entire frame would look as follows:

(def dynamic-string-payload
  (dynamic-buffer
   (frame-type
    (frame-encoder [value]
                   length (short-type) (count value)
                   string (string-type (count value)) value)
    (frame-decoder [buffer offset]
                   length (short-type)
                   string (string-type (read length buffer offset)))
    second ;; Value Formatter
    )))

second here is just a value formatter. When we read off the value from the buffer, we see the short as well as string, but it's just a helper for correct decomposition, therefore we should discard it and take only the second value, which is a string itself.

In order to compose/decompose it, you should use compose and decompose functions:

(compose dynamic-string-payload ["super-duper-random-string" "long-ans-senseless-stringyoyoyo"])

This one will return a buffer. Same with decompose, that receives dynamic buffer and a value, and returns deserialized value.

You can go ahead and create even more complicated patterns. For example, you can construct a map of strings (as in Cassandra binary CQL protocol), where the map is specified by

<short>|(repeated <string>|<string>)

Where each <string> is actually

<short>|<string itself>

It's implementation is a little bit more complex, but still reasonably simple. First, we define a dynamic string frame in the same manner as we made with netstrings:

(def dynamic-string
  (frame-type
   (frame-encoder [value]
                  length (short-type) (count value)
                  string (string-type (count value))
                  value)
   (frame-decoder [buffer offset]
                  length (short-type)
                  string (string-type (read length buffer offset)))
   second))

Next off, key-value pairs. Each one of them is nothing more than a string repeated twice.

(def key-value-pair
  (composite-frame
   dynamic-string
   dynamic-string))

Next is dynamic map, which is a frame type that holds a length which is short-type and repeated-frame of key-value-pairs:

(def dynamic-map
  (frame-type
   (frame-encoder [value]
                  length (short-type) (count value)
                  map    (repeated-frame key-value-pair (count value)) value)
   (frame-decoder [buffer offset]
                  length (short-type)
                  map    (repeated-frame key-value-pair (read length buffer offset)))
   second))

Now, our dynamic map is ready for composition and decomposition:

(let [dynamic-type (dynamic-buffer dynamic-map)]
  (compose dynamic-type [[["key1" "value1"] ["key1" "value1"] ["key1" "value1"]]]) ;; Returns a constructred buffer

  (-> dynamic-type
      (compose [[["key1" "value1"] ["key1" "value1"] ["key1" "value1"]]])
      decompose) ;; Decomposes it back to the key-value pairs

Working With Bits

In Java, there are no data types for bits, therefore we've added some wrapper functions for existing types, that may represent your values as series of 1s and 0es. For example, you can convert an integer 101 to it's binary representation:

(to-bit-map (int32-type) 101)

This will return a bitmap of 0000 0000 0000 0000 0000 0000 0110 0101 (represented as vector of true and false), which is a binary representation of 101.

Same way, you can convert a bitmap consisting of true and false back to it's actual value with from-bit-map function.

Hex Dump

It is possible to produce a hex-dump of a buffer created with Buffy using clojurewerkz.buffy.util/hex-dump. It will produce the following representation:

            +--------------------------------------------------+
            | 0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f  |
 +----------+--------------------------------------------------+------------------+
 | 00000000 | 48 69 65 72 20 69 73 74  20 65 69 6e 20 42 65 69 | Hier ist ein Bei |
 | 00000010 | 73 70 69 65 6c 74 65 78  74 2e 20 44 65 72 20 48 | spieltext. Der H |
 | 00000020 | 65 78 64 75 6d 70 20 69  73 74 20 61 75 66 20 64 | exdump ist auf d |
 | 00000030 | 65 72 20 6c 69 6e 6b 65  6e 20 53 65 69 74 65 20 | er linken Seite  |
 | 00000040 | 7a 75 20 73 65 68 65 6e  2e 20 4e 65 75 65 20 5a | zu sehen. Neue Z |
 | 00000050 | 65 69 6c 65 6e 20 6f 64  65 72 20 41 62 73 c3 a4 | eilen oder Abs.. |
 | 00000060 | 74 7a 65 20 73 69 6e 64  20 64 61 6e 6e 20 61 75 | tze sind dann au |
 | 00000070 | 63 68 20 22 5a 65 69 63  68 65 6e 22 20 6d 69 74 | ch "Zeichen" mit |
 | 00000080 | 20 65 69 6e 65 6d 20 62  65 73 74 69 6d 6d 74 65 |  einem bestimmte |
 | 00000090 | 6e 20 43 6f 64 65 20 28  30 61 29 00 00 00 00 00 | n Code (0a)..... |
 +----------+--------------------------------------------------+------------------+

Community

To subscribe for announcements of releases, important changes and so on, please follow @ClojureWerkz on Twitter.

Supported Clojure Versions

Buffy requires Clojure 1.4+.

Continuous Integration Status

Continuous Integration status

Buffy Is a ClojureWerkz Project

Buffy is part of the group of Clojure libraries known as ClojureWerkz, together with

and several others.

Development

Buffy uses Leiningen 2. Make sure you have it installed and then run tests against supported Clojure versions using

lein all test

Then create a branch and make your changes on it. Once you are done with your changes and all tests pass, submit a pull request on GitHub.

License

Copyright (C) 2013-2016 Alex Petrov, Michael S. Klishin and the ClojureWerkz Team.

Double licensed under the Eclipse Public License (the same as Clojure) or the Apache Public License 2.0.

More Repositories

1

elastisch

A minimalistic Clojure client for ElasticSearch, supports both HTTP and native transports
Clojure
386
star
2

meltdown

Clojure interface to Reactor, an event-driven programming and stream processing toolkit for the JVM
Clojure
208
star
3

money

A Clojure library that deals with monetary values and currencies. Built on top of Joda Money.
Clojure
151
star
4

eep

Embedded Event Processing in Clojure
Clojure
140
star
5

ogre

Clojure library for querying Apache TinkerPop graphs
Clojure
126
star
6

cassaforte

Modern, high-level Clojure driver (client) for Cassandra build around CQL 3
Clojure
123
star
7

balagan

Clojure data structure manipulation and querying library
Clojure
119
star
8

titanium

Clojure graph library built on top of Titan
Clojure
106
star
9

mailer

An ActionMailer-inspired mailer library. Combines Postal, Clostache, some conventions and support for multiple delivery modes
Clojure
105
star
10

route-one

Tiny Clojure library that generates HTTP resource routes (as in Ruby on Rails, Jersey, Django, Sinatra, Flask and similar)
Clojure
94
star
11

scrypt

A Clojure library for the scrypt key derivation function. Use it to encrypt passwords and other sensitive data.
Clojure
85
star
12

envision

Clojure Data Visualisation library, based on Statistiker and D3
Clojure
78
star
13

machine_head

Clojure MQTT client
Clojure
71
star
14

spyglass

A Clojure Memcached client (also: Couchbase, Kestrel). Built on top of SpyMemcached, supports ASCII and binary protocols, strives to be 100% feature complete.
Clojure
67
star
15

statistiker

Minimalistic statistics library for Clojure
Clojure
63
star
16

gizmo

Gizmo is an effortless way to create web applications in Clojure
Clojure
41
star
17

elephant

Modern Clojure client for the Stripe API
Clojure
38
star
18

archimedes

Clojure library for Blueprints (part of the Tinkerpop graph stack).
Clojure
38
star
19

docslate

Base repository for the ClojureWerkz project documentation guides
CSS
35
star
20

serialism

A tiny Clojure library that serializes and deserializes values into popular formats based on provided content type
Clojure
31
star
21

persephone

Clojure DSL that generates [Neo4J] Cypher queries
Clojure
15
star
22

monger.docs

Documentation site for Monger
JavaScript
14
star
23

romulan

LMAX Disruptor in Clojure embrace
Clojure
14
star
24

support

A support library ClojureWerkz projects (Langohr, Monger, Neocons, Elastisch, Quartzite, Welle, and others) can rely on
Clojure
14
star
25

streampunk

Clojure library for stream summarization, cardinality estimation, all that jazz
Clojure
11
star
26

ssese

Clojure Server-Sent Events (SSE) client built with Netty 5.
Clojure
9
star
27

elastisch.docs

Documentation site for Elastisch
HTML
8
star
28

triennium

Small efficient MQTT topic routing library
Clojure
6
star
29

lein-template

A Leiningen 2.0 project template for all new ClojureWerkz libraries
Clojure
6
star
30

legacy-blog

blog.clojurewerkz.org
JavaScript
5
star
31

vat

Minimalistic client for vatapi.com
Clojure
5
star
32

quartzite.docs

Documentation site for Quartzite, a powerful Clojure scheduling library built on top of Quartz
CSS
5
star
33

neocons.docs

Documentation guides for Neocons
HTML
4
star
34

langohr.examples

Code examples used in Langohr documentation guides
Clojure
4
star
35

titanium.docs

Documentation site for Titanium, see http://titanium.clojurewerkz.org
CSS
4
star
36

machine_head.examples

Code examples for Machine Head, a Clojure MQTT client
Clojure
3
star
37

validateur.docs

Documentation site for Validateur
Sass
2
star
38

welle.docs

Documentation site for Welle, an expressive Clojure client for Riak
CSS
2
star
39

clojurewerkz.org

clojurewerkz.org website
HTML
2
star
40

gizmo.docs

Docs for Gizmo, Clojurewerkz web library
Ruby
1
star
41

spyglass.docs

Spyglass documentation site
Sass
1
star
42

ogre.docs

Documentation site for Ogre, see http://ogre.clojurewerkz.org
CSS
1
star
43

archimedes.docs

Documentation site for Archimedes
CSS
1
star
44

mold

Clojure client for CloudFoundry (primarily UAA and Cloud Controller)
Clojure
1
star
45

elastisch.shield

Elastic Shield support for Elastisch
Clojure
1
star
46

machine_head.docs

Machine Head documentation site
HTML
1
star