• Stars
    star
    1,469
  • Rank 30,819 (Top 0.7 %)
  • Language
    Clojure
  • License
    MIT License
  • Created about 13 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Clojure JSON and JSON SMILE (binary json format) encoding/decoding

Cheshire

'Cheshire Puss,' she began, rather timidly, as she did not at all know whether it would like the name: however, it only grinned a little wider. 'Come, it's pleased so far,' thought Alice, and she went on. 'Would you tell me, please, which way I ought to go from here?'

'That depends a good deal on where you want to get to,' said the Cat.

'I don't much care where--' said Alice.

'Then it doesn't matter which way you go,' said the Cat.

'--so long as I get SOMEWHERE,' Alice added as an explanation.

'Oh, you're sure to do that,' said the Cat, 'if you only walk long enough.'

Cheshire is fast JSON encoding, based off of clj-json and clojure-json, with additional features like Date/UUID/Set/Symbol encoding and SMILE support.

Clojure code with docs

Clojars Project Continuous Integration status

Why?

clojure-json had really nice features (custom encoders), but was slow; clj-json had no features, but was fast. Cheshire encodes JSON fast, with added support for more types and the ability to use custom encoders.

Usage

[cheshire "5.13.0"]

;; Cheshire v5.13.0 uses Jackson 2.17.0

;; In your ns statement:
(ns my.ns
  (:require [cheshire.core :refer :all]))

Encoding

;; generate some json
(generate-string {:foo "bar" :baz 5})

;; write some json to a stream
(generate-stream {:foo "bar" :baz 5} (clojure.java.io/writer "/tmp/foo"))

;; generate some SMILE
(generate-smile {:foo "bar" :baz 5})

;; generate some JSON with Dates
;; the Date will be encoded as a string using
;; the default date format: yyyy-MM-dd'T'HH:mm:ss'Z'
(generate-string {:foo "bar" :baz (java.util.Date. 0)})

;; generate some JSON with Dates with custom Date encoding
(generate-string {:baz (java.util.Date. 0)} {:date-format "yyyy-MM-dd"})

;; generate some JSON with pretty formatting
(generate-string {:foo "bar" :baz {:eggplant [1 2 3]}} {:pretty true})
;; {
;;   "foo" : "bar",
;;   "baz" : {
;;     "eggplant" : [ 1, 2, 3 ]
;;   }
;; }

;; generate JSON escaping UTF-8
(generate-string {:foo "It costs Β£100"} {:escape-non-ascii true})
;; => "{\"foo\":\"It costs \\u00A3100\"}"

;; generate JSON and munge keys with a custom function
(generate-string {:foo "bar"} {:key-fn (fn [k] (.toUpperCase (name k)))})
;; => "{\"FOO\":\"bar\"}"

;; generate JSON without escaping the characters (by writing it to a file)
(spit "foo.json" (json/generate-string {:foo "bar"} {:pretty true}))

In the event encoding fails, Cheshire will throw a JsonGenerationException.

Custom Pretty Printing Options

If Jackson's default pretty printing library is not what you desire, you can manually create your own pretty printing class and pass to the generate-string or encode methods:

(let [my-pretty-printer (create-pretty-printer
                          (assoc default-pretty-print-options
                                 :indent-arrays? true))]
  (generate-string {:foo [1 2 3]} {:pretty my-pretty-printer}))

See the default-pretty-print-options for a list of options that can be changed.

Decoding

;; parse some json
(parse-string "{\"foo\":\"bar\"}")
;; => {"foo" "bar"}

;; parse some json and get keywords back
(parse-string "{\"foo\":\"bar\"}" true)
;; => {:foo "bar"}

;; parse some json and munge keywords with a custom function
(parse-string "{\"foo\":\"bar\"}" (fn [k] (keyword (.toUpperCase k))))
;; => {:FOO "bar"}

;; top-level strings are valid JSON too
(parse-string "\"foo\"")
;; => "foo"

;; parse some SMILE (keywords option also supported)
(parse-smile <your-byte-array>)

;; parse a stream (keywords option also supported)
(parse-stream (clojure.java.io/reader "/tmp/foo"))

;; parse a stream lazily (keywords option also supported)
(parsed-seq (clojure.java.io/reader "/tmp/foo"))

;; parse a SMILE stream lazily (keywords option also supported)
(parsed-smile-seq (clojure.java.io/reader "/tmp/foo"))

In 2.0.4 and up, Cheshire allows passing in a function to specify what kind of types to return, like so:

;; In this example a function that checks for a certain key
(decode "{\"myarray\":[2,3,3,2],\"myset\":[1,2,2,1]}" true
        (fn [field-name]
          (if (= field-name "myset")
            #{}
            [])))
;; => {:myarray [2 3 3 2], :myset #{1 2}}

The type must be "transient-able", so use either #{} or []

Custom Encoders

Custom encoding is supported from 2.0.0 and up, if you encounter a bug, please open a github issue. From 5.0.0 onwards, custom encoding has been moved to be part of the core namespace (not requiring a namespace change)

;; Custom encoders allow you to swap out the api for the fast
;; encoder with one that is slightly slower, but allows custom
;; things to be encoded:
(ns myns
  (:require [cheshire.core :refer :all]
            [cheshire.generate :refer [add-encoder encode-str remove-encoder]]))

;; First, add a custom encoder for a class:
(add-encoder java.awt.Color
             (fn [c jsonGenerator]
               (.writeString jsonGenerator (str c))))

;; There are also helpers for common encoding actions:
(add-encoder java.net.URL encode-str)

;; List of common encoders that can be used: (see generate.clj)
;; encode-nil
;; encode-number
;; encode-seq
;; encode-date
;; encode-bool
;; encode-named
;; encode-map
;; encode-symbol
;; encode-ratio

;; Then you can use encode from the custom namespace as normal
(encode (java.awt.Color. 1 2 3))
;; => "java.awt.Color[r=1,g=2,b=3]"

;; Custom encoders can also be removed:
(remove-encoder java.awt.Color)

;; Decoding remains the same, you are responsible for doing custom decoding.

NOTE: `cheshire.custom` has been deprecated in version 5.0.0

Custom and Core encoding have been combined in Cheshire 5.0.0, so there is no longer any need to require a different namespace depending on what you would like to use.

Aliases

There are also a few aliases for commonly used functions:

encode -> generate-string
encode-stream -> generate-stream
encode-smile -> generate-smile
decode -> parse-string
decode-stream -> parse-stream
decode-smile -> parse-smile

Features

Cheshire supports encoding standard clojure datastructures, with a few additions.

Cheshire encoding supports:

Clojure data structures

  • strings
  • lists
  • vectors
  • sets
  • maps
  • symbols
  • booleans
  • keywords (qualified and unqualified)
  • numbers (Integer, Long, BigInteger, BigInt, Double, Float, Ratio, Short, Byte, primitives)
  • clojure.lang.PersistentQueue

Java classes

  • Date
  • UUID
  • java.sql.Timestamp
  • any java.util.Set
  • any java.util.Map
  • any java.util.List

Custom class encoding while still being fast

Also supports

  • Stream encoding/decoding
  • Lazy decoding
  • Pretty-printing JSON generation
  • Unicode escaping
  • Custom keyword coercion
  • Arbitrary precision for decoded values:

Cheshire will automatically use a BigInteger if needed for non-floating-point numbers, however, for floating-point numbers, Doubles will be used unless the *use-bigdecimals?* symbol is bound to true:

(ns foo.bar
  (require [cheshire.core :as json]
           [cheshire.parse :as parse]))

(json/decode "111111111111111111111111111111111.111111111111111111111111111111111111")
;; => 1.1111111111111112E32 (a Double)

(binding [parse/*use-bigdecimals?* true]
  (json/decode "111111111111111111111111111111111.111111111111111111111111111111111111"))
;; => 111111111111111111111111111111111.111111111111111111111111111111111111M (a BigDecimal)

Change Log

Change log is available on GitHub.

Speed

Cheshire is about twice as fast as data.json.

Check out the benchmarks in cheshire.test.benchmark; or run lein benchmark. If you have scenarios where Cheshire is not performing as well as expected (compared to a different library), please let me know.

Experimental things

In the cheshire.experimental namespace:

$ echo "Hi. \"THIS\" is a string.\\yep." > /tmp/foo

$ lein repl
user> (use 'cheshire.experimental)
nil
user> (use 'clojure.java.io)
nil
user> (println (slurp (encode-large-field-in-map {:id "10"
                                                  :things [1 2 3]
                                                  :body "I'll be removed"}
                                                 :body
                                                 (input-stream (file "/tmp/foo")))))
{"things":[1,2,3],"id":"10","body":"Hi. \"THIS\" is a string.\\yep.\n"}
nil

encode-large-field-in-map is used for streamy JSON encoding where you want to JSON encode a map, but don't want the map in memory all at once (it returns a stream). Check out the docstring for full usage.

It's experimental, like the name says. Based on Tigris.

Advanced customization for factories

See this and this for a list of features that can be customized if desired. A custom factory can be used like so:

(ns myns
  (:require [cheshire.core :as core]
            [cheshire.factory :as factory]))

(binding [factory/*json-factory* (factory/make-json-factory
                                  {:allow-non-numeric-numbers true})]
  (json/decode "{\"foo\":NaN}" true))))))

See the default-factory-options map in factory.clj for a full list of configurable options. Smile factories can also be created, and factories work exactly the same with custom encoding.

Future Ideas/TODOs

  • move away from using Java entirely, use Protocols for the custom encoder (see custom.clj)
  • allow custom encoders (see custom.clj)
  • figure out a way to encode namespace-qualified keywords
  • look into overriding the default encoding handlers with custom handlers
  • better handling when java numbers overflow ECMAScript's numbers (-2^31 to (2^31 - 1))
  • handle encoding java.sql.Timestamp the same as java.util.Date
  • add benchmarking
  • get criterium benchmarking ignored for 1.2.1 profile
  • look into faster exception handling by pre-allocating an exception object instead of creating one on-the-fly (maybe ask Steve?)
  • make it as fast as possible (ongoing)

License

Release under the MIT license. See LICENSE for the full license.

Thanks

Thanks go to Mark McGranaghan for clj-json and Jim Duey for the name suggestion. :)

More Repositories

1

clj-http

An idiomatic clojure http client wrapping the apache client. Officially supported version.
Clojure
1,755
star
2

clojure-opennlp

Natural Language Processing in Clojure (opennlp)
Clojure
747
star
3

elasticsearch-in-action

Offical code repository for the Elasticsearch in Action book from Manning
Shell
370
star
4

eos

Welcome to the Emacs of Things, aka the Emacs Operating System
Emacs Lisp
261
star
5

es-mode

An Emacs major mode for interacting with Elasticsearch
Emacs Lisp
192
star
6

itsy

A threaded web-spider written in Clojure
Clojure
180
star
7

lein-bikeshed

A Leiningen plugin designed to tell you your code is bad, and that you should feel bad
Clojure
177
star
8

ox-tufte

Emacs' Org-mode export backend for Tufte HTML
Emacs Lisp
98
star
9

dakrone-dotfiles

misc configuration files
Vim Script
89
star
10

cld

Language detection for Clojure
Clojure
42
star
11

emacs-java-imports

Add java imports easily in Emacs
Emacs Lisp
40
star
12

clojuredocs-client

A tiny client for the http://clojuredocs.org API
Clojure
36
star
13

ricepaper

Simple library and CLI tool for adding URLs to InstaPaper
Ruby
35
star
14

nsm-console

Network Security Monitoring Console
Ruby
23
star
15

forkify

Do work from a pool of processes using forks. Like threadify with processes.
Ruby
18
star
16

lein-autotest

A Leiningen plugin to start Lazytest's autowatch
Clojure
17
star
17

fastri

Fastri, now with 1.9 support
Ruby
13
star
18

cadastre

Survey a clojure project and extract valuable metadata
Clojure
13
star
19

dakrone-theme

dakrone's custom emacs color theme
Emacs Lisp
11
star
20

dakrone-light-theme

Dakrone's custom light Emacs theme
Emacs Lisp
11
star
21

dakrone.github.com

webness
HTML
10
star
22

eisago

Next-gen clojuredocs importer, API, and website.
Clojure
9
star
23

syndicate

Fun with NLP and synonyms.
Clojure
9
star
24

tigris

Stream-to-stream JSON string escaping
Java
7
star
25

lein-clojuredocs

Generate data about your project to submit to clojuredocs.org
Clojure
7
star
26

lids

Locality Intrusion Detection System
C++
6
star
27

elasticsearch-clojure-plugin

A proof-of-concept Elasticsearch plugin written entirely in Clojure
Clojure
6
star
28

denverclojure

Denver Clojure meetup website
Clojure
5
star
29

rcapr

A Ruby library to interact with the pcapr website (http://pcapr.net)
Ruby
4
star
30

integrity-cap-notifier

Notifier for integrity that does capistrano deployments when a build passes
Ruby
4
star
31

clomoios

Context searching using NLP magic
Clojure
4
star
32

one-offs

Simple-file projects, one-offs and miscellaneous stuff.
Ruby
3
star
33

skyyy

A small ruby script to deal with Skype over dbus
Ruby
3
star
34

ruby-datasuite

A simple interactive/scriptable random data generation and verification test tool.
3
star
35

cbench

A small clojure benchmarking helper
Clojure
3
star
36

nile

Stream utilities for everyday Clojure use
Clojure
3
star
37

labview_rails

A lab machine tracking system using rails (for sysadmins)
Ruby
3
star
38

biblybot

Bibly bot is a Google Wave robot for whatever I feel like
Clojure
3
star
39

corpus

a tool used to train a detokenization library
Clojure
2
star
40

churn

Churn data given a directory and change rate
Java
2
star
41

jsontest

testing speeds of clojure json libs
Clojure
2
star
42

elasticsearch-nrepl

Embedded nREPL in ElasticSearch
Java
2
star
43

felix

A handy garbage monitor for monitoring when clojure objects are GC'd
Clojure
2
star
44

clj-http-async

clj-http, but with Apache's async client instead
Clojure
2
star
45

metube

Youtube downloading as an immutant service
Clojure
2
star
46

clj2010

Stats from #clojure, forked from https://bitbucket.org/tebeka/clj2010/overview
1
star
47

norad

SQS Message consumption for Immutant queues
Clojure
1
star
48

download-test

don't look
1
star
49

cd-client

clojuredocs API client
1
star
50

chrojos

a clojure library for parsing irrational datetime strings
Clojure
1
star
51

syn

Don't look at me yet
Clojure
1
star
52

gh-upload

Github file uploader
Clojure
1
star
53

recipi

Personal web server for storing recipes
Clojure
1
star
54

chunktest

test
Clojure
1
star
55

screamy

Immutant queue notifications
Clojure
1
star
56

org-criterium

Work in progress
Clojure
1
star
57

ars-capture

Periodically capture adaptive replica stats and store them in a different ES cluster
Clojure
1
star