• Stars
    star
    296
  • Rank 136,076 (Top 3 %)
  • Language
    Clojure
  • License
    Eclipse Public Li...
  • Created over 8 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A clojuresque key-value/document store protocol with core.async.

konserve

https://img.shields.io/badge/slack-join_chat-brightgreen.svg https://img.shields.io/clojars/v/io.replikativ/konserve.svg https://circleci.com/gh/replikativ/konserve.svg?style=shield https://img.shields.io/github/last-commit/replikativ/konserve/development.svg https://versions.deps.co/replikativ/konserve/status.svg

Simple durability, made flexible.

A simple document store protocol defined with synchronous and core.async semantics to allow Clojuresque collection operations on associative key-value stores, both from Clojure and ClojureScript for different backends. Data is generally serialized with edn semantics or, if supported, as native binary blobs and can be accessed similarly to clojure.core functions get-in, assoc-in and update-in. update-in especially allows to run functions atomically and returns old and new value. Each operation is run atomically and must be consistent (in fact ACID), but further consistency is not supported (Riak, CouchDB and many scalable solutions don’t have transactions over keys for that reason). This is meant to be a building block for more sophisticated storage solutions (Datomic also builds on kv-stores). A simple append-log for fast write operations is also implemented.

Features

  • cross-platform between Clojure and ClojureScript
  • lowest-common denominator interface for an associative datastructure with edn semantics
  • thread-safety with atomicity over key operations
  • consistent error handling for core.async
  • fast serialization options (fressian, transit, …), independent of the underlying kv-store
  • very low overhead protocol, including direct binary access for high throughput
  • no additional dependencies and setup required for IndexedDB in the browser and the file backend on the JVM
  • avoids blocking io, the filestore for instance will not block any thread on reading. Fully asynchronous support for writing and other stores is in the pipeline.

Garbage Collector

Konserve has a garbage collector that can be called manually when the store gets too crowded. For that, the function konserve.gc/sweep! allows you to provide a cut-off date to evict old keys and a whitelist for keys that should be kept.

Error handling

For synchronous execution normal exceptions will be thrown. For asynchronous error handling we follow the semantics of go-try and <? introduced here. We have the superv.async library around the error handling in core.async, but since there is no need to push it onto the users of konserve, you just need these two macros that properly handle the errors. <? needs to check for an exception and rethrow and go-try needs to catch and pass it along as a return value such that it does not get lost.

Usage

Add to your dependencies: http://clojars.org/io.replikativ/konserve/latest-version.svg

Synchronous Execution

Run the following synchronous code if you are not using core.async in your scope:

(ns test-db
  (:require [konserve.filestore :refer [connect-fs-store]]
            [konserve.core :as k]))

(def store (connect-fs-store "/tmp/store" :opts {:sync? true}))

(k/assoc-in store ["foo" :bar] {:foo "baz"} {:sync? true})
(k/get-in store ["foo"] nil {:sync? true})
(k/exists? store "foo" {:sync? true})

(k/assoc-in store [:bar] 42 {:sync? true})
(k/update-in store [:bar] inc {:sync? true})
(k/get-in store [:bar] nil {:sync? true})
(k/dissoc store :bar {:sync? true})

(k/append store :error-log {:type :horrible} {:sync? true})
(k/log store :error-log {:sync? true})

(let [ba (byte-array (* 10 1024 1024) (byte 42))]
  (time (k/bassoc store "banana" ba {:sync? true})))

(k/bget store "banana"
        (fn [{is :input-stream}]
          (your-read-does-all-work-here is))
        {:sync? true})

Asynchronous Execution

In a ClojureScript REPL you can evaluate the expressions from the REPL each wrapped in a go-block.

(ns test-db
  (:require [konserve.memory :refer [new-mem-store]]
            [clojure.core.async :refer [go <!]]))

(go (def my-db (<! (new-mem-store)))) ;; or (go (def my-db (<!

From a Clojure REPL run the following functions for the core.async variants of the code.

(ns test-db
  (:require [konserve.filestore :refer [connect-fs-store]]
            [konserve.core :as k]
            [clojure.core.async :refer [go <!]]))

(go
  (def store (<! (connect-fs-store "/tmp/store")))

  (<! (k/assoc-in store ["foo" :bar] {:foo "baz"}))
  (<! (k/get-in store ["foo"]))
  (<! (k/exists? store "foo"))

  (<! (k/assoc-in store [:bar] 42))
  (<! (k/update-in store [:bar] inc))
  (<! (k/get-in store [:bar]))
  (<! (k/dissoc store :bar))

  (<! (k/append store :error-log {:type :horrible}))
  (<! (k/log store :error-log))

  (let [ba (byte-array (* 10 1024 1024) (byte 42))]
    (time (<! (k/bassoc store "banana" ba)))))

Supported Backends

In-Memory Store

For simple purposes a memory store wrapping an Atom is implemented for Clojure and ClojureScript.

Usage:

(ns test-db
  (:require [konserve.memory :refer [new-mem-store]]
            [konserve.core :as k]))

(def my-db (new-mem-store))

fs-store

A file-system store in Clojure and for Node are provided as elementary reference implementations for the two most important platforms. No setup and no additional dependencies are needed.

The file-system store currently uses fressian in Clojure and fress in ClojureScript and is quite efficient. Both implementations use the same on-disk format and can load the same store (but not concurrently). It also allows to access values as a normal file-system file, e.g. to open it with a native database like HDF5 in Java. You can decide not to fsync on every write by a configuration of {:sync-blob? false}, if a potential, but unlikely data loss is not critical for you (e.g. for a session store). Note that the database will not be corrupted in this case, you can just lose some write operations before the crash.

Usage:

(ns test-db
  (:require [konserve.memory :refer [connect-fs-store]]
            [konserve.core :as k]))

(def my-folder "path/to/folder")
(def my-db (connect-fs-store my-folder))

IndexedDB

IndexedDB is provided as reference implementation for ClojureScript browser backends.

Usage:

(ns test-db
  (:require [konserve.memory :refer [connect-idb-store]]
            [konserve.core :as k])
  (:require-macros [cljs.core.async.macros :refer [go]]))

(def dbname "example-db")
(go (def my-db (<! (connect-idb-store dbname))))

External Backends

We recently updated konserve not only to provide the option to choose between synchronous and asynchronous execution, but also to provide protocols that simplify the implementation of external backends. Unfortunately most of the external backends are deprecated now because of this.

Supported backend implementations are:

Please let us know if you are interested in other backends or if you need help with implementing one.

The following projects are incompatible with the latest konserve release, but describe the usage of the underlying store API and could still be helpful to implement new backends for the underlying store:

Serialization formats

Different formats for edn serialization like fressian, transit or a simple pr-str version are supported and can be combined with different stores. Stores have a reasonable default setting. You can also extend the serialization protocol to other formats if you need it. You can provide incognito support for records, if you need them.

Tagged Literals

You can read and write custom records according to incognito.

Compression and encryption

Compression and encryption are supported by the default store implementation that is used by all current backends. They can be activated in the store configuration as follows:

{:encryptor {:type :aes
             :key "s3cr3t"}
 :compressor {:type :lz4}}

LZ4 compression is currently only supported on the JVM. AES encryption is supported on both JVM and JS targets with the same cold storage format, i.e. the same store can be read and written from Clojure and ClojureScript runtimes. We use AES/CBC/PKCS{5/7}Padding with 256 bit and a different salt for each written value.

Backend implementation guide

We provide a backend implementation guide.

Projects building on konserve

  • The protocol is used in production and originates as an elementary storage protocol for replikativ and datahike.
  • kampbell maps collections of entities to konserve and enforces specs.

Combined usage with other writers

konserve assumes currently that it accesses its keyspace in the store exclusively. It uses hasch to support arbitrary edn keys and hence does not normally clash with outside usage even when the same keys are used. To support multiple konserve clients in the store the backend has to support locking and proper transactions on keys internally, which is the case for backends like CouchDB, Redis and Riak.

Todo

Changelog

0.7.274

  • implement dual async+sync code expansion
  • generalize filestore logic to ease backend development

0.6.0-alpha1

  • introduce common storage layouts and store serialization context with each key value pair, this will facilitate migration code in the future
  • implementation for the filestore (thanks to @FerdiKuehne)
  • introduce metadata to track edit timestamps
  • add garbage collector
  • introduce superv.async error handling
  • extend API to be more like Clojure’s (thanks to @MrEbbinghaus)
  • add logging
  • update on ClojureScript support still pending

0.5.1

  • fix nested value extraction in filestore, thanks to @csm

0.5

  • cljs fressian support
  • filestore for node.js

0.5-beta3

  • experimental caching support

0.5-beta1

  • improved filestore with separate metadata storage
  • experimental clojure.core.cache support

0.4.12

  • fix exists for binary

0.4.11

  • friendly printing of stores on JVM

0.4.9

  • fix a racecondition in the lock creation
  • do not drain the threadpool for the filestore

0.4.7

  • support distinct dissoc (not implicit key-removal on assoc-in store key nil)

0.4.5

  • bump deps

0.4.4

  • make fsync configurable

0.4.3

  • remove full.async until binding issues are resolved

0.4.2

  • simplify and fix indexeddb
  • do clean locking with syntactic macro sugar

0.4.1

  • fix cljs support

0.4.0

  • store the key in the filestore and allow to iterate stored keys (not binary atm.)
  • implement append functions to have high throughput append-only logs
  • use core.async based locking on top-level API for all stores
  • allow to delete a file-store

0.3.6

  • experimental JavaScript bindings

0.3.4

  • use fixed incognito version

0.3.0 - 0.3.2

  • fix return value of assoc-in

0.3.0-beta3

  • Wrap protocols in proper Clojure functions in the core namespace.
  • Implement assoc-in in terms of update-in
  • Introduce serialiasation protocol with the help of incognito and decouple stores

0.3.0-beta1

  • filestore: disable cache
  • factor out all tagged literal functions to incognito
  • use reader conditionals
  • bump deps

0.2.3

  • filestore: flush output streams, fsync on fs operations
  • filestore can be considered beta quality
  • couchdb: add -exists?
  • couchdb: move to new project
  • remove logging and return ex-info exceptions in go channel

0.2.2

  • filestore: locking around java strings is a bad idea, use proper lock objects
  • filestore: do io inside async/thread (like async’s pipeline) to not block the async threadpool
  • filestore: implement a naive cache (flushes once > 1000 values)
  • filestore, indexeddb: allow to safely custom deserialize file-inputstream in transaction/lock
  • filestore, indexeddb, memstore: implement -exists?

0.2.1

  • filestore: fix fressian collection types for clojure, expose read-handlers/write-handlers
  • filestore: fix -update-in behaviour for nested values
  • filestore: fix rollback renaming order

0.2.0

  • experimental native ACID file-store for Clojure
  • native binary blob support for file-store, IndexedDB and mem-store

Contributors

  • Björn Ebbinghaus
  • Daniel Szmulewicz
  • Konrad Kühne
  • Christian Weilbach

License

Copyright © 2014-2023 Christian Weilbach and contributors

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

More Repositories

1

datahike

A durable Datalog implementation adaptable for distribution.
Clojure
1,584
star
2

replikativ

An open, scalable and distributive infrastructure for a data-driven community of applications.
Clojure
330
star
3

superv.async

This is a Clojure(Script) library that extends core.async with error handling and includes a number of convenience functions and macros.
Clojure
171
star
4

hasch

Cross-platform (JVM and JS atm.) edn data structure hashing for Clojure.
Clojure
108
star
5

kabel

A library for simple wire-like connectivity semantics.
Clojure
102
star
6

datalog-parser

Generic datalog parser compliant to datomic, datascript and datahike queries.
Clojure
69
star
7

hitchhiker-tree

Functional, persistent, off-heap, high performance data structure
Clojure
43
star
8

datahike-server

Datahike remote system
Clojure
35
star
9

geheimnis

Cross-platform cryptography between cljs and clj.
Clojure
35
star
10

datahike-frontend

A front-end connecting to a Datahike back-end.
Clojure
33
star
11

incognito

Safe transport of unknown record types in distributed systems.
Clojure
22
star
12

durable-persistence

Explorations in durable persistent datastructures for Clojure.
Clojure
22
star
13

datahike-jdbc

Datahike JDBC data storage backend
Clojure
16
star
14

datahike-postgres

Datahike with Postgres as data storage
Clojure
16
star
15

chat42

A small web chat demonstration with replikativ.
Clojure
14
star
16

topiq

A distributed social network for serious discussions and funny topiqs :).
Clojure
13
star
17

twitter-collector

A simple twitter collector using replikativ.
Clojure
11
star
18

filesync-replikativ

A filesystem synchronization tool similar to Dropbox over replikativ.
Clojure
11
star
19

konserve-carmine

A redis backend with carmine for konserve.
Clojure
10
star
20

kabel-auth

Authentication middleware for kabel.
Clojure
9
star
21

zufall

random name generator
Clojure
7
star
22

chat42app

A react native demo for chat42.
Clojure
6
star
23

mesalog

CSV data loader for Datalog databases
Clojure
5
star
24

flechtwerk

flechtwerk provides visualization of commit graphs of CDVCS.
Clojure
5
star
25

datahike-redis

Clojure
5
star
26

datahike-s3

Datahike backend for S3.
Clojure
5
star
27

konserve-leveldb

A LevelDB backend for konserve.
Clojure
4
star
28

replikativ-demo

Example project for replikativ in clj.
Clojure
4
star
29

datahike-client

A datahike remote client
Clojure
3
star
30

konserve-rocksdb

A RocksDB backend for konserve
Clojure
3
star
31

datahike-benchmark

Measuring of datahike performance
Clojure
3
star
32

replikativ-cljs-demo

Example project for replikativ in cljs.
Clojure
3
star
33

datahike-leveldb

Datahike with LevelDB as data storage
Clojure
2
star
34

datahike-server-transactor

Transactor implementation for datahike that uses datahike-server.
Clojure
2
star
35

konserve-clutch

A CouchDB backend for konserve with clutch.
Clojure
2
star
36

polo-collector

A collector of trading data on the Poloniex exchange.
Clojure
2
star
37

konserve-welle

A Riak backend for konserve with Welle.
Clojure
2
star
38

sherpa

A version migration tool for Datahike.
Clojure
1
star
39

konserve-s3

S3 backend for konserve.
Clojure
1
star
40

obhut

Docker setup for operational log analytics
1
star
41

konserve-redis

Redis backend for konserve.
Clojure
1
star
42

konserve-jdbc

A JDBC backend for konserve.
Clojure
1
star
43

datahike-fdb

FoundationDB backend for Datahike
Clojure
1
star
44

datahike-migrations

A migration tool for Datahike
Clojure
1
star
45

wanderung-core

Migration protocols for Datahike
Clojure
1
star
46

sendandi

Protocol above datalog databases for common functions
Clojure
1
star
47

datahike-rocksdb

Datahike RocksDB support.
Clojure
1
star
48

wanderung-datascript

DataScript-Datahike migrations
Clojure
1
star
49

pydatahike

Python bindings for Datahike.
Python
1
star