• Stars
    star
    102
  • Rank 335,584 (Top 7 %)
  • Language
    Clojure
  • License
    Eclipse Public Li...
  • Created almost 9 years ago
  • Updated over 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A library for simple wire-like connectivity semantics.

kabel CircleCI

kabel (German for cable/wire) is a minimal, modern connection library modelling a bidirectional wire to pass Clojure values between peers. Peers in Clojure and ClojureScript are symmetric and hence allow symmetric cross-platform implementations. Clojure peers can connect to Clojure and ClojureScript peers in the same way and vice versa. kabel can use any bidirectional messaging channel, currently it supports web-sockets. It also ships a transit middleware for efficient serialization. It works on different JavaScript runtimes, currently tested are the Browser, node.js and React-Native.

Rationale

Instead of implementing a REST interface websockets provide several benefits, even if you do single requests. Most importantly the distinction between server and client is unnecessary, because both can push messages to each other, effectively having one input and one output channel. Together with edn messages over the wire this simplifies the semantics significantly. The tradeoff is that REST is standardized and offers better interoperablity for other clients.

Since we work on a crossplatform p2p software for confluent replicated datatypes with replikativ, we could not reuse any of the other ClojureScript client-side only websocket libraries, e.g. sente or chord. For us all IO happens over the input and output channel with core.async, so we can implement cross-platform functionality in a very terse and expressive fashion, e.g. in the pull-hooks for replikativ. But you do not need to write platform neutral symmetric middlewares, so on the JVM you can of course do IO without core.async.

We also extended and build on superv.async to catch all exceptions in an Erlang style monitoring fashion and propagate them back through a parametrizable error channel. We are thinking about ways to refactor kabel, so that it can be decoupled from this error handling without losing the error handling guarantees.

Usage

Add this to your project dependencies:

Clojars Project

From the examples folder (there is also a cljs client there):

(ns kabel.examples.pingpong
  (:require [kabel.client :as cli]
            [kabel.http-kit :as http-kit]
            [kabel.peer :as peer]
            [superv.async :refer [<?? go-try S go-loop-try <? >? put?]]
            [clojure.core.async :refer [chan]]
            ;; you can use below transit if you prefer
            [kabel.middleware.transit :refer [transit]]
            [hasch.core :refer [uuid]]))


;; this url is needed for the server to open the proper
;; socket and for the client to know where to connect to
(def url "ws://localhost:47291")


;; server peer code
(defn pong-middleware [[S peer [in out]]]
  (let [new-in (chan)
        new-out (chan)]
    ;; we just mirror the messages back
    (go-loop-try S [i (<? S in)]
      (when i
        (>? S out i)
        (recur (<? S in))))
    ;; Note that we pass through the supervisor, peer and new channels
    [S peer [new-in new-out]]))

;; this is useful to track messages, so each peer should have a unique id
(def server-id #uuid "05a06e85-e7ca-4213-9fe5-04ae511e50a0")

(def server (peer/server-peer S (http-kit/create-http-kit-handler! S url server-id) server-id
                              ;; here you can plug in your (composition of) middleware(s)
                              pong-middleware
                              ;; we chose no serialization (pr-str/read-string by default)
                              identity
                              ;; we could also pick the transit middleware
                              #_transit))

;; we need to start the peer to open the socket
(<?? S (peer/start server))


(def client-id #uuid "c14c628b-b151-4967-ae0a-7c83e5622d0f")

;; client
(def client (peer/client-peer S client-id
                              ;; Here we have a simple middleware to trigger some roundtrips
                              ;; from the client
                              (fn [[S peer [in out]]]
                                (let [new-in (chan)
                                      new-out (chan)]
                                  (go-try S
                                          (put? S out "ping")
                                          (println "1. client incoming message:" (<? S in))
                                          (put? S out "ping2")
                                          (println "2. client incoming message:" (<? S in)))
                                  [S peer [new-in new-out]]))
                              ;; we need to pick the same middleware for serialization
                              ;; (no auto-negotiation yet)
                              identity
                              #_transit))


;; let's connect the client to the server
(<?? S (peer/connect S client url))


(comment
  ;; and stop the server
  (<?? S (peer/stop speer))
  )

The client-side works the same in ClojureScript from the browser.

Applications

Design

Example pub-sub architecture of replikativ

There is a pair of channels for each connection, but at the core the peer has a pub-sub architecture. Besides using some middleware specific shared memory like a database you can more transparently pass messages to other clients and middlewares through this pub-sub core or subscribe to specific message types on it. It uses the pub-sub semantics of core.async:

(let [[bus-in bus-out] (get-in @peer [:volatile :chans])
      b-chan (chan)]
  (async/sub bus-out :broadcast b-chan)
  (async/put! bus-in {:type :broadcast :hello :everybody})
  (<!! b-chan))

Middlewares

You can find general middlewares in the corresponding folder. In general middlewares themselves form a "wire" and can filter, transform, inject and pass through messages.

Serialization

Serialization is also done in a middleware, a transit middleware is currently provided and used by default. If you do not use any serialization middleware than a simple pr-str <-> read-string mechanism is combined with a very simple binary header to track different serialization types over the wire including raw binary data.

External

We provide the following middlewares separately: - Passwordless authentication (and authorisation) based on email verification or password and inter-peer trust network as p2p middleware.

Useful middlewares still missing:

  • QoS monitoring, latency and throughput measures
  • remote debugging, sending superv.async exceptions back to the server
  • other usefull ring middlewares which can be wrapped?

Connectivity

More transport alternatives like long-polling with SSEs, WebRTC, NFC, ... or normal sockets should not be hard to add.

TODO

  • implement configuration as circling of config handshake before serialization to allow adaptable configurations between peers, e.g. transit+msgpack on JVM, transit + json with javascript peers etc. The configuration message circles through all middlewares until a consensus/fix-point is reached and then middlewares start their lifecycle.
  • factor platform neutral logging
  • implement node.js websocket server
  • implement more of wamp client protocol (and maybe router)
  • investigate https://github.com/touch/pubsure-core

Contributors

  • Konrad Kühne
  • Sang-Kyu Park
  • Brian Marco
  • Christian Weilbach

License

Copyright © 2015-2017 Christian Weilbach, 2015 Konrad Kühne

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.

More Repositories

1

datahike

A fast, immutable, distributed & compositional Datalog engine for everyone.
Clojure
1,630
star
2

replikativ

An open, scalable and distributive infrastructure for a data-driven community of applications.
Clojure
330
star
3

konserve

A clojuresque key-value/document store protocol with core.async.
Clojure
298
star
4

superv.async

This is a Clojure(Script) library that extends core.async with error handling and includes a number of convenience functions and macros.
Clojure
171
star
5

hasch

Cross-platform (JVM and JS atm.) edn data structure hashing for Clojure.
Clojure
108
star
6

datalog-parser

Generic datalog parser compliant to datomic, datascript and datahike queries.
Clojure
69
star
7

hitchhiker-tree

Functional, persistent, off-heap, high performance data structure
Clojure
43
star
8

datahike-server

Datahike remote system
Clojure
36
star
9

geheimnis

Cross-platform cryptography between cljs and clj.
Clojure
35
star
10

datahike-frontend

A front-end connecting to a Datahike back-end.
Clojure
33
star
11

incognito

Safe transport of unknown record types in distributed systems.
Clojure
22
star
12

durable-persistence

Explorations in durable persistent datastructures for Clojure.
Clojure
22
star
13

datahike-postgres

Datahike with Postgres as data storage
Clojure
17
star
14

datahike-jdbc

Datahike JDBC data storage backend
Clojure
16
star
15

chat42

A small web chat demonstration with replikativ.
Clojure
14
star
16

topiq

A distributed social network for serious discussions and funny topiqs :).
Clojure
13
star
17

twitter-collector

A simple twitter collector using replikativ.
Clojure
11
star
18

filesync-replikativ

A filesystem synchronization tool similar to Dropbox over replikativ.
Clojure
11
star
19

konserve-carmine

A redis backend with carmine for konserve.
Clojure
10
star
20

kabel-auth

Authentication middleware for kabel.
Clojure
9
star
21

zufall

random name generator
Clojure
7
star
22

chat42app

A react native demo for chat42.
Clojure
6
star
23

mesalog

CSV data loader for Datalog databases
Clojure
5
star
24

flechtwerk

flechtwerk provides visualization of commit graphs of CDVCS.
Clojure
5
star
25

datahike-redis

Clojure
5
star
26

datahike-s3

Datahike backend for S3.
Clojure
5
star
27

konserve-leveldb

A LevelDB backend for konserve.
Clojure
4
star
28

replikativ-demo

Example project for replikativ in clj.
Clojure
4
star
29

datahike-client

A datahike remote client
Clojure
3
star
30

konserve-rocksdb

A RocksDB backend for konserve
Clojure
3
star
31

datahike-benchmark

Measuring of datahike performance
Clojure
3
star
32

replikativ-cljs-demo

Example project for replikativ in cljs.
Clojure
3
star
33

konserve-jdbc

A JDBC backend for konserve.
Clojure
2
star
34

datahike-leveldb

Datahike with LevelDB as data storage
Clojure
2
star
35

datahike-server-transactor

Transactor implementation for datahike that uses datahike-server.
Clojure
2
star
36

konserve-clutch

A CouchDB backend for konserve with clutch.
Clojure
2
star
37

polo-collector

A collector of trading data on the Poloniex exchange.
Clojure
2
star
38

konserve-welle

A Riak backend for konserve with Welle.
Clojure
2
star
39

sherpa

A version migration tool for Datahike.
Clojure
1
star
40

konserve-s3

S3 backend for konserve.
Clojure
1
star
41

obhut

Docker setup for operational log analytics
1
star
42

datahike-fdb

FoundationDB backend for Datahike
Clojure
1
star
43

konserve-redis

Redis backend for konserve.
Clojure
1
star
44

sendandi

Protocol above datalog databases for common functions
Clojure
1
star
45

datahike-rocksdb

Datahike RocksDB support.
Clojure
1
star
46

wanderung-core

Migration protocols for Datahike
Clojure
1
star
47

wanderung-datascript

DataScript-Datahike migrations
Clojure
1
star
48

datahike-migrations

A migration tool for Datahike
Clojure
1
star
49

pydatahike

Python bindings for Datahike.
Python
1
star