• Stars
    star
    1,394
  • Rank 33,705 (Top 0.7 %)
  • Language
    Go
  • License
    Other
  • Created about 8 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

In-memory NoSQL database with ACID transactions, Raft consensus, and Redis API

SummitDB

SummitDB is an in-memory, NoSQL key/value database. It persists to disk, uses the Raft consensus algorithm, is ACID compliant, and built on a transactional and strongly-consistent model. It supports custom indexes, geospatial data, JSON documents, and user-defined JS scripting.

Under the hood it utilizes Finn, Redcon, BuntDB, GJSON, and Otto.

Features

Getting started

Getting SummitDB

The easiest way to get SummitDB is to use one of the pre-built release binaries which are available for OSX, Linux, and Windows. Instructions for using these binaries are on the GitHub releases page.

If you want to try the latest version, you can build SummitDB from the master branch.

Building SummitDB

SummitDB can be compiled and used on Linux, OSX, Windows, FreeBSD, ARM (Raspberry PI) and probably others since the codebase is 100% Go. We support both 32 bit and 64 bit systems. Go must be installed on the build machine.

To build simply:

$ make

It's a good idea to install the redis-cli.

$ make redis-cli

To run tests:

$ make test

Docker

Check out the SummitDB images in Docker Hub.

Running

First start a single-member cluster:

$ ./summitdb-server

This will start the server listening on port 7481 for client and server-to-server communication.

Next, let's set a single key, and then retrieve it:

$ ./redis-cli -p 7481 SET mykey "my value"
OK
$ ./redis-cli -p 7481 GET mykey
"my value"

Adding members:

$ ./summitdb-server -p 7482 -dir data2 -join localhost:7481
$ ./summitdb-server -p 7483 -dir data3 -join localhost:7481

That's it. Now if node1 goes down, node2 and node3 will continue to operate.

Differences between SummitDB and Redis

It may be worth noting that while SummitDB supports many Redis features, it is not a strict Redis clone. Redis has a lot of commands and data types that are not available in SummitDB such as Sets, Hashes, Sorted Sets, and PubSub. SummitDB also has many features that are not available in Redis such as:

  • Ordered key space - SummitDB provides one key space that is a large B-tree. An ordered key space allows for stable paging through keys using the KEYS command. Redis uses an unordered dictionary structure and provides a specialized SCAN command for iterating through keys.
  • Everything a string - SummitDB stores only strings which are exact binary representations of what the user stores. Redis has many internal data types, such as strings, hashes, floats, sets, etc.
  • Raft clusters - SummitDB uses the Raft consensus algorithm to provide high-availablity. Redis provides Master/Slave replication.
  • Javascript - SummitDB uses Javascript for user-defined scripts. Redis uses Lua.
  • Indexes - SummitDB provides an API for indexing the key space. Indexes allow for quickly querying and iterating on values. Redis has specialized data types like Sorted Sets and Hashes which can provide secondary indexing.
  • Spatial indexes - SummitDB provides the ability to create spatial indexes. A spatial index uses an R-tree under the hood, and each index can be up to 20 dimensions. This is useful for geospatial, statistical, time, and range data. Redis has the GEO API which allows for using storing and querying geospatial data using the Geohashes.
  • JSON documents - SummitDB allows for storing JSON documents and indexing fields directly. Redis has Hashes and a JSON parser via Lua.

In-memory with disk persistence

SummitDB store all data in memory. Yet each writable command is appended to a file that is used to rebuild the database if the database needs to be restarted.

This is similar to Redis AOF persistence.

JSON Documents

SummitDB provides the commands JSET, JGET, JDEL for working with json documents.

JSET and JDEL uses the sjson path syntax and JGET uses the gjson path syntax.

Here are some examples:

> JSET user:101 name Tom
OK
> JSET user:101 age 46
OK
> GET user:101
"{\"age\":46,\"name\":\"Tom\"}"
> JGET user:101 age
"46"
> JSET user:101 name.first Tom
OK
> JSET user:101 name.last Anderson
OK
> GET user:101
"{\"age\":46,\"name\":{\"last\":\"Anderson\",\"first\":\"Tom\"}}"
> JDEL user:101 name.last
(integer) 1
> GET user:101
"{\"age\":46,\"name\":{\"first\":\"Tom\"}}"
> JSET user:101 friends.0 Carol
OK
> JSET user:101 friends.1 Andy
OK
> JSET user:101 friends.3 Frank
OK
> GET user:101
"{\"friends\":[\"Carol\",\"Andy\",null,\"Frank\"],\"age\":46,\"name\":{\"first\":\"Tom\"}}"
> JGET user:101 friends.1
"Andy"

JSON Indexes

Indexes can be created on individual fields inside JSON documents.

For example, let's say you have the following documents:

{"name":{"first":"Tom","last":"Johnson"},"age":38}
{"name":{"first":"Janet","last":"Prichard"},"age":47}
{"name":{"first":"Carol","last":"Anderson"},"age":52}
{"name":{"first":"Alan","last":"Cooper"},"age":28}

Create an index:

> SETINDEX last_name user:* JSON name.last

Then add some JSON:

> SET user:1 '{"name":{"first":"Tom","last":"Johnson"},"age":38}'
> SET user:2 '{"name":{"first":"Janet","last":"Prichard"},"age":47}'
> SET user:3 '{"name":{"first":"Carol","last":"Anderson"},"age":52}'
> SET user:4 '{"name":{"first":"Alan","last":"Cooper"},"age":28}'

Query with the ITER command:

> ITER last_name
1) "user:3"
2) "{\"name\":{\"first\":\"Carol\",\"last\":\"Anderson\"},\"age\":52}"
3) "user:4"
4) "{\"name\":{\"first\":\"Alan\",\"last\":\"Cooper\"},\"age\":28}"
5) "user:1"
6) "{\"name\":{\"first\":\"Tom\",\"last\":\"Johnson\"},\"age\":38}"
7) "user:2"
8) "{\"name\":{\"first\":\"Janet\",\"last\":\"Prichard\"},\"age\":47}"

Or perhaps you want to index on age:

> SETINDEX age user:* JSON age
> ITER age
1) "user:4"
2) "{\"name\":{\"first\":\"Alan\",\"last\":\"Cooper\"},\"age\":28}"
3) "user:1"
4) "{\"name\":{\"first\":\"Tom\",\"last\":\"Johnson\"},\"age\":38}"
5) "user:2"
6) "{\"name\":{\"first\":\"Janet\",\"last\":\"Prichard\"},\"age\":47}"
7) "user:3"
8) "{\"name\":{\"first\":\"Carol\",\"last\":\"Anderson\"},\"age\":52}"

It's also possible to multi-index on two fields:

> SETINDEX last_name_age user:* JSON name.last JSON age

For full JSON indexing syntax check out the SETINDEX and ITER commands.

Fencing Tokens

A fencing token is simply a number that increases. It's guaranteed to be consistent across the cluster and can never be deleted or decreased. The value is a 64-bit unsigned integer. The first FENCE call will return "1". This can be useful in applications that need things like distributed locking and preventing race conditions. FENCEGET will read the token without incrementing it.

> FENCE mytoken
"1"
> FENCE mytoken
"2"
> FENCE mytoken
"3"
> FENCEGET mytoken
"3"
> FENCE mytoken
"4"

Built-in Raft Commands

Here are a few commands for monitoring and managing the cluster:

  • RAFTADDPEER addr
    Adds a new member to the Raft cluster
  • RAFTREMOVEPEER addr
    Removes an existing member
  • RAFTPEERS
    Lists known peers and their status
  • RAFTLEADER
    Returns the Raft leader, if known
  • RAFTSNAPSHOT
    Triggers a snapshot operation
  • RAFTSTATE
    Returns the state of the node
  • RAFTSTATS
    Returns information and statistics for the node and cluster

Consistency and Durability

SummitDB is tuned by design for strong consistency and durability. A server shutdown, power event, or kill -9 will not corrupt the state of the cluster or lose data.

All data persists to disk. SummitDB uses an append-only file format that stores for each command in exact order of execution. Each command consists of a one write and one fsync. This provides excellent durability.

Read Consistency

The --consistency param has the following options:

  • low - all nodes accept reads, small risk of stale data
  • medium - only the leader accepts reads, itty-bitty risk of stale data during a leadership change
  • high - only the leader accepts reads, the raft log index is incremented to guarantee no stale data. this is the default

For example, setting the following options:

$ summitdb --consistency high

Provides the highest level of consistency. The default is high.

Leadership Changes

In a Raft cluster only the leader can apply commands. If a command is attempted on a follower you will be presented with the response:

> SET x y
-TRY 127.0.0.1:7481

This means you should try the same command at the specified address.

Hot Backups

SummitDB supports hot-backing up a node. You can retrieve and restore a snapshot of the database to a file using the BACKUP command.

> BACKUP
BULK REPLY OF DATA

Or using an HTTP connection like such:

curl localhost:7481/backup -o backup.db

The backup file format is a series of commands which are stored as RESP Arrays. The command:

SET mykey 123

Is stored on disk as:

"*3\r\n$3\r\nSET\r\n$5\r\nmykey\r\n$3\r\n123\r\n"

To restore a system from a backup, issue each command to the leader. For example, using the nc command you could execute:

$ cat backup.db | nc localhost 7481

Commands

Below is the complete list of commands.

Keys and values
APPEND, BITCOUNT, BITOP, BITPOS, DBSIZE, DECR, DECRBY, DEL, EXISTS, EXPIRE, EXPIREAT, FENCE, FENCEGET, FLUSHDB, GET, GETBIT, GETRANGE, GETSET, INCR, INCRBY, INCRBYFLOAT, KEYS, MGET, MSET, MSETNX, PDEL, PERSIST, PEXPIRE, PEXPIREAT, PTTL, RENAME, RENAMENX, SET, SETBIT, SETRANGE, STRLEN, TTL

JSON JSET, JGET, JDEL

Indexes and iteration
DELINDEX, INDEXES, ITER, RECT, SETINDEX

Transactions
MULTI, EXEC, DISCARD

Scripts
EVAL, EVALRO, EVALSHA, EVALSHARO, SCRIPT LOAD, SCRIPT FLUSH

Raft management
RAFTADDPEER, RAFTREMOVEPEER, RAFTLEADER, RAFTSNAPSHOT, RAFTSTATE, RAFTSTATS

Server
BACKUP

Contact

Josh Baker @tidwall

License

SummitDB source code is available under the MIT License.

More Repositories

1

gjson

Get JSON values quickly - JSON parser for Go
Go
12,768
star
2

tile38

Real-time Geospatial and Geofencing
Go
8,655
star
3

evio

Fast event-loop networking for Go
Go
5,747
star
4

buntdb

BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
Go
4,196
star
5

redcon

Redis compatible server framework for Go
Go
2,005
star
6

sjson

Set JSON values very quickly in Go
Go
1,978
star
7

SwiftWebSocket

Fast Websockets in Swift for iOS and OSX
Swift
1,532
star
8

jj

JSON Stream Editor (command line utility)
Go
1,293
star
9

btree

B-tree implementation for Go
Go
937
star
10

hashmap.c

Hash map implementation in C.
C
631
star
11

uhaha

High Availability Raft Framework for Go
Go
579
star
12

pinhole

3D Wireframe Drawing Library for Go
Go
554
star
13

finn

Fast Raft framework using the Redis protocol for Go
Go
541
star
14

wal

Write ahead log for Go.
Go
525
star
15

tg

Geometry library for C - Fast point-in-polygon
C
502
star
16

Safe

Modern Concurrency and Synchronization for Swift.
Swift
417
star
17

pretty

Efficient JSON beautifier and compactor for Go
Go
354
star
18

chanx

A simple interface wrapper around a Go channel.
Go
321
star
19

rtree

An R-tree implementation for Go
Go
285
star
20

btree.c

B-tree implementation in C
C
246
star
21

GoSwift

Go Goodies for Swift. Including goroutines, channels, defer, and panic.
Swift
234
star
22

shardmap

A simple and efficient thread-safe sharded hashmap for Go
Go
211
star
23

hashmap

A simple and efficient hashmap package for Go. Open addressing, robin hood hashing, and xxh3 algorithm. Supports generics.
Go
203
star
24

celltree

A fast in-memory prefix tree that uses uint64 for keys and allows for duplicate entries.
Go
201
star
25

gjson.rs

Get JSON values quickly - JSON parser for Rust
Rust
195
star
26

cities

10,000 Cities with Latitude, Longitude, and Elevation in Go
Go
161
star
27

tinylru

A fast little LRU cache for Go
Go
144
star
28

Avios

Realtime H264 decoding library for iOS.
Swift
129
star
29

jd

Interactive JSON Editor
Go
128
star
30

pinhole-js

3D Wireframe Drawing Library for HTML Canvas
JavaScript
119
star
31

mmap

Load file-backed memory
Go
117
star
32

geojson

GeoJSON for Go. Used by Tile38
Go
116
star
33

doppio

Doppio is a fast LRU cache on top of Ristretto, Redcon, and Evio. Support for the Redis protocol.
Go
115
star
34

rtree.rs

A fast R-tree for Rust
Rust
112
star
35

match

Simple string pattern matcher for Go
Go
97
star
36

rtree.c

An R-tree implementation in C
C
96
star
37

tinybtree

Just an itsy bitsy b-tree in Go
Go
94
star
38

jsonc

Parse json with comments and trailing commas.
Go
79
star
39

rhh

A simple and efficient hashmap package for Go. Uses open addressing, Robin Hood hashing, and xxhash algorithm.
Go
79
star
40

kvnode

key value server. redis api, leveldb storage, raft support
Go
79
star
41

raft-fastlog

Raft in-memory backend implementation with persistence
Go
78
star
42

resp

Reader, Writer, and Server implementation for the Redis RESP Protocol.
Go
78
star
43

modern-server

Basic web server framework with HTTP/2 and Let's Encrypt.
Go
76
star
44

limiter

A goroutine limiter for Go
Go
76
star
45

redraft

Redis + Raft server implementation
Go
75
star
46

redcon.rs

Redis compatible server framework for Rust
Rust
72
star
47

rocksdb-server

Fast Redis clone written in C using RocksDB as a backend.
C++
72
star
48

lotsa

Simple Go library for executing lots of operations spread over any number of threads
Go
71
star
49

box

Efficiently box values in Go. Optimized for primitives, strings, and byte slices.
Go
69
star
50

evio-lite

Fast event-loop networking for Go (the lite version)
Go
68
star
51

transform

Using io.Reader for data transformation in Go
Go
67
star
52

uspto-trademark

How to file a USPTO trademark without an attorney for $225
66
star
53

pjson

A JSON stream parser for Go
Go
66
star
54

geodesic

Go package for performing accurate measurements of Earth. Includes the geodesic routines from GeographicLib.
Go
59
star
55

expr

Expression evaluator for Go
Go
58
star
56

json.c

Fast JSON parser for C
C
56
star
57

spinlock

A spinlock implementation for Go.
Go
56
star
58

raft-wal

Write ahead Raft log for Go
Go
55
star
59

rtred

RTree implementation for Go
Go
53
star
60

bfile

A buffer pool file I/O library for Go
Go
53
star
61

mvt

Draw Mapbox Vector Tiles (MVT) in Go
Go
52
star
62

redcon.c

Redis compatible server framework for C
C
47
star
63

cache-server

A minimal key/value server written in Rust with Redis API support.
Rust
46
star
64

spanmap

A fast collection type that uses uint64 for keys.
Go
44
star
65

go-node

Run Javascript in Go using Node.js
Go
42
star
66

ticketd

A distributed service for monotonically increasing tickets.
Go
42
star
67

hexd

Please love the nicely formatted hex.
Go
42
star
68

DeflateSwift

Deflate Compression for Swift
Swift
40
star
69

proximity-chat

Chat app for real-time chats with people within 500 meters.
JavaScript
39
star
70

lru

A simple and efficient LRU cache package for Go
Go
38
star
71

sider

A Redis clone written in Go
Go
36
star
72

xv

An expression evaluator for C
C
34
star
73

geometry

Efficient 2D geometry library for Go.
Go
32
star
74

pkg.sh

A generalized package manager for whatever code.
Shell
32
star
75

rfront

An HTTP protocol frontend for Redis-compatible services.
Go
31
star
76

redis-gis

Redis fork with Geospatial support based on Tile38 commands
C
31
star
77

RetroSwiper

Load classic games from magnetic swipe cards
Rust
31
star
78

uhasql

A high availability Sqlite service
C
30
star
79

redbench

Benchmarking for custom Redis commands and modules
Go
30
star
80

sds

simple data streams for go
Go
29
star
81

spmap

A hashmap for Go that uses crypto random seeds, hash hints, open addressing, and robin hood hashing.
Go
29
star
82

IoniconsSwift

Ionicons for Swift and iOS
Swift
29
star
83

raft-leveldb

Raft backend using LevelDB
Go
28
star
84

evio.c

A framework for building event based networking applications.
C
28
star
85

tinyqueue

Binary heap priority queues in Go
Go
27
star
86

kvbench

Server for benchmarking pure Go key/value databases
Go
26
star
87

gjson-play

A playground for GJSON. Runs in the browser.
JavaScript
25
star
88

randjson

Make random JSON in Go
Go
25
star
89

redlog

Redis style logger for Go
Go
25
star
90

SnapHTTP

An incredibly simple HTTP client library for Swift.
Swift
24
star
91

qtree

jeez it's just a quadtree chill out
Go
24
star
92

rtime

Retrieve the current time from remote servers
Go
21
star
93

fast-spatial-joins

Go vs GPU: Fast Spatial Joins
Go
21
star
94

assert

An assert function for Go that works like the one in C.
Go
20
star
95

pair

create low memory key/value objects in Go
Go
20
star
96

match.c

Simple string pattern matcher for C
C
19
star
97

secret

A simple utility for encrypting and decrypting data in Go (AES-256-CFB)
Go
19
star
98

pony

🌈 🐴 Turn your terminal text into an absolutely beautiful display of dazzling colors....
Go
19
star
99

btree-benchmark

Benchmark utility for the tidwall/btree Go package
Go
19
star
100

lru-server

A convenient LRU cache server that supports REST API and Let's Encrypt.
Go
18
star