Discover willemt/pearldb Open Source project by Willem (@willemt)

https://travis-ci.org/willemt/pearldb.png

What?

PearlDB is a durable HTTP Key-Value pair database. It uses LMDB for storing data, and H2O for HTTP.

PearlDB is completely written in C.

Persistent connections and pipelining are built-in.

PearlDB uses bmon to batch LMDB writes.

Goals

Speed
Low latency
Durability - An HTTP response means the write is on disk
Simplicity outside (RESTful inteface)
Simplicity inside (succinct codebase)
HTTP caching - Because the CRUD is RESTful you could hypothetically use an HTTP reverse proxy cache to scale out reads. You could use multiple caches to create an eventually consistent database

Ubuntu Quick Start

sudo add-apt-repository -y ppa:willemt/pearldb
sudo apt-get update
sudo apt-get install pearldb

Usage

Examples below make use of the excellent httpie

Starting

build/pearl --daemonize --port 8000 --db_size 1 --pid_file pearl.pid
echo daemonizing...

daemonizing...

Get

We obtain a value by GET'ng the key.

In this case the key is "x". But we get a 404 if it doesn't exist.

http -h --ignore-stdin 127.0.0.1:8000/x/

HTTP/1.1 404 NOT FOUND
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

You MUST specify a path.

http -h --ignore-stdin 127.0.0.1:8000/ | head -n 1

HTTP/1.1 400 BAD PATH

Put

We use PUT for creating or updating a key value pair. PUTs are durable - we only respond when the change has been made to disk.

echo "MY VALUE" | http -h PUT 127.0.0.1:8000/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
transfer-encoding: chunked

PUTs have an immediate change on the resource. There is full isolation, and therefore no dirty reads.

Now we can finally retrieve our data via a GET:

http --ignore-stdin 127.0.0.1:8000/x/

MY VALUE

The slash at the end is optional.

http --ignore-stdin 127.0.0.1:8000/x

MY VALUE

The user must specify the capacity of the database upfront. PearlDB does not support automatic resizing. A PUT will fail if it would put the database over capacity.

head -c 1000000 /dev/urandom | base64 > tmp_file
du -h tmp_file | awk '{ print $1 }'
cat tmp_file | http -h PUT 127.0.0.1:8000/1/
rm tmp_file

1.3M
HTTP/1.1 400 NOT ENOUGH SPACE
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

You can't PUT under nested resources.

echo 'DATA' | http -h PUT 127.0.0.1:8000/x/nested_resource/

HTTP/1.1 400 BAD PATH
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

Put without a key (POST)

If you want PearlDB to generate a key for you, just use POST.

echo "MY POSTED VALUE" | http -h POST 127.0.0.1:8000/ > posted.txt
cat posted.txt

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
location: ...
transfer-encoding: chunked

The Location header in the response has the URI of the newly created resource. The URI is the URL safe base64 encoded UUID4.

http --ignore-stdin -b GET 127.0.0.1:8000$(grep location: posted.txt | sed -e 's/location: //' | tr -d '\r\n')

MY POSTED VALUE

Providing a URL (ie. key) with POST doesn't make sense, and will result in a 400.

echo "MY POSTED VALUE" | http -h POST 127.0.0.1:8000/xxxx/

HTTP/1.1 400 BAD
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

Get keys

You can get the keys that match a prefix by using the /key/XXX/ nested resource.

echo '' | http PUT 127.0.0.1:8000/1/ > /dev/null
echo '' | http PUT 127.0.0.1:8000/199/ > /dev/null
echo '' | http PUT 127.0.0.1:8000/102/ > /dev/null
echo '' | http PUT 127.0.0.1:8000/2/ > /dev/null
http GET 127.0.0.1:8000/key/1/

1
102
199

Without a prefix you get all keys.

http GET 127.0.0.1:8000/key// | sed -e '/^.*=$/d'

Existence Check

To check for existence use the HEAD method. This is great, because PearlDB doesn't waste bandwidth sending the document body.

http -h --ignore-stdin HEAD 127.0.0.1:8000/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4

Delete

DELETEs are durable - we only respond when the change has been made to disk.

http -h --ignore-stdin DELETE 127.0.0.1:8000/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
transfer-encoding: chunked

Of course, after a DELETE the key doesn't exist anymore:

http -h --ignore-stdin 127.0.0.1:8000/x/

HTTP/1.1 404 NOT FOUND
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

Compare and Swap (CAS)

A form of opportunistic concurrency control is available through ETags.

When the client provides the Prefers: ETag header on a GET request we generate an ETag. A client can then use the If-Match header with the ETag to perform a conditional update, (ie. a CAS operation). If the ETag has changed then the PUT operation will fail. CAS operations are great because there is no locking; if a CAS operation fails for one client that means it has succeeded for another, ie. there has been progress.

Imagine two clients trying to update the same key. Client 1 requests an ETag. The ETag is provided via the etag header.

echo 'SWEET DATA' | http -h --ignore-stdin PUT 127.0.0.1:8000/x/ > /dev/null
http -h --ignore-stdin GET 127.0.0.1:8000/x/ Prefers:ETag > etag.txt
cat etag.txt

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
etag: ...
transfer-encoding: chunked

If client 1 requests an ETag again, the same ETag is sent:

http -h --ignore-stdin GET 127.0.0.1:8000/x/ Prefers:ETag > etag2.txt
cat etag2.txt
diff <(grep etag etag.txt) <(grep etag etag2.txt)

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
etag: ...
transfer-encoding: chunked

Client 2 does a PUT on x. This will invalidate the ETag.

echo 'SURPRISE' | http -h PUT 127.0.0.1:8000/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
transfer-encoding: chunked

Client 1 uses a conditional PUT to update "x" using the If-Match tag. Because the ETag was invalidated, we don't commit, and respond with 412 Precondition Failed.

echo 'MY NEW VALUE BASED OFF OLD VALUE' | http -h PUT 127.0.0.1:8000/x/ If-Match:$(grep etag: etag.txt | sed -e 's/etag: //' | tr -d '\r\n')

HTTP/1.1 412 BAD ETAG
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

Once this happens we can retry the PUT after we do a new GET.

http -h GET 127.0.0.1:8000/x/ Prefers:ETag > etag3.txt
cat etag3.txt

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
etag: ...
transfer-encoding: chunked

The PUT will succeed because the ETag is still valid.

echo 'NEW VALUE' | http -h PUT 127.0.0.1:8000/x/ If-Match:$(grep etag: etag3.txt | sed -e 's/etag: //' | tr -d '\r\n')

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
transfer-encoding: chunked

However, if we use the ETag again it will fail.

echo 'NEW VALUE2' | http -h PUT 127.0.0.1:8000/x/ If-Match:$(grep etag: etag3.txt | sed -e 's/etag: //' | tr -d '\r\n')

HTTP/1.1 412 BAD ETAG
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
content-length: 0

Notes about ETags:

On reboots, PearlDB loses all ETag information
On launch PearlDB generates a random ETag prefix
ETags are expected to have a short life (ie. < 1 day)

OPTIONS

You can check what HTTP methods are available to a resource using the OPTIONS method. This is useful as some systems like HAProxy use the OPTIONS method as a healthcheck.

http -h --ignore-stdin OPTIONS 127.0.0.1:8000/x/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
allow: HEAD,GET,PUT,DELETE,OPTIONS
transfer-encoding: chunked

http -h --ignore-stdin OPTIONS 127.0.0.1:8000/

HTTP/1.1 200 OK
Date: ..., ... .... ........ GMT
Connection: keep-alive
Server: h2o/2.2.4
allow: POST,OPTIONS
transfer-encoding: chunked

Shutting down

cat pearl.pid | xargs kill -9
echo shutdown

shutdown

Building

sudo apt-get install git cmake automake libtool libssl-dev
make libuv
make libh2o
make libck
make

willemt/pearldb

willemt

Reviews

Repository Details