Slim - surprisingly space efficient data types in Golang

Slim is collection of surprisingly space efficient data types, with corresponding serialization APIs to persisting them on-disk or for transport.

Why slim
Performance and memory overhead
Synopsis
Filter mode and KV mode.
Try it
- Versions
Who are using slim
Feedback and contributions
Authors
License

Why slim

As data on internet keeps increasing exponentially, the capacity gap between memory and disk becomes greater.

Most of the time, a data itself does not need to be loaded into expensive main memory. Only the much more important information, WHERE-A-DATA-IS, deserve a seat in main memory.

This is what slim does, keeps as little information as possible in main memory, as a minimized index of huge amount external data.

SlimIndex: is a common index structure, building on top of SlimTrie.
SlimTrie is the underlying index data structure, evolved from trie.

Features:
- Minimized: 11 bits per key(far less than an 64-bits pointer!!).
- Stable: memory consumption is stable in various scenarios. The Worst case converges to average consumption tightly. See benchmark.
- Loooong keys: You can have VERY long keys(16K bytes), without any waste of memory(and money). Do not waste your life writing another prefix compression:). (aws-s3 limits key length to 1024 bytes). Memory consumption only relates to key count, not to key length.
- Ordered: like btree, keys are stored. Range-scan will be ready in 0.6.0.
- Fast: ~150 ns per Get(). Time complexity for a get is O(log(n) + k); n: key count; k: key length.
- Ready for transport: a single proto.Marshal() is all it requires to serialize, transport or persisting on disk etc.

Performance and memory overhead

3.3 times faster than the btree.
2.3 times faster than binary search.

Memory overhead is about 11 bit per key.

The data struct in this benchmark is a slice of key-value pairs with a SlimTrie serving as the index. The slim itself is built in the filter mode, to maximize memory reduction and performance. The whole struct slimKV is a fully functional kv-store, just like a static btree.

type slimKV struct {
    slim *trie.SlimTrie
    Elts []*KVElt
}
type KVElt struct {
    Key string
    Val int32
}

You can find the benchmark code in benchmark;

Synopsis

1. Index on-disk key-values

One of the typical usages of slim is to index serialized data on disk(e.g., key value records in a SSTable). By keeping a slim in memory, one can quickly find the on-disk offset of the record by a key.

Show me the code ......

package index_test

import (
	"fmt"
	"strings"

	"github.com/openacid/slim/index"
)

type Data string

func (d Data) Read(offset int64, key string) (string, bool) {
	kv := strings.Split(string(d)[offset:], ",")[0:2]
	if kv[0] == key {
		return kv[1], true
	}
	return "", false
}

func Example() {

	// Accelerate external data accessing (in memory or on disk) by indexing
	// them with a SlimTrie:

	// `data` is a sample of some unindexed data. In our example it is a comma
	// separated key value series.
	//
	// In order to let SlimTrie be able to read data, `data` should have
	// a `Read` method:
	//     Read(offset int64, key string) (string, bool)
	data := Data("Aaron,1,Agatha,1,Al,2,Albert,3,Alexander,5,Alison,8")

	// keyOffsets is a prebuilt index that stores key and its offset in data accordingly.
	keyOffsets := []index.OffsetIndexItem{
		{Key: "Aaron", Offset: 0},
		{Key: "Agatha", Offset: 8},
		{Key: "Al", Offset: 17},
		{Key: "Albert", Offset: 22},
		{Key: "Alexander", Offset: 31},
		{Key: "Alison", Offset: 43},
	}

	// `SlimIndex` is simply a container of SlimTrie and its data.
	st, err := index.NewSlimIndex(keyOffsets, data)
	if err != nil {
		fmt.Println(err)
	}

	// Lookup
	v, found := st.Get("Alison")
	fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "Alison", found, v)

	v, found = st.Get("foo")
	fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "foo", found, v)

	// Output:
	// key: "Alison"
	//   found: true
	//   value: "8"
	// key: "foo"
	//   found: false
	//   value: ""
}

2. Sparse index

Create an index item for every 4(or more as you wish) keys.

Let several adjacent keys share one index item reduces a lot memory cost if there are huge amount keys in external data. Such as to index billions of 4KB objects on a 4TB disk(because one disk IO costs 20ms for either reading 4KB or reading 1MB).

Show me the code ......

package index_test

import (
	"fmt"
	"strings"

	"github.com/openacid/slim/index"
)

type RangeData string

func (d RangeData) Read(offset int64, key string) (string, bool) {
	for i := 0; i < 4; i++ {
		if int(offset) >= len(d) {
			break
		}

		kv := strings.Split(string(d)[offset:], ",")[0:2]
		if kv[0] == key {
			return kv[1], true
		}
		offset += int64(len(kv[0]) + len(kv[1]) + 2)

	}
	return "", false
}

func Example_indexRanges() {

	// Index ranges instead of keys:
	// In this example at most 4 keys shares one index item.

	data := RangeData("Aaron,1,Agatha,1,Al,2,Albert,3,Alexander,5,Alison,8")

	// keyOffsets is a prebuilt index that stores range start, range end and its offset.
	keyOffsets := []index.OffsetIndexItem{
		// Aaron  +--> 0
		// Agatha |
		// Al     |
		// Albert |

		// Alexander +--> 31
		// Alison    |

		{Key: "Aaron", Offset: 0},
		{Key: "Agatha", Offset: 0},
		{Key: "Al", Offset: 0},
		{Key: "Albert", Offset: 0},

		{Key: "Alexander", Offset: 31},
		{Key: "Alison", Offset: 31},
	}

	st, err := index.NewSlimIndex(keyOffsets, data)
	if err != nil {
		panic(err)
	}

	v, found := st.RangeGet("Aaron")
	fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "Aaron", found, v)

	v, found = st.RangeGet("Al")
	fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "Al", found, v)

	v, found = st.RangeGet("foo")
	fmt.Printf("key: %q\n  found: %t\n  value: %q\n", "foo", found, v)

	// Output:
	// key: "Aaron"
	//   found: true
	//   value: "1"
	// key: "Al"
	//   found: true
	//   value: "2"
	// key: "foo"
	//   found: false
	//   value: ""
}

3. Range scan

Slim can also be used as a traditional in-memory kv-store: Building a slim with Opt{ Complete: Bool(true) }, it won't strip out any information(e.g., it won't eliminate single-branch labels) and it will functions the same as a btree. This snippet shows how to iterate key values.

Show me the code ......

package trie

import (
	"fmt"

	"github.com/openacid/slim/encode"
)

func ExampleSlimTrie_ScanFrom() {
	var keys = []string{
		"",
		"`",
		"a",
		"ab",
		"abc",
		"abca",
		"abcd",
		"abcd1",
		"abce",
		"be",
		"c",
		"cde0",
		"d",
	}
	values := makeI32s(len(keys))

	codec := encode.I32{}
	st, _ := NewSlimTrie(codec, keys, values, Opt{
		Complete: Bool(true),
	})

	// untilD stops when encountering "d".
	untilD := func(k, v []byte) bool {
		if string(k) == "d" {
			return false
		}

		_, i32 := codec.Decode(v)
		fmt.Println(string(k), i32)
		return true
	}

	fmt.Println("scan (ab, +∞):")
	st.ScanFrom("ab", false, true, untilD)

	fmt.Println()
	fmt.Println("scan [be, +∞):")
	st.ScanFrom("be", true, true, untilD)

	fmt.Println()
	fmt.Println("scan (ab, be):")
	st.ScanFromTo(
		"ab", false,
		"be", false,
		true, untilD)

	// Output:
	//
	// scan (ab, +∞):
	// abc 4
	// abca 5
	// abcd 6
	// abcd1 7
	// abce 8
	// be 9
	// c 10
	// cde0 11
	//
	// scan [be, +∞):
	// be 9
	// c 10
	// cde0 11
	//
	// scan (ab, be):
	// abc 4
	// abca 5
	// abcd 6
	// abcd1 7
	// abce 8
}

Filter mode and KV mode.

Slim can be built into either a filter(like bloom filter but with key order preserved.) or a real kv-store(like btree) There is an option in NewSlimTrie(..., option) to control the building behavior. Ref: Opt

To use slim as a kv-store, set the option to Complete then there won't be false positives.

To use it as a filter, set InnerPrefix, LeafPrefix to false(Complete implies InnerPrefix==true and LeafPrefix==true). Then slim won't store any single branch label in the trie it builds.

With InnerPrefix==true, it does not reduce a single label branch that leads to an inner node.

With LeafPrefix==true, it does not reduce a single label branch that leads to a leaf node.

E.g.:

// Complete
InnerPrefix: true
LeafPrefix: true
^ -a-> 1 -b-> $
 `-c-> 2 -x-> 3 -y-> $
               `-z-> $

InnerPrefix: true
LeafPrefix: false
^ -a-> $
 `-c-> 2 -x-> 3 -y-> $
               `-z-> $

InnerPrefix: false
LeafPrefix: true
^ -a-> 1 -b-> $
 `-c-> 3 -y-> $
        `-z-> $

InnerPrefix: false
LeafPrefix: false
^ -a-> $
 `-c-> 3 -y-> $
        `-z-> $

The memory consumption in filter mode and kv mode differs significantly. The following chart shows memory consumption by 1 million var-length string, 10 to 20 byte in different mode:

-	size	gzip-size
sample data size	15.0M	14.0M
Complete:true	14.0M	10.0M
InnerPrefix:ture	1.3M	0.9M
all false	1.3M	0.8M

Try it

Install

go get github.com/openacid/slim/trie

Change-log: Change-log

Versions

A newer version y being compatible with an older version x means y can load data serialized by x. But x should never try to load data serialized by a newer version y.