• This repository has been archived on 15/Nov/2021
  • Stars
    star
    98
  • Rank 345,882 (Top 7 %)
  • Language
    Go
  • License
    MIT License
  • Created over 6 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Low-level key/value store in pure Go.

Build Status Go Report Card Documentation

Description

Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), values stored on disk.

Description on russian: https://habr.com/post/354224/

slowpoke

Motivation

Replace Bolt with a simpler and more efficient engine.

Slowpoke (from version 2.0) based on pudge

How it works

Keys are stored in memory with persistence to disk. Values stored on disk only.

Slowpoke is parallel

Server

GRPC Server example: okdb

Complex examples

typegram

zen platform for authors and their subscribers with a minimalistic design and user-friendly interface.

golang-gin-realworld-example-app

This codebase was created to demonstrate a fully fledged fullstack application built with Golang/Gin/Slowpoke including CRUD operations, authentication, routing, pagination, and more.

Basic example

package main

import (
	"fmt"

	"github.com/recoilme/slowpoke"
)

func main() {
	// create database
	file := "test/example.db"
	// close all opened database
	defer slowpoke.CloseAll()
	// init key/val
	key := []byte("foo")
	val := []byte("bar")
	//store
	slowpoke.Set(file, key, val)
	// get
	res, _ := slowpoke.Get(file, key)
	//result
	fmt.Println(string(res))
}

Lazy example

func TestGob(t *testing.T) {
	file := "test/gob.db"
	DeleteFile(file)
	defer CloseAll()
	type Post struct {
		Id       int
		Content  string
		Category string
	}

	for i := 0; i < 20; i++ {
		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
		err := SetGob(file, post.Id, post)
		ch(err, t)
	}

	for i := 0; i < 20; i++ {
		var post = new(Post)
		err := GetGob(file, i, post)
		ch(err, t)
		fmt.Println("i:", i, "Post:", post)
	}
}

Advanced example

type Post struct {
	Id       int
	Content  string
	Category string
}

func main() {
	posts := "test/posts"
	tags := "test/tags"
	var pairs [][]byte
	for i := 0; i < 40; i++ {
		id := make([]byte, 4)
		binary.BigEndian.PutUint32(id, uint32(i))
		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)}
		b, _ := json.Marshal(post)
		pairs = append(pairs, id)
		pairs = append(pairs, b)
		tag := fmt.Sprintf("%s:%08d", strconv.Itoa(i/10), i)
		//store only tags keys
		slowpoke.Set(tags, []byte(tag), nil)
	}
	//store posts fast
	slowpoke.Sets(posts, pairs)

	//get last 2 post key with offset 2
	limit := uint32(2)
	offset := uint32(2)
	order := false //desc
	keys, _ := slowpoke.Keys(posts, nil, limit, offset, order)
	fmt.Println(keys) //[[0 0 0 37] [0 0 0 36]]

	//get key/ values
	res := slowpoke.Gets(posts, keys)
	for k, v := range res {
		if k%2 == 0 {
			fmt.Print(binary.BigEndian.Uint32(v))
		} else {
			var p Post
			json.Unmarshal(v, &p)
			fmt.Println(p)
		}
	}
	//37{37 Content:37 Category:3}
	//36{36 Content:36 Category:3}

	//free from memory
	slowpoke.Close(posts)
	slowpoke.Close(tags)

	//open Db and read tags by prefix 2:* in ascending order
	tagsKeys, _ := slowpoke.Keys(tags, []byte("2:*"), 0, 0, true)
	for _, v := range tagsKeys {
		fmt.Print(string(v) + ", ")
	}
	//2:00000020, 2:00000021, 2:00000022, 2:00000023, 2:00000024, 2:00000025, 2:00000026, 2:00000027, 2:00000028, 2:00000029,
}

Api

All methods are thread-safe.

  • Set/Sets/SetGob

Store val and key. If the file does not exist it will be created.

  • Get/Gets/GetGob

Return the value for the given key or nil and an error. Get will open the database if necessary.

  • Keys

Return keys in ascending/descending order.

With limit and offset.

If from is not nil, return keys lexicographically greater than the from value.

If from ends with asterix *, return keys with the prefix equal to from without the asterix.

Documentation

Status

Used in production (master branch)

Benchmark

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

pogreb goleveldb bolt badgerdb pudge slowpoke pudge(mem)
1M (Put+Get), seconds 187 38 126 34 23 23 2
1M Put, ops/sec 5336 34743 8054 33539 47298 46789 439581
1M Get, ops/sec 1782423 98406 499871 220597 499172 445783 1652069
FileSize,Mb 568 357 552 487 358 358 358

More Repositories

1

pudge

Fast and simple key/value store written using Go's standard library
Go
349
star
2

tgram

typegram: open source publishing platform
Go
336
star
3

freemp

Free Media Player (FreeMp)
Java
102
star
4

malevich

Android displaying bitmaps library
Java
94
star
5

sniper

A simple and efficient thread-safe key/value store for Go
Go
70
star
6

losslessmix

Mixing models of stable diffusion without weights loss
Python
63
star
7

b52

b52 is a fast experimental Key/value database. With support for the memcache protocol.
Go
29
star
8

okdbc

A fast, light-weight key/value store with http & memcache(TCP/UDP) interface.
C
28
star
9

tf

Telefeedbot
Go
15
star
10

itnews

It news
Java
14
star
11

proxyhouse

Collects many small insterts to ClickHouse and send in big inserts
Go
11
star
12

dogenews

Go
11
star
13

2qcache

2Q: A Low Overhead High Performance Buffer Management Replacement Algorith
Java
8
star
14

nimdb

Pudge Db - it's modern key/value storage with memcached protocol support.
C
8
star
15

art

Adaptive Radix Tree done right
Go
7
star
16

sortedset

Package ordset provide ordered set, with strings comparator, backed by arrays
Go
7
star
17

pixel

Simple pixel counter
Go
6
star
18

okdb

Go
6
star
19

train

sdxl train
Python
4
star
20

recoilme

3
star
21

rss2tg

Monitor an rss list by words list and post new links to telegram
Go
3
star
22

graceful

Graceful shutdown, by unignored signals
Go
2
star
23

swinfo

Go
2
star
24

Mielophone

Napster like application
2
star
25

httpproto

http proto parser
Go
1
star
26

p

p
HTML
1
star
27

ogimg

Go
1
star
28

googleplaymusic

googleplaymusic
Go
1
star
29

syncfile

Package syncfile implements concurrent write/read to/from file
Go
1
star
30

boltapi

boltapi
Go
1
star
31

flyml

perfomant mashine learning libraryes in Go
Go
1
star
32

verysmartdog

verysmart.dog
Go
1
star