• Stars
    star
    130
  • Rank 277,575 (Top 6 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Pillars for Transactional Systems and Data Grids

Pillars for Transactional Systems and Data Grids

Build Status Latest Version Rust Documentation

Lever is a library for writing transactional systems (esp. for in-memory data). It consists of various parts:

  • index: Indexes and lookup structures
  • stats: Statistics structures
  • sync: Synchronization primitives for transactional systems
  • table: Various KV table kinds backed by transactional algorithms
  • txn: Transactional primitives and management

Lever is using MVCC model to manage concurrency. It supplies building blocks for in-memory data stores for transactional endpoints, databases and systems. Unblocked execution path is main aim for lever while not sacrificing failover mechanisms.

Lever provides STM, lock-free, wait-free synchronization primitives and various other tools to facilitate writing transactional in-memory systems.

Sync

Synchronization primitives which can allow users to write concurrent task structures. Lever don't have runtime or async code. Whole library is based on top of POSIX threads and agnostic IO. That said, these are the few structures which can be used in sync package:

  • Lock-free ReentrantRwLock
  • Spinlocks
  • Fair locks

Tables

Lever's table system can be can be used like this:

use lever::prelude::*;
use std::sync::Arc;

fn main() {
    let lotable: Arc<LOTable<String, u64>> = Arc::new(LOTable::new());

    // RW from 1_000 threads concurrently.
    let thread_count = 1_000;
    let mut threads = vec![];

    for thread_no in 0..thread_count {
        let lotable = lotable.clone();

        let t = std::thread::Builder::new()
            .name(format!("t_{}", thread_no))
            .spawn(move || {
                let key = format!("{}", thread_no);
                lotable.insert(key.clone(), thread_no);
                let _ = lotable.get(&key).unwrap();
            })
            .unwrap();

        threads.push(t);
    }

    for t in threads.into_iter() {
        t.join().unwrap();
    }
}

Mind that Lever comes with MVCC. MVCC is fully implemented with BOCC style for optimistic locking. Lever is under heavy work, other txn resolution algos and concurrency control mechanisms are under active development.

Transaction System

Transaction system has couple of primitives. Which are begin, commit and various methods for management of the transaction over the course of execution.

Example transaction would be like:

use lever::prelude::*;

let mut customers = TVar::new(123_456);

txn.begin(|t| {
    let mut churned = t.read(&customers);
    churned += 1;
    t.write(&mut customers, churned);
});

println!("I have {} customers right now. I gained 1.", customers.get_data());

For more examples please visit examples directory.

Performance

Initial benchmarks show very high throughput for varying workloads.

Workloads are separated in benchmarks like:

  • Pure reads from concurrent 8 threads
  • 80-20 R/RW mixed from concurrent 8 threads
  • Pure writes from concurrent 8 threads

Lever is performant. E.g. Lever's table implementations are doing 25+ million operations under 1,9 seconds. Whole thing is used in production and continuously improved. This crate consolidates plenty of primitives, tools, structures and such. You can try benchmarking yourself. Benchmarking code is included.

Notes for the user

Note that transactions never and ever inherit heavy work in their code path since they are mostly intended for accessing to shared memory and concurrency enabled by their code paths success.

Isolation separated to threads not onto a global memory. That's why it is extremely fast.

Rollbacks are automatic, it won't interfere with your program, or bail. There will be system which incorporates fatal aborts. That work is ongoing.

TODO

  • Fatal aborts
  • Other concurrency schemes
  • Various conflict resolution strategies.
  • Grid communication
  • ...

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

More Repositories

1

tokamak

Fusion Reactor for Rust - Atom Rust IDE
HTML
399
star
2

nuclei

Proactive IO & Runtime system
Rust
249
star
3

orkhon

Orkhon: ML Inference Framework and Server Runtime
Rust
146
star
4

callysto

Stream processing & Service framework.
Rust
112
star
5

kaos

Chaotic Testing Harness
Rust
58
star
6

korq

Kubernetes Dynamic Log Tailing Utility
Rust
42
star
7

cuneiform

Cache & In-Memory optimizations for Rust, revived from the slabs of Sumer.
Rust
30
star
8

proq

Idiomatic Async Prometheus Query (PromQL) Client for Rust.
Rust
20
star
9

rust-dersleri

๐Ÿฆ€Rust Turkiye - Rust Dersleri
Rust
14
star
10

degauss

Avro schema compatibility checker
Rust
14
star
11

tokamak-terminal

Tokamak Terminal
CoffeeScript
11
star
12

smbclient-sys

SMB client(SAMBA impl) FFI bindings [libsmbclient] for Rust
Rust
11
star
13

ringbahn

High performance multiple backend web server
Elixir
7
star
14

cuneiform-fields

Field level cache optimizations for Rust (no_std)
Rust
7
star
15

luaenv

Custom Lua Execution Environments for use with Koneki LDT
Lua
6
star
16

openvas_tasks

Information and history receptor for OpenVAS tasks.
Python
6
star
17

qube

Kubernetes API client with async features (based on kubeclient)
Rust
6
star
18

tracing-json

Tracing Structured Json Logging Adapter
Rust
5
star
19

trafo

Rust rewrite of the util-linux
Rust
4
star
20

kanagawa

Web framework based on Tide with Proactive IO
Rust
4
star
21

olayufku

Schema Registry for BigQuery
JavaScript
2
star
22

relay

Hot-updatable reverse proxy
Rust
2
star
23

airvantage-api-nodejs

Airvantage API Node.JS example
CSS
2
star
24

zor

Properly run Rust benchmarks
Shell
2
star
25

docker-volume-redis

Redis based volume plugin for Docker
Go
2
star
26

travertine

SNMP browser for all platforms (SNMPB fork)
C
2
star
27

leo

Leo - Heuristic Variant Analysis Tool written in Rust
1
star
28

RubMat

discrete math repl
Ruby
1
star
29

cekic

Experimental system model generator for RTS
Java
1
star
30

avu

Kaminsky DNS cache poisoning tool
C++
1
star
31

dark_model

Time travel MDP for Dark Netflix Series (includes protagonists' Season 2 jumps)
1
star
32

volvox

All-in-One developer modes for Emacs
Emacs Lisp
1
star
33

radiant-turkish_language_pack-extension

Radiant CMS Turkish Language Pack Extension
Ruby
1
star
34

x86instexporter

Intel x86/x86_64 instruction set exporter (to ease the pain of rust-lang/stdarch contributions)
Python
1
star
35

async-process-bench

Rust
1
star
36

turnusol

Easy to use framework for SBCs.(Single-Board Computers)
Erlang
1
star
37

latch

Latch iz eksperimental 1k/4k kompilation syztem
Rust
1
star
38

nuclei-attributes

Attributes for Nuclei
Rust
1
star