Lightning Memory Database (LMDB) for ZIO

Why ZIO-lmdb ? Because I wanted a very simple embedded (in the same process) ACID database for small applications while keeping deployment, maintenance, upgrades as simple as possible.

ZIO-lmdb is based on the powerful lmdb-java library and bring a higher level API in order to enhance the developer experience.

So ZIO-lmdb is an embedded key/value database, with an easy to use opinionated API, choices have been made to make the developer experience as simple as possible :

• JSON based storage using zio-json,
• safe update by using a lambda which will be called with the previous value if it exists and returns the new value,
• identifiers are managed by the developer, just use UUID or ULID.

API is designed to not lie, all functions signatures describe precisely what you must expect from them, thanks to ZIO and Scala3.

Definitions

For a better understanding, this library use a slightly different vocabulary from LMDB original one :

• Database : (LMDB talk about Environment)
  • The place where the database file is stored on your file system
  • A set of configuration for this database (expected maximum size, expected collection number)
• Collection : (LMDB talk about Database)
  • A sorted map where to store your data
  • One database contains multiple collection
• Transaction : (the same for LMDB)
  • for global coherency within the same database
  • only one simultaneous write access is possible within the same database

Configuration

Configuration is based on the standard ZIO config mechanism, the default configuration provider uses environnment variables or java properties to resolve this library configuration parameters.

Usages example

//> using scala  "3.3.0"
//> using dep "fr.janalyse::zio-lmdb:1.3.0"
//> using javaOpt "--add-opens", "java.base/java.nio=ALL-UNNAMED", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED"

import zio.*, zio.lmdb.*, zio.json.*
import java.io.File, java.util.UUID, java.time.OffsetDateTime

case class Record(uuid: UUID,name: String,age: Int,addedOn: OffsetDateTime) derives JsonCodec

object SimpleExample extends ZIOAppDefault {
  override def run = example.provide(LMDB.liveWithDatabaseName("lmdb-data-simple-example"), zio.Scope.default)

  val collectionName = "examples"
  val example        = for {
    _         <- LMDB.collectionCreate[Record](collectionName).ignore
    examples  <- LMDB.collectionGet[Record](collectionName)
    recordId  <- Random.nextUUID
    dateTime  <- Clock.currentDateTime
    record     = Record(recordId, "John Doe", 42, dateTime)
    _         <- examples.upsertOverwrite(recordId.toString, record)
    gotten    <- examples.fetch(recordId.toString).some
    collected <- examples.collect()
    _         <- Console.printLine(s"collection $collectionName contains ${collected.size} records")
    _         <- ZIO.foreach(collected)(record => Console.printLine(record))
    lmdb      <- ZIO.service[LMDB]
    _         <- Console.printLine("""LMDB standard tools can be used to manage the database content : sudo apt-get install lmdb-utils""")
    _         <- Console.printLine(s"""To get some statistics     : mdb_stat -s $collectionName ${lmdb.databasePath}/""")
    _         <- Console.printLine(s"""To dump collection content : mdb_dump -p -s $collectionName ${lmdb.databasePath}/""")
  } yield ()
}

SimpleExample.main(Array.empty)

To run the previous logic, you'll have to provide the LMDB layer, two layers are available :

• LMDB.live : Fully configurable using standard zio-config
• LMDB.liveWithDatabaseName("chosen-database-name") : to override/force the database name (quite useful when writing scala scripts)

ZIO-LMDB based Applications

• sotohp - photos management which uses zio-lmdb intensively
• code-examples-manager - snippets/gists management lmdb used for caching and data sharing
• zwords - wordle like game which can be played zwords game

Code snippets using ZIO-LMDB, runnable with scala-cli

• ZIO LMDB simple example (scala-3)
• ZIO LMDB simple example (scala-2)
• ZIO LMDB feeding with French town postal codes
• ZIO LMDB using custom configuration provider
• Extract photos records from elasticsearch and save them into LMDB
• Export code examples and executions results from lmdb to elastisearch

Operating lmdb databases

LMDB standard tools can be used to manage the databases content : sudo apt-get install lmdb-utils

• to get some database statistics : mdb_stat -a database_directory_path/
• to dump the content of a database : mdb_dump -a -p database_directory_path/
• to dump the content of a database collection : mdb_dump -s collectionName -p database_directory_path/
• to restore some collection or the entire database use the command named mdb_load which uses the same format as for mdb_dump

As zio-lmdb is using json format, dumps are just text, which can be edited and then loaded back. So simple data migration is straightforward.

Requirements

When LVMDB is used as persistence store with recent JVM it requires some JVM options :

--add-opens java.base/java.nio=ALL-UNNAMED
--add-opens java.base/sun.nio.ch=ALL-UNNAMED

Contributors :)

• François Armand : for scala 2.13 support initiative