• Stars
    star
    240
  • Rank 161,889 (Top 4 %)
  • Language
    Elixir
  • License
    MIT License
  • Created over 7 years ago
  • Updated almost 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Ecto adapter for Mnesia Erlang term database.

Ecto adapter for Mnesia Erlang term database

Hex.pm Downloads Latest Version License Build Status Coverage Status Ebert

Ecto 2.X adapter for Mnesia Erlang term database. In most cases it can be used as drop-in replacement for other adapters.

Supported features:

  • Compatible Ecto.Repo API.
  • Automatically converts Ecto.Query structs to Erlang match_spec.
  • Emulated query.select and query.order_bys, select .. in [..]. (Emulating is slow for any large dataset, O(n * log n).)
  • Auto-generated (via sequence table) :id primary keys.
  • Migrations and database setup via Ecto.Migrations.
  • Transactions.
  • Secondary indexes.

Planned features:

  • Native primary key and unique index constraints.
  • Custom primary keys.
  • Other transactional contexts.

Not supported features (create issue and vote if you need them):

  • Type casting. Mnesia can store any data in any field, including strings, numbers, atoms, tuples, floats or even PID's. All types in your migrations will be silently ignored.
  • Mnesia clustering and auto-clustering.
  • Lookups in json fields.
  • Schemaless queries.
  • Composite primary keys.
  • Unique/all other constraints (including associations).
  • JOINs.
  • min, max, avg and other aggregation functions.
  • Intervals.

In general. This adapter is still not passing all Ecto integration tests and in active development. But it already can be helpful in simple use-cases.

Why Mnesia?

We have a production task that needs low read-latency database and our data fits in RAM, so Mnesia is the best choice: it's part of OTP, shares same space as our app does, work fast in RAM and supports transactions (it's critical for fintech projects).

Why do we need an adapter? We don't want to lock us to any specific database, since requirements can change. Ecto allows to switch databases by simply modifying the config, and we might want to go back to Postres or another DB.

Clustering and using Mnesia for your project

If you use Mnesia - you either get a distributed system from day one or a single node with low availability. Very few people really want any of that options. Specifically Mnesia it's neither an AP, nor a CP database; requires you to handle network partitions (split brains) manually; has much less documentation available compared to a more common databases (like PostgreSQL).

Please, pick your tools wisely and think through how you would use them in production.

Mnesia configuration from config.exs

config :ecto_mnesia,
  host: {:system, :atom, "MNESIA_HOST", Kernel.node()},
  storage_type: {:system, :atom, "MNESIA_STORAGE_TYPE", :disc_copies}

config :mnesia,
  dir: 'priv/data/mnesia' # Make sure this directory exists

Notice that {:system, [TYPE], ENV_NAME, default_value} tuples can be replaced with any raw values.

They tell adapter to read configuration from environment in run-time, so you will be able to set MNESIA_HOST and MNESIA_STORAGE_TYPE environment variables, which is very useful when you releasing app in production and don't want to rebuild all code on each config change.

If you want to know more how this tool works take look at Confex package.

Storage Types

  • :disc_copies - store data in both RAM and on disc. Recommended value for most cases.
  • :ram_copies - store data only in RAM. Data will be lost on node restart. Useful when working with large datasets that don't need to be persisted.
  • :disc_only_copies - store data only on disc. This will limit database size to 2GB and affect adapter performance.

Table Types (Engines)

In migrations you can select which kind of table you want to use:

create_if_not_exists table(:my_table, engine: :set) do
  # ...
end

Supported types:

  • :set - expected your records to have at least one unique primary key that should be in first column.
  • :ordered_set - default type. Same as :set, but Mnesia will store data in a table will be ordered by primary key.
  • :bag - expected all records to be unique, but no primary key is required. (Internally, it will use first field as a primary key).
Ordered Set Performance

Ordered set comes in a cost of increased complexity of write operations:

Set

Operation Average Worst Case
Space O(n) O(n)
Search O(1) O(n)
Insert O(1) O(n)
Delete O(1) O(n)

Ordered Set

Operation Average Worst Case
Space O(n) O(n)
Search O(log n) O(n)
Insert O(log n) O(n)
Delete O(log n) O(n)

Installation

It is available in Hex, the package can be installed as:

  1. Add ecto_mnesia to your list of dependencies in mix.exs:
def deps do
  [{:ecto_mnesia, "~> 0.9.0"}]
end
  1. Ensure ecto_mnesia is started before your application:
def application do
  [applications: [:ecto_mnesia]]
end
  1. Use EctoMnesia.Adapter as your Ecto.Repo adapter:
config :my_app, MyRepo,
  adapter: EctoMnesia.Adapter
  1. Optionally set custom Mnesia data dir (don't forget to create it):
config :mnesia, :dir, 'priv/data/mnesia'

The docs can be found at https://hexdocs.pm/ecto_mnesia.

Thanks

We want to thank meh for his Amnesia package that helped a lot in initial Mnesia investigations. Some pieces of code was copied from his repo.

Also big thanks to josevalim for Elixir, Ecto and active help while this adapter was developed.

More Repositories

1

sage

A dependency-free tool to run distributed transactions in Elixir, inspired by Sagas pattern.
Elixir
900
star
2

annon.api

Configurable API gateway that acts as a reverse proxy with a plugin system.
Elixir
329
star
3

confex

Useful helper to read and use application configuration from environment variables.
Elixir
295
star
4

logger_json

JSON console backend for Elixir Logger.
Elixir
212
star
5

gandalf.api

Open-Source Decision Engine and Scoring Table for Big-Data.
PHP
98
star
6

multiverse

Elixir package that allows to add compatibility layers via API gateways.
Elixir
93
star
7

ecto_trail

EctoTrail allows to store Ecto changeset changes in a separate audit_log table.
Elixir
54
star
8

gandalf.web

Open-Source Decision Engine and Scoring Table for Big-Data.
JavaScript
45
star
9

renew

Mix task to create mix projects that builds into Docker containers.
Elixir
33
star
10

mouth

Simple adapter based SMS sending library
Elixir
29
star
11

rbmq

Simple API for spawning RabbitMQ Producers and Consumers.
Elixir
22
star
12

gen_task

Generic Task behavior that helps encapsulate errors and recover from them in classic GenStage workers.
Elixir
22
star
13

k8s-utils

Kubernetes utils for debugging our development or production environments
Shell
20
star
14

ael.api

Media content storage access control system based on Signed URL's.
Elixir
18
star
15

ecto_paging

Cursor-based pagination for Ecto.
Elixir
14
star
16

annon.web

Annon API Gateway Dashboard - manage API Gateway settings, review and replay requests from history.
JavaScript
11
star
17

bsoneach

Elixir package that applies a function to each document in a BSON file.
Elixir
9
star
18

jvalid

Json Scheme validation helper, that allows to store schemes in a separate files.
Elixir
8
star
19

man.api

Template Rendering Engine as a Service
Elixir
6
star
20

annon.infra

Infrastructure helpers for Annon API Gateway.
Shell
6
star
21

postboy.api

Asynchronous delivery service for Email or SMS messages.
Elixir
4
star
22

alpine-cassandra

Cassandra in Alpine Linux box.
Shell
4
star
23

eview

Nebo #15 Views for our Elixir API applications.
Elixir
4
star
24

mithril.api

Authentication and role management service.
Elixir
4
star
25

react-nebo15-events

Event Manager for application on React JS
JavaScript
4
star
26

lumen-intercom

Intercom service for lumen
PHP
3
star
27

annon.ktl

`annonktl` is a Annon API Gateway management CLI.
Elixir
3
star
28

man.web

MΓ‘n Templates Rendering Service
JavaScript
3
star
29

react-nebo15-validate

Validation module for React application by Nebo15
JavaScript
3
star
30

alpine-php

Linux Alpine container with PHP 5.6/7 and Mongo driver.
Shell
2
star
31

lumen-mandrill

PHP
2
star
32

alpine-erlang

Erlang container based on Alpine Linux with Kubernetes support.
Dockerfile
2
star
33

alpine-elixir

Alpine Box with Elixir and Erlang installed.
Dockerfile
2
star
34

react-nebo15-components

React components
JavaScript
2
star
35

alpine-postgre

PostgreSQL Docker Images based on Alpine Linux and with the same API as official repo has.
Shell
2
star
36

annon.status.web

Annon API Gateway Status Page - see statuses of the APIs from Annon API Gateway.
JavaScript
1
star
37

egndf

gndf.io client for Elixir
Elixir
1
star
38

d3-structure

🐨 Build your d3-graphics with data structures and one entrypoint!
JavaScript
1
star
39

mbank.lib.angular-compiled

Compiled version for Angular MBank libruary
JavaScript
1
star
40

nebo-error-messages

JavaScript
1
star
41

alpine-mongodb

Alpine container with MongoDB
Shell
1
star
42

react-boilerplate

Boilerplate for React Univeral application, that is using Redux, React-router and Express
JavaScript
1
star
43

d3-builder

JavaScript
1
star
44

gulp-by-path

Transform files grouped by file.relative
JavaScript
1
star
45

lumen-rest

PHP
1
star
46

nebo-localize

JavaScript
1
star
47

drunken-russian

Tasks manager for PHP and MongoDB. 100% alcohol free.
PHP
1
star
48

oxygen

Deprecated. Use Elixir Registry instead of this package.
Elixir
1
star
49

url-parser

URL Parser module
JavaScript
1
star
50

alpine-fluentd-elasticsearch

Alpine Box with Fluentd and ElasticSearch plugin
1
star
51

nebo-validate

JavaScript
1
star
52

onedayofmine.web

OneDayOfMine Project - it's our old and first unsupported startup, which we want to re-implement in future
PHP
1
star
53

javascript-library-boilerplate

Javascript Library Boilerplate
1
star
54

react-nebo15-currency-input

React currency input by Nebo15
JavaScript
1
star
55

credits4all.web.admin

Playground based on Parse.com
CSS
1
star
56

tokenizer.api

Card Tokenization Service
Elixir
1
star
57

dogstat

Elixir client for StatsD servers.
Elixir
1
star
58

alpine-postgre-backup

Backup system for PostgreSQL containers.
Shell
1
star
59

ci-utils

Continuous Integration scripts for different languages
1
star