• Stars
    star
    196
  • Rank 198,553 (Top 4 %)
  • Language
    Elixir
  • License
    Apache License 2.0
  • Created over 6 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An implementation of TOML for Elixir projects, compliant with the latest specification

TOML for Elixir

Main Hex.pm Version Hex Docs Total Download Last Updated

This is a TOML library for Elixir projects.

It is compliant with version 1.0 of the official TOML specification. You can find a brief overview of the feature set below, but you are encouraged to read the full spec at the link above (it is short and easy to read!).

Features

  • Decode from string, file, or stream
  • Fully compliant with the latest version of the TOML spec
  • Is tested against toml-test, a test suite for spec-compliant TOML encoders/decoders, used by implementations in multiple languages. The test suite has been integrated into this project to be run under Mix so that we get better error information and so it can run as part of the test suite.
  • Decoder produces a map with values using the appropriate Elixir data types for representation
  • Supports extension via value transformers (see Toml.Transform docs for details)
  • Supports use as a configuration provider in Distillery 2.x+ (use TOML files for configuration!)
  • Decoder is written by hand to take advantage of various optimizations.
  • Library passes Dialyzer checks

Comparison To Other Libraries

I compared toml to four other libraries:

  • toml_elixir
  • tomlex
  • jerry
  • etoml

Of these four, none correctly implement the 0.5.0 specification. Either they are targeting older versions of the spec (in etoml, it is built against pre-0.1), are not fully implemented (i.e. don't support all features), or have bugs which prevent them from properly parsing a 0.5.0 example file (the test/fixtures/example.toml file in this repository).

If you are looking for a TOML library, at present toml is the only one which full implements the spec and correctly decodes example.toml.

Installation

This library is available on Hex as :toml, and can be added to your deps like so:

def deps do
  [
    {:toml, "~> 0.7"}
  ]
end

NOTE: You can determine the latest version on Hex by running mix hex.info toml.

Type Conversions

In case you are curious how TOML types are translated to Elixir types, the following table provides the conversions.

NOTE: The various possible representations of each type, such as hex/octal/binary integers, quoted/literal strings, etc., are considered to be the same base type (e.g. integer and string respectively in the examples given).

TOML Elixir
String String.t (binary)
Integer integer
inf :infinity
+inf :infinity
-inf :negative_infinity
nan :nan
+nan :nan
-nan :negative_nan
Boolean boolean
Offset Date-Time DateTime.t
Local Date-Time NaiveDateTime.t
Local Date Date.t
Local Time Time.t
Array list
Table map
Table Array list(map)

Implementation-specific Behaviors

Certain features of TOML have implementation-specific behavior:

  • -inf, inf, and +inf are all valid infinity values in TOML. In Erlang/Elixir, these don't have exact representations. Instead, by convention, :infinity is used for positive infinity, as atoms are always larger than integers when using comparison operators, so :infinity > <any integer> will always be true. However, negative infinity cannot be represented, as numbers are always considered smaller than every other type in term comparisons. Instead, we represent it with :negative_infinity, so that the type information is not lost, but you must be careful to deal with it specifically in comparisons/sorting/etc.
  • -nan, nan, and +nan are all valid NaN (not a number) values in TOML. In Erlang/Elixir, NaN is traditionally represented with :nan, but there is no representation for negative NaN, and no API actually produces :nan, instead invalid numbers typically raise errors, in the typical spirit of "let it crash" in the face of errors. For purposes of preserving type information though, we use the :nan convention, and :negative_nan for -NaN. You will need to take care to deal with these values manually if the values need to be preserved.
  • The maximum precision of times in the various time types is microseconds (i.e. precision to six decimal places), if you provide higher precision values (i.e. nanoseconds), the extra precision will be lost.
  • Hex, octal, and binary numbers are converted to integers, so serializing those values after decoding them from a TOML document will be in their decimal representation.

Example Usage

The following is a brief overview of how to use this library. First, let's take a look at an example TOML file, as borrowed from the TOML homepage:

# This is a TOML document.

title = "TOML Example"

[owner]
name = "Tom Preston-Werner"
dob = 1979-05-27T07:32:00-08:00 # First class dates

[database]
server = "192.168.1.1"
ports = [ 8001, 8001, 8002 ]
connection_max = 5000
enabled = true

[servers]

  # Indentation (tabs and/or spaces) is allowed but not required
  [servers.alpha]
  ip = "10.0.0.1"
  dc = "eqdc10"

  [servers.beta]
  ip = "10.0.0.2"
  dc = "eqdc10"

[clients]
data = [ ["gamma", "delta"], [1, 2] ]

# Line breaks are OK when inside arrays
hosts = [
  "alpha",
  "omega"
]

Parsing

iex> input = """
[database]
server = "192.168.1.1"
"""
...> {:ok, %{"database" => %{"server" => "192.168.1.1"}}} = Toml.decode(input)
...> {:ok, %{database: %{server: "192.168.1.1"}}} = Toml.decode(input, keys: :atoms)
...> stream = File.stream!("example.toml")
...> {:ok, %{"database" => %{"server" => "192.168.1.1"}}} = Toml.decode_stream(stream)
...> {:ok, %{"database" => %{"server" => "192.168.1.1"}}} = Toml.decode_file("example.toml")
...> invalid = """
[invalid]
a = 1 b = 2
"""
...> {:error, {:invalid_toml, reason}} = Toml.decode(invalid); IO.puts(reason)
expected '\n', but got 'b' in nofile on line 2:

    a = 1 b = 2
         ^

:ok

Transforms

Support for extending value conversions is provided by the Toml.Transform behavior. An example is shown below:

Given the following TOML document:

[servers.alpha]
ip = "192.168.1.1"
ports = [8080, 8081]

[servers.beta]
ip = "192.168.1.2"
ports = [8082, 8083]

And the following modules:

defmodule Server do
  defstruct [:name, :ip, :ports]
end

defmodule IPStringToCharlist do
  use Toml.Transform
  
  def transform(:ip, v) when is_binary(v) do
    String.to_charlist(v)
  end
  def transform(_k, v), do: v
end

defmodule CharlistToIP do
  use Toml.Transform
  
  def transform(:ip, v) when is_list(v) do
    case :inet.parse_ipv4_address(v) do
      {:ok, address} ->
        address
      {:error, reason} ->
        {:error, {:invalid_ip_address, reason}}
    end
  end
  def transform(:ip, v), do: {:error, {:invalid_ip_address, v}}
  def transform(_k, v), do: v
end

defmodule ServerMapToList do
  use Toml.Transform
  
  def transform(:servers, v) when is_map(v) do
    for {name, server} <- v, do: struct(Server, Map.put(server, :name, name))
  end
  def transform(_k, v), do: v
end

You can convert the TOML document to a more strongly-typed version using the above transforms like so:

iex> transforms = [IPStringToCharlist, CharlistToIP, ServerMapToList]
...> {:ok, result} = Toml.decode("example.toml", keys: :atoms, transforms: transforms)
%{servers: [%Server{name: :alpha, ip: {192,168,1,1}}, ports: [8080, 8081] | _]}

The transforms given here are intended to show how they can be composed: they are applied in the order provided, and the document is transformed using a depth-first, bottom-up traversal. Put another way, you transform the leaves of the tree before the branches; as shown in the example above, this means the :ip key is converted to an address tuple before the :servers key is transformed into a list of Server structs.

Using with Elixir Releases (1.9+)

To use this library as a configuration provider in Elixir, use the following example of how one might use it in their release configuration, and tailor it to your needs:

config_providers: [
  {Toml.Provider, [
    path: {:system, "XDG_CONFIG_DIR", "myapp.toml"},
    transforms: [...]
  ]}
]

See the "Using as a Config Provider" section for more info.

Using with Distillery

Like the above, use the following example as a guideline for how you use this in your own release configuration (i.e. in rel/config.exs):

release :myapp do
  # ...snip...
  set config_providers: [
    {Toml.Provider, [path: "${XDG_CONFIG_DIR}/myapp.toml", transforms: [...]]}
  ]
end

Using as a Config Provider

The usages described above will result in Toml.Provider being invoked during boot, at which point it will evaluate the given path and read the TOML file it finds. If one is not found, or is not accessible, the provider will raise an error, and the boot sequence will terminate unsuccessfully. If it succeeds, it persists settings in the file to the application environment (i.e. you access it via Application.get_env/2).

You can pass the same options in the arguments list for Toml.Provider as you can to Toml.decode/2, but :path is required, and :keys only supports :atoms and :atoms! values.

The config provider expects a certain format to the TOML file, namely that keys at the root of the document correspond to applications which need to be configured. If it encounters keys at the root of the document which are not tables, they are ignored.

# This is an example of something that would be ignored
title = "My config file"

# We're expecting something like this:
[myapp]
key = "value"

# To use a bit of Phoenix config, you translate to TOML like so:
[myapp."MyApp.Endpoint"]
cache_static_manifest = "priv/static/cache_manifest.json"

[myapp."MyApp.Endpoint".http]
port = "4000"

[myapp."MyApp.Endpoint".force_ssl]
hsts = true

# Or logger..
[logger]
level = "info"

[logger.console]
format = "[$level] $message \n"

Roadmap

  • Add benchmarking suite
  • Provide options for converting keys to atom, similar to Jason/Poison/etc.
  • Optimize lexer to always send offsets to decoder, rather than only in some cases
  • Try to find pathological TOML files to test

License

This project is licensed Apache 2.0, see the LICENSE file in this repo for details.

More Repositories

1

distillery

Simplify deployments in Elixir with OTP releases!
Elixir
2,944
star
2

timex

A complete date/time library for Elixir projects.
Elixir
1,732
star
3

swarm

Easy clustering, registration, and distribution of worker processes for Erlang/Elixir
Elixir
1,201
star
4

exrm

Automatically generate a release for your Elixir project!
Elixir
923
star
5

exprotobuf

Protocol Buffers in Elixir made easy!
Elixir
485
star
6

libgraph

A graph data structure library for Elixir projects
Elixir
454
star
7

conform

Easy, powerful, and extendable configuration tooling for releases.
Elixir
378
star
8

keys.js

Easy keybindings for browser applications!
JavaScript
360
star
9

alpine-elixir-phoenix

An Alpine Linux base image containing Elixir, Erlang, Node, Hex, and Rebar. Ready for Phoenix applications!
Makefile
349
star
10

alpine-elixir

A Dockerfile based on my alpine-erlang image for Elixir applications
Makefile
202
star
11

combine

A parser combinator library for Elixir projects
Elixir
194
star
12

libring

A fast consistent hash ring implementation in Elixir
Elixir
192
star
13

timex_ecto

An adapter for using Timex DateTimes with Ecto
Elixir
161
star
14

exirc

IRC client adapter for Elixir projects
Elixir
152
star
15

artificery

A toolkit for creating terminal user interfaces in Elixir
Elixir
121
star
16

alpine-erlang

An alpine image with Erlang installed, intended for releases
Dockerfile
82
star
17

strukt

Extends defstruct with schemas, changeset validation, and more
Elixir
71
star
18

ex_unit_clustered_case

An extension for ExUnit for simplifying tests against a clustered application
Elixir
57
star
19

uniq

Provides UUID generation, parsing, and formatting. Supports RFC 4122, and the v6 draft extension
Elixir
52
star
20

distillery-aws-example

An example application to go with the AWS guide in the Distillery documentation
Elixir
50
star
21

distillery-umbrella-test

An example Elixir application for working with umbella apps and Distillery.
Elixir
49
star
22

pluginhost

An example C# application which provides runtime extensibility via plugins
C#
31
star
23

picosat_elixir

Elixir + Erlang bindings for the PicoSAT solver
C
17
star
24

docker-release-toolkit

My personal toolkit for building releases with Docker
Makefile
16
star
25

distillery-test

Elixir application which demonstrates a bare-minimum release-ready app using Distillery.
Elixir
16
star
26

8bit-background

A sweet series of 8-bit backgrounds, which changes based on the time of day.
Shell
15
star
27

libswagger

A Swagger client library for Elixir projects
Elixir
15
star
28

stringex

A string extensions library for node.js
JavaScript
15
star
29

scaladiff

A diff library built in Scala, for Scala projects
Scala
13
star
30

dotfiles

My assorted dotfiles
Shell
10
star
31

resume

The TeX source for my current resume.
TeX
7
star
32

fogbugz-cli

FogBugz Command Line Client
Ruby
7
star
33

aria

An experiment in programming language design
7
star
34

s2i-alpine-base

An S2I base image using Alpine Linux
Python
5
star
35

RPNCalculator

Reverse Polish Notation calculator built in Scala for a impromptu code challenge at work
Scala
5
star
36

uniq_compat

A compatibility shim for ::elixir_uuid when used with :uniq
Elixir
4
star
37

aws-dist-test

Clone of distillery-aws-example to illustrate distribution
Elixir
4
star
38

centos7-elixir

A CentOS7 base image for use with the Distillery AWS guide
Dockerfile
4
star
39

firefly

TBD
Elixir
3
star
40

conform_exrm

Conform plugin for ExRM
Elixir
3
star
41

alpine-toolbox

A toolbox image based on Alpine Linux for when I want to troubleshoot things in my OpenShift cluster
Makefile
3
star
42

SharpTools

A collection of practical C# code to build upon
C#
2
star
43

functools

Functional programming tools for Go
Go
2
star
44

toolbox.js

A general purpose javascript utility library. Contains everything you need, and most anything worth having (other than DOM manipulation).
JavaScript
1
star
45

erl_tar2

A re-implementation of erl_tar to support common tar archive formats
Erlang
1
star
46

s2i-elixir

An S2I image which provides Elixir, Erlang, and Node.js
Shell
1
star
47

blog

My personal blog source
CSS
1
star
48

s2i-erlang

An S2I image which provides Erlang and Node.js
Shell
1
star