• Stars
    star
    493
  • Rank 88,702 (Top 2 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created almost 10 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

HOCON parser for Python

pyhocon

pypi Supported Python Versions Build Status Downloads Codacy Badge License Coverage Status Requirements Status

HOCON parser for Python

Specs

https://github.com/typesafehub/config/blob/master/HOCON.md

Installation

It is available on pypi so you can install it as follows:

$ pip install pyhocon

Usage

The parsed config can be seen as a nested dictionary (with types automatically inferred) where values can be accessed using normal dictionary getter (e.g., conf['a']['b'] or using paths like conf['a.b']) or via the methods get, get_int (throws an exception if it is not an int), get_string, get_list, get_float, get_bool, get_config.

from pyhocon import ConfigFactory

conf = ConfigFactory.parse_file('samples/database.conf')
host = conf.get_string('databases.mysql.host')
same_host = conf.get('databases.mysql.host')
same_host = conf['databases.mysql.host']
same_host = conf['databases']['mysql.host']
port = conf['databases.mysql.port']
username = conf['databases']['mysql']['username']
password = conf.get_config('databases')['mysql.password']
password = conf.get('databases.mysql.password', 'default_password') #  use default value if key not found

Example of HOCON file

//
// You can use # or // for comments
//
{
  databases {
    # MySQL
    active = true
    enable_logging = false
    resolver = null
    # you can use substitution with unquoted strings. If it it not found in the document, it defaults to environment variables
    home_dir = ${HOME} # you can substitute with environment variables
    "mysql" = {
      host = "abc.com" # change it
      port = 3306 # default
      username: scott // can use : or =
      password = tiger, // can optionally use a comma
      // number of retries
      retries = 3
    }
  }

  // multi line support
  motd = """
            Hello "man"!
            How is it going?
         """
  // this will be appended to the databases dictionary above
  databases.ips = [
    192.168.0.1 // use white space or comma as separator
    "192.168.0.2" // optional quotes
    192.168.0.3, # can have a trailing , which is ok
  ]

  # you can use substitution with unquoted strings
  retries_msg = You have ${databases.mysql.retries} retries

  # retries message will be overriden if environment variable CUSTOM_MSG is set
  retries_msg = ${?CUSTOM_MSG}
}

// dict merge
data-center-generic = { cluster-size = 6 }
data-center-east = ${data-center-generic} { name = "east" }

// list merge
default-jvm-opts = [-XX:+UseParNewGC]
large-jvm-opts = ${default-jvm-opts} [-Xm16g]

Conversion tool

We provide a conversion tool to convert from HOCON to the JSON, .properties and YAML formats.

usage: tool.py [-h] [-i INPUT] [-o OUTPUT] [-f FORMAT] [-n INDENT] [-v]

pyhocon tool

optional arguments:
  -h, --help                 show this help message and exit
  -i INPUT, --input INPUT    input file
  -o OUTPUT, --output OUTPUT output file
  -c, --compact              compact format
  -f FORMAT, --format FORMAT output format: json, properties, yaml or hocon
  -n INDENT, --indent INDENT indentation step (default is 2)
  -v, --verbosity            increase output verbosity

If -i is omitted, the tool will read from the standard input. If -o is omitted, the result will be written to the standard output. If -c is used, HOCON will use a compact representation for nested dictionaries of one element (e.g., a.b.c = 1)

JSON

$ cat samples/database.conf | pyhocon -f json

{
  "databases": {
    "active": true,
    "enable_logging": false,
    "resolver": null,
    "home_dir": "/Users/darthbear",
    "mysql": {
      "host": "abc.com",
      "port": 3306,
      "username": "scott",
      "password": "tiger",
      "retries": 3
    },
    "ips": [
      "192.168.0.1",
      "192.168.0.2",
      "192.168.0.3"
    ]
  },
  "motd": "\n            Hello \"man\"!\n            How is it going?\n         ",
  "retries_msg": "You have 3 retries"
}

.properties

$ cat samples/database.conf | pyhocon -f properties

databases.active = true
databases.enable_logging = false
databases.home_dir = /Users/darthbear
databases.mysql.host = abc.com
databases.mysql.port = 3306
databases.mysql.username = scott
databases.mysql.password = tiger
databases.mysql.retries = 3
databases.ips.0 = 192.168.0.1
databases.ips.1 = 192.168.0.2
databases.ips.2 = 192.168.0.3
motd = \
            Hello "man"\!\
            How is it going?\

retries_msg = You have 3 retries

YAML

$ cat samples/database.conf | pyhocon -f yaml

databases:
  active: true
  enable_logging: false
  resolver: None
  home_dir: /Users/darthbear
  mysql:
    host: abc.com
    port: 3306
    username: scott
    password: tiger
    retries: 3
  ips:
    - 192.168.0.1
    - 192.168.0.2
    - 192.168.0.3
motd: |

            Hello "man"!
            How is it going?

retries_msg: You have 3 retries

Includes

We support the include semantics using one of the followings:

include "test.conf"
include "http://abc.com/test.conf"
include "https://abc.com/test.conf"
include "file://abc.com/test.conf"
include file("test.conf")
include required(file("test.conf"))
include url("http://abc.com/test.conf")
include url("https://abc.com/test.conf")
include url("file://abc.com/test.conf")
include package("package:assets/test.conf")

When one uses a relative path (e.g., test.conf), we use the same directory as the file that includes the new file as a base directory. If the standard input is used, we use the current directory as a base directory.

For example if we have the following files:

cat.conf:

{
  garfield: {
    say: meow
  }
}

dog.conf:

{
  mutt: {
    say: woof
    hates: {
      garfield: {
        notes: I don't like him
        say: meeeeeeeooooowww
      }
      include "cat.conf"
    }
  }
}

animals.conf:

{
  cat : {
    include "cat.conf"
  }

  dog: {
    include "dog.conf"
  }
}

Then evaluating animals.conf will result in the followings:

$ pyhocon -i samples/animals.conf
{
  "cat": {
    "garfield": {
      "say": "meow"
    }
  },
  "dog": {
    "mutt": {
      "say": "woof",
      "hates": {
        "garfield": {
          "notes": "I don't like him",
          "say": "meow"
        }
      }
    }
  }
}

As you can see, the attributes in cat.conf were merged to the ones in dog.conf. Note that the attribute "say" in dog.conf got overwritten by the one in cat.conf.

Duration/Period support

Difference from HOCON spec

  • nanoseconds supported only in the sense that it is converted to microseconds with lowered accuracy (divided by 1000 and rounded to int).
  • m suffix only applies to minutes. Spec specifies that m can also be applied to months, but that would cause a conflict in syntax.
  • months and years only available if dateutils is installed (relativedelta is used instead of timedelta).

Misc

with_fallback

  • with_fallback: Usage: config3 = config1.with_fallback(config2) or config3 = config1.with_fallback('samples/aws.conf')

from_dict

d = OrderedDict()
d['banana'] = 3
d['apple'] = 4
d['pear'] = 1
d['orange'] = 2
config = ConfigFactory.from_dict(d)
assert config == d

TODO

Items Status
Comments βœ…
Omit root braces βœ…
Key-value separator βœ…
Commas βœ…
Whitespace βœ…
Duplicate keys and object merging βœ…
Unquoted strings βœ…
Multi-line strings βœ…
String value concatenation βœ…
Array concatenation βœ…
Object concatenation βœ…
Arrays without commas βœ…
Path expressions βœ…
Paths as keys βœ…
Substitutions βœ…
Self-referential substitutions βœ…
The += separator βœ…
Includes βœ…
Include semantics: merging βœ…
Include semantics: substitution βœ…
Include semantics: missing files ❌
Include semantics: file formats and extensions ❌
Include semantics: locating resources ❌
Include semantics: preventing cycles ❌
Conversion of numerically-index objects to arrays βœ…
API Recommendations Status
Conversion of numerically-index objects to arrays ❌
Automatic type conversions ❌
Units format ❌
Duration format ❌
Size in bytes format ❌
Config object merging and file merging ❌
Java properties mapping ❌

Contributors

Thanks

More Repositories

1

postgres-aws-s3

aws_s3 postgres extension to import/export data from/to s3 (compatible with aws_s3 extension on AWS RDS)
PLpgSQL
141
star
2

blog-spark-streaming-log-aggregation

Example of use of Spark Streaming with Kafka
Scala
90
star
3

blog-spark-food-recommendation

Simple example on how to use recommenders in Spark / MLlib
Scala
70
star
4

blog-spark-naive-bayes-reuters

Simple example on how to use Naive Bayes on Spark using the popular Reuters 21578 dataset
Scala
23
star
5

python-functional-guide

Small guide for those transitioning from a functional programming language to Python
22
star
6

hive-solr

Hive Storage Handler for SOLR
Java
16
star
7

blog-storm-adnetwork-example

Example of use of Storm for our blog chimpler.wordpress.com
Java
13
star
8

blog-scala-javacv

Scala
13
star
9

async-stream

Async Stream to compress/uncompress gzip, bzip, zstd, parquet, orc
Python
10
star
10

blog-spark-kmeans

Segmenting Audience with KMeans and Voronoi Diagram using Spark and MLlib
Scala
5
star
11

pytcher

Routing web tree framework for python
Python
5
star
12

catdb

Tool to move data around different databases
Python
2
star
13

blog-solr-cloud-example

Example of use of Solr Cloud for our blog chimpler.wordpress.com
Python
2
star
14

tweet-heatmap

tweet-heatmap
Scala
2
star
15

blog-mysql-vertica-mongodb-impala

Comparing MySQL, Vertica, MongoDB and Impala
Shell
2
star
16

asyncstream

Asyncstream with compression support (gzip, snappy, bzip2, zstd, parquet, orc)
Python
1
star
17

libgdx-scala.g8

libgdx scala template (under heavy construction) - NOT WORKING
Shell
1
star
18

simtick

Tick time series codec using delta encoding
Java
1
star