• Stars
    star
    278
  • Rank 148,454 (Top 3 %)
  • Language
    Python
  • License
    MIT License
  • Created over 12 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Python library provides common APIs to load and dump configuration files in various formats

python-anyconfig

[Latest Version] [Python versions] MIT License [Github Actions: Test status] [Coverage Status] [Codecov Status] [Code Quality by Scrutinizer] [Code Quality by LGTM] [Doc Status]

Introduction

python-anyconfig [1] is a python library provides common APIs to load and dump configuration files in various formats with some useful features such as contents merge, templates, query, schema validation and generation support.

[1]This name took an example from the 'anydbm' python standard library.

Features

python-anyconfig provides very simple and unified APIs to process configuration files in various formats and related functions:

  • Loading configuration files:

    anyconfig.load (path_specs, ac_parser=None, ac_dict=None, ac_template=False, ac_context=None, **options)

    loads configuration data from path_specs. path_specs may be a list of file paths, files or file-like objects, ~pathlib.Path class object, a namedtuple ~anyconfig.globals.IOInfo objects represents some inputs to load data from, and return a dict or dict like object, or a primitive types' data other than dict represents loaded configuration.

    anyconfig.loads (content, ac_parser=None, ac_dict=None, ac_template=False, ac_context=None, **options)

    loads configuration data from a string just like json.loads does.

  • Dumping configuration files:

    anyconfig.dump (data, out, ac_parser=None, **options)

    dumps a configuration data data in given format to the output out, may be a file, file like object.

    anyconfig.dumps (data, ac_parser=None, **options)

    dumps a configuration data loaded from a string

  • Open configuration files:

    anyconfig.open (path, mode=None, ac_parser=None, **options)

    open configuration files with suitable flags and return file/file-like objects, and this object can be passed to the anyconfig.load().

  • Merge dicts:

    anyconfig.merge (self, other, ac_merge=MS_DICTS, **options)

    Update (merge) a mapping object 'self' with other mapping object 'other' or an iterable 'other' yields (key, value) tuples according to merge strategy 'ac_merge'.

  • Schema validation and generation of configuration files:

    anyconfig.validate (data, schema, ac_schema_safe=True, ac_schema_errors=False, **options)

    validates configuration data loaded with anyconfig.load() with JSON schema [2] object also loaded with anyconfig.load(). anyconfig.load() may help loading JSON schema file[s] in any formats anyconfig supports.

    anyconfig.gen_schema (data, **options)

    generates a mapping object represents a minimum JSON schema to validate configuration data later. This result object can be serialized to any formats including JSON with anyconfig.dump or anyconfig.dumps.

It enables to load configuration file[s] in various formats in the same manner, and in some cases, even there is no need to take care of the actual format of configuration file[s] like the followings:

import anyconfig

# Config type (format) is automatically detected by filename (file
# extension) in some cases.
conf1 = anyconfig.load("/path/to/foo/conf.d/a.yml")

# Similar to the above but the input is pathlib.Path object.
import pathlib
path_1 = pathlib.Path("/path/to/foo/conf.d/a.yml")
conf1_1 = anyconfig.load(path_1)

# Similar to the first one but load from file object opened:
with anyconfig.open("/path/to/foo/conf.d/a.yml") as fileobj:
    conf1_2 = anyconfig.load(fileobj)

# Loaded config data is a mapping object, for example:
#
#   conf1["a"] => 1
#   conf1["b"]["b1"] => "xyz"
#   conf1["c"]["c1"]["c13"] => [1, 2, 3]

# Or you can specify the format (config type) explicitly if its automatic
# detection may not work.
conf2 = anyconfig.load("/path/to/foo/conf.d/b.conf", ac_parser="yaml")

# Likewise.
with anyconfig.open("/path/to/foo/conf.d/b.conf") as fileobj:
    conf2_2 = anyconfig.load(fileobj, ac_parser="yaml")

# Specify multiple config files by the list of paths. Configurations of each
# files will be merged.
conf3 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"])

# Similar to the above but all or one of config file[s] might be missing.
conf4 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"],
                       ac_ignore_missing=True)

# Specify config files by glob path pattern:
conf5 = anyconfig.load("/etc/foo.d/*.json")

# Similar to the above, but parameters in the former config file will be simply
# overwritten by the later ones instead of merge:
conf6 = anyconfig.load("/etc/foo.d/*.json", ac_merge=anyconfig.MS_REPLACE)

Also, it can process configuration files which are jinja2-based template files:

  • Enables to load a substantial configuration rendered from half-baked configuration template files with given context
  • Enables to load a series of configuration files indirectly 'include'-d from a/some configuration file[s] with using jinja2's 'include' directive.
In [1]: import anyconfig

In [2]: open("/tmp/a.yml", 'w').write("a: {{ a|default('aaa') }}\n")

In [3]: anyconfig.load("/tmp/a.yml", ac_template=True)
Out[3]: {'a': 'aaa'}

In [4]: anyconfig.load("/tmp/a.yml", ac_template=True, ac_context=dict(a='bbb'))
Out[4]: {'a': 'bbb'}

In [5]: open("/tmp/b.yml", 'w').write("{% include 'a.yml' %}\n")  # 'include'

In [6]: anyconfig.load("/tmp/b.yml", ac_template=True, ac_context=dict(a='ccc'))
Out[6]: {'a': 'ccc'}

And python-anyconfig enables to validate configuration files in various formats with using JSON schema like the followings:

# Validate a JSON config file (conf.json) with JSON schema (schema.yaml).
# If validatation succeeds, `rc` -> True, `err` -> ''.
conf1 = anyconfig.load("/path/to/conf.json")
schema1 = anyconfig.load("/path/to/schema.yaml")
(rc, err) = anyconfig.validate(conf1, schema1)  # err is empty if success, rc == 0

# Validate a config file (conf.yml) with JSON schema (schema.yml) while
# loading the config file.
conf2 = anyconfig.load("/a/b/c/conf.yml", ac_schema="/c/d/e/schema.yml")

# Validate config loaded from multiple config files with JSON schema
# (schema.json) while loading them.
conf3 = anyconfig.load("conf.d/*.yml", ac_schema="/c/d/e/schema.json")

# Generate jsonschema object from config files loaded and get string
# representation.
conf4 = anyconfig.load("conf.d/*.yml")
scm4 = anyconfig.gen_schema(conf4)
scm4_s = anyconfig.dumps(scm4, "json")

And you can query loaded data with JMESPath [3] expressions:

In [2]: dic = dict(a=dict(b=[dict(c="C", d=0)]))

In [3]: anyconfig.loads(anyconfig.dumps(dic, ac_parser="json"),
   ...:                 ac_parser="json", ac_query="a.b[0].c")
Out[3]: u'C'

In [4]:

And in the last place, python-anyconfig provides a CLI tool called anyconfig_cli to process configuration files and:

  • Convert a/multiple configuration file[s] to another configuration files in different format
  • Get configuration value in a/multiple configuration file[s]
  • Validate configuration file[s] with JSON schema
  • Generate minimum JSON schema file to validate given configuration file[s]
[2]http://json-schema.org
[3]http://jmespath.org

Supported configuration formats

python-anyconfig supports various file formats if requirements are satisfied and backends in charge are enabled and ready to use:

  • Always supported formats of which backends are enabled by default:
Always supported formats
Format Type Requirement
JSON json json (standard lib) or simplejson [4]
Ini-like ini configparser (standard lib)
Pickle pickle pickle (standard lib)
XML xml ElementTree (standard lib)
Java properties [5] properties None (native implementation with standard lib)
B-sh shellvars None (native implementation with standard lib)
  • Supported formats of which backends are enabled automatically if requirements are satisfied:
Supported formarts if requirements are satisfied
Format Type Requirement
YAML yaml ruamel.yaml [6] or PyYAML [7]
TOML toml toml [8]
  • Supported formats of which backends are enabled automatically if required plugin modules are installed: python-anyconfig utilizes plugin mechanism provided by setuptools [9] and may support other formats if corresponding plugin backend modules are installed along with python-anyconfig:
Supported formats by pluggable backend modules
Format Type Required backend
Amazon Ion ion anyconfig-ion-backend [10]
BSON bson anyconfig-bson-backend [11]
CBOR cbor anyconfig-cbor-backend [12] or anyconfig-cbor2-backend [13]
ConifgObj configobj anyconfig-configobj-backend [14]
MessagePack msgpack anyconfig-msgpack-backend [15]

The supported formats of python-anyconfig on your system are able to be listed by 'anyconfig_cli -L' like this:

$ anyconfig_cli -L
Supported config types: bson, configobj, ini, json, msgpack, toml, xml, yaml
$

or with the API 'anyconfig.list_types()' will show them:

In [8]: anyconfig.list_types()
Out[8]: ['bson', 'configobj', 'ini', 'json', 'msgpack', 'toml', 'xml', 'yaml']

In [9]:
[4]https://pypi.python.org/pypi/simplejson
[5]ex. https://docs.oracle.com/javase/7/docs/api/java/util/Properties.html
[6]https://pypi.python.org/pypi/ruamel.yaml
[7]https://pypi.python.org/pypi/PyYAML
[8]https://pypi.python.org/pypi/toml
[9]http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins
[10]https://pypi.python.org/pypi/anyconfig-ion-backend
[11]https://pypi.python.org/pypi/anyconfig-bson-backend
[12]https://pypi.python.org/pypi/anyconfig-cbor-backend
[13]https://pypi.python.org/pypi/anyconfig-cbor2-backend
[14]https://pypi.python.org/pypi/anyconfig-configobj-backend
[15]https://pypi.python.org/pypi/anyconfig-msgpack-backend

Installation

Requirements

Many runtime dependencies are resolved dynamically and python-anyconfig just disables specific features if required dependencies are not satisfied. Therefore, only python standard library is required to install and use python-anyconfig at minimum.

The following packages need to be installed along with python-anyconfig to enable the features.

Feature Requirements Notes
YAML load/dump ruamel.yaml or PyYAML ruamel.yaml will be used instead of PyYAML if it's available to support the YAML 1.2 specification.
TOML load/dump toml none
BSON load/dump bson bson from pymongo package may work and bson [16] does not
Template config Jinja2 [17] none
Validation with JSON schema jsonschema [18] Not required to generate JSON schema.
Query with JMESPath expression jmespath [19] none
[16]https://pypi.python.org/pypi/bson/
[17]https://pypi.python.org/pypi/Jinja2/
[18]https://pypi.python.org/pypi/jsonschema/
[19]https://pypi.python.org/pypi/jmespath/

How to install

There is a couple of ways to install python-anyconfig:

  • Binary RPMs:

    If you're running Fedora 27 or later, or CentOS, you can install RPMs from these official yum repos. And if you're running Red Hat Enterprise Linux 7 or later, you can install RPMs from EPEL repos [20] .

    Or if you want to install the latest version, optionally, you can enable my copr repo, http://copr.fedoraproject.org/coprs/ssato/python-anyconfig/ .

  • PyPI: You can install python-anyconfig from PyPI with using pip:

    $ pip install anyconfig
  • pip from git repo:

    $ pip install git+https://github.com/ssato/python-anyconfig/
  • Build RPMs from source: It's easy to build python-anyconfig with using rpm-build and mock:

    # Build Source RPM first and then build it with using mock (better way)
    $ python setup.py bdist_rpm --source-only && mock dist/python-anyconfig-<ver_dist>.src.rpm

    or

    # Build Binary RPM to install
    $ python setup.py bdist_rpm

    and install RPMs built.

  • Build from source: Of course you can build and/or install python modules in usual way such like 'python setup.py bdist'.

[20]Thanks to Brett-san! https://src.fedoraproject.org/rpms/python-anyconfig/

Help and feedbak

If you have any issues / feature request / bug reports with python-anyconfig, please open issue tickets on github.com, https://github.com/ssato/python-anyconfig/issues.

The following areas are still insufficient, I think.

  • Make python-anyconfig robust for invalid inputs
  • Make python-anyconfig scalable: some functions are limited by max recursion depth.
  • Make python-anyconfig run faster: current implementation might be too complex and it run slower than expected as a result.
  • Documentation:
    • Especially API docs need more fixes and enhancements! CLI doc is non-fulfilling also.
    • English is not my native lang and there may be many wrong and hard-to-understand expressions.

Any feedbacks, helps, suggestions are welcome! Please open github issues for these kind of problems also!

More Repositories

1

packagemaker

Simple tool to help creating packages (rpm, deb, tgz, etc.)
Python
14
star
2

ansible-lint-custom-rules

Some ansible-lint custom rule examples
Python
12
star
3

sos-analyzer

A tool to scan and analyze data collected by sosreport
Python
12
star
4

miniascape

Virtualization miniascape - configs and data for libvirt KVM guests
Python
11
star
5

python-smhtml

Simple MHTML parsing library
Python
9
star
6

rpmkit

rpm related misc tools
Jinja
9
star
7

misc

misc files
Python
8
star
8

python-anytemplate

A python template abstraction layer module and a CLI based on it
Python
8
star
9

haskell

various haskell codes
Haskell
6
star
10

ansible-role-assertive-programming-examples

Ansible example role to demonstrate asseritve programming style in Ansible
Shell
4
star
11

ansible-role-generate-kickstart

Ansible role to generate kickstart installation files
Shell
2
star
12

ansible-role-nw-backup-config

An example ansible role to backup configuration files on network nodes
Shell
2
star
13

ansible-builder-example

Example configurations and misc stuff for ansible builder
2
star
14

python-myrepo

Tool to create yum repositories and manage rpms in them (python version)
Python
2
star
15

ansible-playbooks-rhui-3

Ansible playbooks to install and setup RHUI v3 servers
Shell
1
star
16

fleure-db

An utility to analyze updateinfo data
Python
1
star
17

ansible-playbook-awx-example

Ansible Playbook Example to run in AWX/Ansible Tower
Shell
1
star
18

fleure

Package level static analysis tool for RPM-based Linux systems
Python
1
star
19

python-anyconfig-fit-backend

A backend module for python-anyconfig to support to load and parse FIT (Flexible and Interoperable Data Transfer) data files
Python
1
star
20

ansible-playbooks-best-practice-examples

Ansible Example playbooks to demonstrate ansible best practices
1
star
21

python-anyconfig-fortios-backend

A backend module for python-anyconfig to support to load and parse fortios' "show \*configuration" outputs
Python
1
star
22

fortios-xutils

Very experimental miscellaneous and extra utilities for fortios (fortigate)
Python
1
star
23

python-anyconfig-ion-backend

Pluggable backend module for python-anyconfig to load and dump Amazon Ion data
Python
1
star
24

python-anyconfig-cbor-backend

A backend module for anyconfig to support CBOR files
Python
1
star
25

python-anyconfig-configobj-backend

A backend module for python-anyconfig to load and dump ConfigObj data
Python
1
star
26

ansible-role-simple-httpd-example

An ansible role to show how to do Ansible Role CI using the following tools, molecule, ansibleas as a molecle's verifier and bats to run more complex test cases.
1
star