bifrost
Archive Kafka data safely to either Amazon's S3 or Microsoft's Azure Blob Storage.
bifrost is a daemon that connects to Kafka, consumes all topics and persists them to the cloud for long term storage and analysis. It uses the baldr file format to store Kafka messages. baldr-files are gzipped before they are written to disk.
There are other services for persisting messages from Kafka to S3, such as secor from pinterest. The main difference between bifrost and these is, that whereas they rely on the Hadoop sequence files for storage, bifrost uses the baldr format. This means that consumers of the persisted messages do not need to rely on often very large implementations of libraries for reading Hadoop sequence files. The baldr file format follows a minimal design that allows for easy and quick streaming with a very small code-footprint. It does not allow for arbitrary indexing.
Usage
bifrost can be run directly from a checkout of the project by using
leiningen. The app requires some basic configuration, namely ZooKeeper
configuration to connect to Kafka and cloud storage credentials to store
baldr-files on the chosen cloud storage. The project contains an example configuration in
etc/config.edn.example.
$ lein run -- --config ./etc/config.edn
To run the app in production, we recomment building an uberjar and run that on the app server.
$ lein uberjar
$ java -jar target/*-standalone.jar --config /opt/uswitch/bifrost/etc/config.edn
The Java temp-dir is used for storing baldr-files locally before
uploading them. Files are removed upon succesful upload and program
exit. To change the temp-directory, override java.io.tmpdir.
Logging is done through logback. To configure logback, please set
logback.configurationFile. The logback configuration is only respected
for uberjars. sl4j is used in development.
Here's a complete example of configuring and running an uberjar in production.
$ java -Djava.io.tmpdir=/mnt/bifrost-tmp \
-Dlogback.configurationFile=/opt/uswitch/bifrost/etc/logback.xml \
-server \
-jar /opt/uswitch/bifrost/lib/*-standalone.jar \
--config /opt/uswitch/bifrost/etc/config.edn
License
Copyright © 2014 uSwitch
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.