Punt

Punt is a lightweight and simple daemon that parses, transforms, mutates, and ships logs into Elasticsearch. Punt was built as a fast and reliable alternative to Logstash, which means it's focus is to fit directly into existing ELK setups. Punt was built at Discord to manage the over 4 billion log lines we process per day.

Features

• Designed to be fast and reliable
• Simple JSON-based configuration file
• Supports rsyslog over UDP/TCP, including multiple framing formats and SSL
• Multiple ingest servers and egress ES clusters
• Management and configuration of ES index templates and type mappings
• Ability to parse and transform structured (JSON) logs
• Threshold/Matching based alerting
• Streaming API for real time log tailing
• Support for attaching maxmind GeoIP2 geolocation data
• Automatic deletion of old indexes based on retention rules
• Exporting of metrics in the Datadog statsd format

Why Not Logstash?

When Discord originally started logging, we used a standard ELK stack setup. Initially this worked well for a low-volume of logs, however as our log volume grew (~750m log lines a day) Logstash quickly began to fall behind. As we spent more and more time tweaking and scaling Logstash/JVM/JRuby, we quickly realised it was not a long-term solution. Punt spawned out of a frustrating weekend dealing with constant Logstash lockups and JVM struggles.

Where Logstash aims to be immensely configurable and pluggable via its DSL, Punt aims to be an extremely performant solution, without compromising or reducing the core features required to handle and store structured log data.

Installation

Go

To install Punt using the Go toolchain, simply

go get github.com/discordapp/punt/cmd/puntd

Package (Debian/Ubuntu)

Punt was designed to be installed as a package on debian systems, and thus includes a simple dpkg build script based on fpm. To build a package simply:

cd packaging/
VERSION=0.0.1 ./build.sh

The package includes a simple upstart script.