node-logstash

What is it ?

It's a NodeJS implementation of Logstash.

What to do with node-logstash ?

node-logstash is a tool to collect logs on servers. It allows sending its logs to a central server and to ElasticSearch for indexing.

On top of the elastic search, you can use a specialized interface like kibana to dive into your logs.

Why a new implementation?

When I tried logstash, I had some problems. This project mainly addresses those problems. This version should have:

• lower memory footprint
• lower CPU footprint
• faster startup delay

Moreover, it's written in NodeJS, which is a perfect language for programs with many IO.

node-logstash is compatible with logstash. You can replace a node-logstash node with a logstash one. The data are formatted in the same way to be compatible with logstash UIs.

How does it work?

The architecture is identical to logstash architecture. You have to instantiate plugins with the node-logstash core. There are three type of modules:

• inputs plugins: where data come into node-logstash. Examples: file, zeromq transport layer
• filter plugins: extract fields from logs, like timestamps. Example: regex plugin
• outputs plugins: where data leave from node-logstash: Examples: ElasticSearch , zeromq transport layer.

A typical node-logstash deployment contains agents to crawl logs and a log server.

On agent, node-logstash is configured with inputs plugins to get logs from your software stack, and one output plugin to send logs to log server (eg. zeromq output plugin).

On log server, logs come through a zeromq input plugin, are processed (fields and timestamps extraction), and send to ElasticSearch.

How to get help?

Please open an issue.

Future of this project

October 25th, 2015.

When I started node-logstash, the ecosystem around logstash and ElasticSearch were almost non-existent. In 2015, the situation is not the same :

• Great ecosystem around ElasticSearch and logstash, FileBeat project
• Logstash is now the only way to push events to ElasticSearch (deprecation of rivers)

So, what is the future of node-logstash?

• as a tool to collect logs on files and send them through the network, node-logstash is still useful with lower size, instant start, lower CPU / Memory footprint (in my tests with logstash 1.5.0). The comparison is different with Lumberjack and FileBeat.
• as log processing tool, it has the same advantages, but the plugin ecosystem is smaller than Logstash.
• as an injection tool in ElasticSearch: ZeroMQ river will soon be unusable (deprecation of rivers). You have to use bulk api to inject data. It should be less efficient than starting an embedded ElasticSearch node, as in the original Logstash.

Current project status

Node-logstash is production ready and used in production. Installation is a classical node project installation, with some scripts for native packaging.

Maintainers: currently I, @bpaquet, am the only maintainer. I will keep dependencies up to date, update the core to follow node version, but I do not have time to add features to the core. See Contributing below.

Weaknesses :

• tests are difficult to maintain, even if they are many and the code coverage is good. Replacing vows by mocha is a good way to improve that, but it's a big rework.

Contributing

Which Pull Requests (PR) will be merged?

Add plugin (output, input or filter)

Conditions to have a PR merged :

• respect jslint
• provide documentation in /docs
• do not modify core. Modifications allowed :
  • add a plugin in Readme.md.
  • add optional dependencies in package.json
• If you provide unit tests, you can write in plugin documentation that the plugin is a plugin core.
• If you do not provide unit tests, please indicate in the documentation: "Status: contributed plugin, maintained by @xxxx. Production ready.", and indicate your Github login.

You are encouraged to ask to merge plugins without tests, which are not production ready.

Core modification

Please respect jslint, and provide all needed unit tests. How to use it?

Installation

Simple way

Use prepackaged deb files.

After install, just add your config files to /etc/node-logstash/plugins.conf.d, and restart node-logstash service node-logstash restart.

To see what options are passed to node-logstash, see here.

To change log level, do node-logstash config:set LOG_LEVEL=debug, and restart node-logstash.

Manual install

• Install NodeJS, version >= 0.12
• Install build tools
  • Debian based system: apt-get install build-essential
  • Centos system: yum install gcc gcc-c++ make
• Install zmq dev libraries: This is required to build the node zeromq module.
  • Debian based system: apt-get install libzmq1. Under recent releases, this package is present in default repositories. On ubuntu lucid, use this ppa. On debian squeeze, use backports.
  • Centos 6: yum install zeromq zeromq-devel. Before, you have to add the rpm zeromq repo : curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
• Clone repository: git clone git://github.com/bpaquet/node-logstash.git && cd node-logstash
• Install dependencies: npm install.

The executable is bin/node-logstash-agent

Configuration formats

There are two formats for configuration:

1. The legacy format uses URLs.
2. The other one is identical to the logstash config format.

Note: if you are using multiple config files, you can mix formats.

Configuration by URL (legacy)

A plugin is instantiated by an URL. Example: input://file:///tmp/toto.log. This URL instantiates an input file plugin which monitors the file /tmp/toto.log.

The URLs can be specified:

• directly on the command line
• in a file (use the --config_file switch)
• in all files in a directory (use the --config_dir switch)

Configuration by logstash config files (recommended)

Example for an input file

input {
  file {
    path => '/tmp/toto.log'
  }
}

You can use if to have an event dependent configuration. See here for details. As for URLs, config can be specified

• directly on the command line
• in a file (use the --config_file switch)
• in all files in a directory (use the --config_dir switch)

Note: the implementation is young, all bugs reports are welcome. Note: both formats can be mixed.

Command lines params

• --log_level to change the log level (emergency, alert, critical, error, warning, notice, info, debug)
• --log_file to redirect logs to a log file.
• --patterns_directories to add some directories (separated by , ), for loading config for regex plugin and grok plugins. Grok patterns files must be located under a grok subdirectory for each specified directory.
• --db_file to specify the file to use a database for file inputs (see below)
• --http_max_sockets to specify the max sockets of http.globalAgent.maxSockets. Default to 100.
• --alarm_file to specify a file which will be created if node-logstash goes in alarm mode (see below).

Examples

Config file for an agent:

input {
  file {
    path => "/var/log/nginx/access.log"
  }
}

output {
  zeromq {
    address => ["tcp://log_server:5555"]
  }
}

Config file for log server:

input {
  zeromq {
    address => ["tcp://0.0.0.0:5555"]
  }
}

filter {
  regex {
    pattern => http_combined
  }
}

output {
  elasticsearch {
    host => localhost
    port => 9200
  }
}

Adding your plugins

You can add easily add your plugins :

Manually :

• create a directory layout on the path of your choice : /var/my_plugins/inputs, /var/my_plugins/outputs, /var/my_plugins/filters
• set the NODE_PATH variable to NODE_PATH=/var/my_plugins:/node_logstash_path/lib
• add your plugins in inputs, outputs or filters directory. In the plugin code, you can reference base plugins with var base_filter = require('lib/base_filter');
• reference your plugin as usual.

With native packaging

The plugins must be deployed in /var/db/node-logstash/custom_plugins. All subdirectories already exist. The NODE_PATH is already set.

Signals

• USR1: stopping or starting all inputs plugins. Can be used to close input when output targets are failing
• USR2: see below file output plugin

Changelog

Changelog

Plugins list

Input plugins

• File
• Syslog
• ZeroMQ
• Redis
• HTTP
• Websocket
• TCP / TLS
• Google app engine
• AMQP
• SQS

Common concepts / parameters :

• Unserializers
• Tags/fields

Filter plugins

• Regex
• Grok
• Mutate Replace
• Grep
• Reverse DNS
• Compute field
• Compute date field
• Split
• Truncate
• Rename
• Multiline
• Json fields
• Geoip
• Eval
• Bunyan
• HTTP Status Classifier
• Remove field when equal

Common concepts / parameters :

• Common parameters
• Tags/fields

Outputs

• ZeroMQ
• ElasticSearch
• Statsd
• Gelf
• File
• HTTP Post
• Websocket
• Redis
• Logio
• TCP / TLS
• AMQP
• SQS
• HEP

Common concepts / parameters :

• Common parameters
• Serializers

Misc

• Elasticsearch mapping

License

Copyright 2012 - 2014 Bertrand Paquet

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.