Elasticsearch, Logstash, Kibana with Curator and Beats support
Ready to go Docker configuration for set up ELK stack in a minutes
Description
- Elasticsearch - official image with data volume in
elasticsearch/datadirectory - Logstash - official image + custom configuration which takes care about Filebeat, Topbeat and Packetbeat index templates for Elasticsearch + multiline option for correct stacktraces representation
- Kibana - official image
- Curator - lightweight 50mb container which could run scheduled tasks against Elasticsearch to manage its indices (delete, optimize, snapshot, etc)
Setup
- Install required Beats shippers on the host which should be monitored
- Install Docker and Docker Compose on the ELK host
- Clone this repository and hit
docker-compose build
Usage
Start everything with one command:
docker-compose up -d
Keep track of your containers execution. For example, controll Curator scheduled tasks:
docker-compose logs curator
Scaling up
This simple configuration will run very happily on your laptop, but it can be easely scaled up for highload production servers with a huge amount of logs and monitoring data.
- Learn about Elasticsearch cluster and horizontal scaling
- Consider Logstash scaling with multiple shipping and indexing instances with MQ in the middle
Notes
- You may want to add Kibana Shield plugin for users authentication
- It might be really helpful to use Elasticsearch Watcher or Yelp ElastAlert to get notified on significant events or anomalies in your data
- You can specify resource limits (like CPU and memory allocation) for each docker container
Feel free to contact me with any issues and questions