• Stars
    star
    925
  • Rank 49,378 (Top 1.0 %)
  • Language
    Ruby
  • License
    Apache License 2.0
  • Created about 7 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Ready to use log management solution for Kubernetes storing data in ClickHouse and providing web UI.


UPDATE (December'20): Please note loghouse is no longer being actively developed.

Back in 2017 (when we created this project), there was no other solution to get what we (in Flant) desperately needed. However creating such tools is out of our primary focus, hence β€” while maintaining loghouse β€” we've been also waiting for other projects to emerge. Luckily, the Kubernetes ecosystem grows and evolves at an amazing pace. Today, we are happy to admit the existence of other solid log management solutions we can rely on. For most cases, we use Loki. (Graylog and Elasticsearch-based solutions β€” i.e. ELK, EFK β€” might be other options to consider.)

It means we don't need to improve loghouse anymore. Since it's still Open Source, you are very welcome to contribute (or even contact us to become a maintainer).


Ready to use log management solution for Kubernetes. Efficiently store big amounts of your logs (in ClickHouse database), process them using a simple query language and monitor them online through web UI. Easy and quick to deploy in an already functioning Kubernetes cluster.

Status is alpha. However we (Flant) use it in our production Kubernetes deployments since September, 2017. Data structure might change during alpha releases, so please be careful when updating (all relevant information is published in corresponding release notes). Data structure will become stable in beta version.

Loghouse-dashboard UI demo in action (~3 Mb):

loghouse web UI

Features

  • Collecting and storing logs from all Kubernetes pods efficiently:
    • Fluentd processes upto 10,000 log entries per second consuming 300 MB of RAM (installed at each K8s node).
    • ClickHouse makes disk space usage minimal. Examples of logs stored in our production deployments: 3.7 million entries require 1.2 GB, 300m β€” 13 GB, 5,35 billion β€” 54 GB.
  • Simple query language: Easy to select entries by exact keys values or regular expressions, multiple conditions are supported with AND/OR. Learn more in query language docs.
  • Selecting entries based on additional containers' data available in Kubernetes API (pod's and container's names, host, namespace, labels, etc).
  • Quickly & straightforward deployable to Kubernetes via Dockerfiles and Helm chart.
  • Web UI made cosy and powerful:
    • Papertrail-like user experience.
    • Customizable time frames: from date to date / from now till given period (last hour, last day, etc) / seek to specific time and show logs around it.
    • Infinite scrolling of older log entries.
    • Save your queries to use in future.
    • Basic permissions (limiting entries shown for users by specifying Kubernetes namespaces).
    • Exporting current query's results to CSV (more formats will be supported).
  • fluentd monitoring via Prometheus with Grafana dashboards for ClickHouse and fluentd.

Installation

To install loghouse, you need to use Helm. Minimal kubernetes cluster version is >=1.9. Also, it is considered that cert-manager is already installed in your cluster.

The whole process is as simple as these two steps:

  1. Add loghouse charts:
# helm repo add loghouse https://flant.github.io/loghouse/charts/
  1. Install a chart.

2.1. Easy way:

# helm fetch loghouse/loghouse --untar
# vim loghouse/values.yaml
# helm install --namespace loghouse -n loghouse loghouse

Note: use --timeout 1200 flag in case slow image pulling.

2.2. Using specific parameters (check variables in chart's values.yaml β€” not documented yet):

# helm install -n loghouse loghouse/loghouse --set 'param=value' ...

Web UI (loghouse-dashboard) will be reachable via address specified in values.yaml config as loghouse_host. You'll be prompted by basic authorization generated via htpasswd and configured in auth parameter of your values.yaml.

To clean old logs in cron, you can use a script in this issue.

Architecture

loghouse architecture

A pod with fluentd collecting logs will be installed on each node of your Kubernetes cluster. Technically, it is implemented by means of DaemonSet having tolerations for all possible taints, so it includes all the nodes available. Logs directories from all hosts are mounted to fluentd pods and watched by fluentd. Kubernetes_metadata filter is applied for all the Docker containers' logs to get additional information about containers via Kubernetes API. Another filter, record_modifier, is then used to "prepare" all the data we have. And the last step is passing this data to fluentd output plugin executing clickhouse-client CLI tool to insert new entries into ClickHouse DBMS.

Note on logs format: If log entry is in JSON, it will be formatted according to its values' types, i.e. each field will be stored in corresponding table: string_fields, number_fields, boolean_fields, null_fields or labels (the last one is used for containers labels to make further filtering and lookups easy). ClickHouse built-in functions will be used to process these data types. If log entry isn't in JSON, it will be stored in string_fields table.

By default, ClickHouse DBMS is deployed as a single instance via StatefulSet which brings this instance to a random K8s node (this behaviour can be changed by using nodeSelector and tolerations to choose a specific node). ClickHouse stores its data in hostPath volume or Persistent Volumes Claim (PVC) created with any storageClass you prefer. You can find other deployment options for ClickHouse (i.e. using an external ClickHouse instance) here.

Web UI (screenshot) is composed of two components:

  • frontend β€” nginx with basic authorization. This authorization is used to limit user's access with logs from given Kubernetes namespaces only;
  • backend β€” Ruby application displaying logs from ClickHouse.

ClickHouse and Fluentd components have optional Prometheus exporters. You can find our default dashboards for Grafana here.

Upgrading

In v0.3.0, Helm chart for loghouse has been rewritten. All API versions as well as Helm hook policy have been updated. To upgrade your loghouse installation to v0.3.0, you have to remove conflicting objects:

kubectl -n loghouse delete jobs,ing --all

Warning! Database schema has been also changed. Please, prepare a backup of ClickHouse data before upgrading. Note that the migration task will be automatically started at the end of the upgrade process.

Roadmap

We're going to add logs filtering, another deployment options (having ClickHouse instances on each K8s node), migrate frontend to AngularJS and backend to Golang, add command-line interface (CLI) and much more.

More details will be available in our issues.

More Repositories

1

shell-operator

Shell-operator is a tool for running event-driven scripts in a Kubernetes cluster
Go
2,381
star
2

ovpn-admin

Simple web UI to manage OpenVPN users.
Go
1,344
star
3

grafana-statusmap

Grafana panel plugin to visualize status of multiple objects over time
TypeScript
829
star
4

addon-operator

A system to manage additional components for Kubernetes cluster in a simple, consistent and automated way.
Go
482
star
5

nginx-http-rdns

Nginx HTTP rDNS module
C
144
star
6

examples

Configurations, scripts & samples used in guides and other articles from Flant engineers.
JavaScript
105
star
7

curator-opensearch

Curator for OpenSearch.
Python
75
star
8

glaball

A CLI tool to manage self-hosted GitLab instances in bulk.
Go
71
star
9

pam_docker

C
61
star
10

elasticsearch-extractor

Simple web UI to extract any index from Elasticsearch snapshot into repository.
Go
46
star
11

netgwm

NetGWM (Network Gateway Manager)
Python
38
star
12

helm-charts

Public Helm charts from Flant.
Smarty
36
star
13

cert-manager-webhook-regru

The webhook and the ClusterIssuer resource for automatic provisioning of reg.ru SSL certificates in Kubernetes
Go
33
star
14

helm-apps

Smarty
24
star
15

gitlab-ci-examples

22
star
16

negentropy

S - security
Go
21
star
17

libjq-go

CGO bindings for jq with cache for compiled programs
Go
15
star
18

flunch

open source non-alcohol drink
14
star
19

jekyll_include_plugin

Plugin for including contents of local/remote plain text files (or parts of them) into your pages. Allows for multilang comments in the included files.
Ruby
13
star
20

uuidable

With this gem you can use UUID instead of id in routes. But id is still primary key.
Ruby
9
star
21

artery

Main messaging system between Rails [micro]services implementing message bus pattern on NATS (for now).
Ruby
9
star
22

clickhouse_exporter

Image for Percona-Lab/clickhouse_exporter
Dockerfile
7
star
23

php_fpm_docker

Run php_fpm pools in docker containers
7
star
24

ipm

ipm (IP Manager)
Python
7
star
25

nginx_saml_authenticator

Ruby
6
star
26

kubernetes-efk

ElasticSearch + Fluentd + Kibana in Kubernetes
Shell
5
star
27

docker_penetration_experiment

C
5
star
28

external_services

Gem helps syncronize objects to different external services like Gitlab, Redmine and any other.
Ruby
5
star
29

superhosting

Ruby
4
star
30

kube-ca-auth-proxy

Shell
4
star
31

kube-client

Kubernetes client wrapper
Go
4
star
32

charts

Smarty
4
star
33

htop

This repository contains htop package for buildizer as native package: package build rules and upstream source.
C
3
star
34

elasticsearch-oneday-exporter

Go
3
star
35

runtex

Python
3
star
36

LMPD

Lightweight Modular Policy Daemon
C
3
star
37

dapp_bot

Telegram dapp bot
Go
3
star
38

elasticsearch-snapshot-exporter

Go
3
star
39

simple_saml

Ruby
3
star
40

kube-resource-unit-converter

Convert Kubernetes Resource Quantity representation to bytes
Go
3
star
41

machine-controller-manager-provider-yandex

Go
3
star
42

redmine_issue_priorities_per_tracker

Ruby
2
star
43

gogost

Go
2
star
44

kube-controller-manager

Dockerfile
2
star
45

flant.github.io

HTML
2
star
46

buildizer

Ruby
2
star
47

redmine_attachments_database_storage

This is a plugin for Redmine which allows to store attachments in database.
Ruby
2
star
48

kube-rabbitmq

Go
1
star
49

jekyll_remote_plantuml_plugin

Plugin for including local/remote PlantUML diagrams into your pages
Ruby
1
star
50

dumpdb

Shell
1
star
51

buildizer-env-ubuntu

1
star
52

shellfold

Ruby
1
star
53

path_mapper

Ruby
1
star
54

buildizer-env-centos

1
star
55

gogomonia-utils

Shell
1
star
56

promicher

Prometheus alerts enricher
Go
1
star
57

rpmchange

Rpm spec change tool.
Ruby
1
star
58

cert-manager-webhook-nicru

Go
1
star
59

website-core

Ruby
1
star