• Stars
    star
    282
  • Rank 146,549 (Top 3 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 7 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

clicktail

clicktail is Altinity's agent for ingesting log file data into ClickHouse and making it available for exploration. Its favorite format is JSON, but understands how to parse a range of other well-known log formats.

clicktail is based on honeytail tool see here and its documentation

Supported Parsers

clicktail supports reading files from STDIN as well as from a file on disk.

Our complete list of parsers can be found in the parsers/ directory, but as of this writing, clicktail will support parsing logs generated by:

Installation

There are three installation options for clicktail:

  • From package repository
  • From downloaded package
  • From source

Install from Apt package repository:

curl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.deb.sh | bash

if you want to force os and dist of your system, use the following command:

curl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.deb.sh | os=ubuntu dist=xenial bash

Then install clicktail itself:

apt-get install clicktail

Install from individual dep package

Download and install the package from here: https://packagecloud.io/Altinity/clickhouse

Install from source:

go get github.com/Altinity/clicktail

to install to a specific path:

GOPATH=/usr/local go get github.com/Altinity/clicktail

...and make it

GOPATH=/usr/local go build
GOPATH=/usr/local go install

the binary will install to /usr/local/bin/clicktail

Configuration

Clicktail supports command line options as well as configuration file. In fact the file is not picked up by default when you are running clicktail from CLI so one should explicitly specify it with -c option.

Use clicktail.conf file to manage options. There are section of the file called Required Options which should be set in the first place.

Config Example

/etc/clicktail/clicktail.conf

[Application Options]
APIHost = http://localhost:8123/
...
[Required Options]
ParserName = mysql
LogFiles = /var/log/mysql/mariadb-slow.log
Dataset = clicktail.mysql_slow_log

Extra options for MySQL parser

There are useful options that could be passed along with mysql slow log entries but its not logged within the file itself, i.e. hostname of actual server. To get this data one can specify MySQL server connection details in config file.

...
[MySQL Parser Options]
Host = localhost:3306   ; or @unix(/var/run/mysqld/mysqld.sock)
User = username
Pass = userpass

Usage

Make sure ClickHouse server has proper schema created. See schema folder for CREATE DATABASE and CREATE TABLE statements.

Or do the following:

Create DB:

cat schema/db.sql | clickhouse-client --multiline

Create Table for MySQL slow logs:

cat schema/mysql.sql | clickhouse-client --multiline

Once schema is prepared you can run binary from CLI with MySQL parser:

clicktail --dataset='clicktail.mysql_slow_log' --parser=mysql --file=/var/log/mysql/mysql-slow.log

Or with Nginx parser:

clicktail -p nginx -f /var/log/nginx/access.log -d clicktail.nginx_log --nginx.conf=/etc/nginx/nginx.conf --nginx.format=combined

After you done with checking out your configuration options, you will need to store them in clicktail.conf in order to run clicktail as a service just like that:

service clicktail start

Retroactive logs loading

If you want to load files you already have into clicktail. You can use the same call as mentioned above but with extra parameter --backfill

clicktail --dataset='clicktail.mysql_slow_log' --parser=mysql --file=/var/log/mysql/mysql-slow.log --backfill

...this will load mysql-slow.log file into ClickTail and end the process.

ClickHouse Setup

Clicktail is required ClickHouse to be accessible as a target server. So you should have ClickHouse server installed.

Follow these steps in order to install ClickHouse server and make it work with Clicktail.

Once you installed Clickhouse you whould probably need it to be open to outside connections. In order to do so update /etc/clickhouse-server/config.xml to add the following line:

<listen_host>0.0.0.0</listen_host>

just after: <listen_host>127.0.0.1</listen_host>

Also make sure ClickHouse port (which is 8123 by default) is open with your firewall.

For more advanced usage, options, and the ability to scrub or drop specific fields, see documentation.

More Repositories

1

clickhouse-operator

Altinity Kubernetes Operator for ClickHouse creates, configures and manages ClickHouse clusters running on Kubernetes
Go
1,853
star
2

clickhouse-backup

Tool for easy backup and restore for ClickHouse® using object storage for backup files.
Go
1,227
star
3

clickhouse-grafana

Altinity Grafana datasource plugin for ClickHouse®
TypeScript
713
star
4

clickhouse-mysql-data-reader

utility to read mysql data
Python
368
star
5

clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse
Python
216
star
6

clickhouse-rpm-install

How to install clickhouse RPM packages
136
star
7

tpc-ds

TPC-DS benchmarks
C
64
star
8

altinity-dashboard

Altinity Dashboard helps you manage ClickHouse installations controlled by clickhouse-operator.
TypeScript
62
star
9

altinityknowledgebase

The repository for the Altinity Knowledge Base located at https://kb.altinity.com
HTML
54
star
10

clickhouse-zabbix-template

Zabbix template for ClickHouse
Shell
42
star
11

clickhouse-zookeeper-recovery

Shell
30
star
12

clickhouse-python-examples

Sample Python code for Clickhouse
Jupyter Notebook
29
star
13

tableau-connector-for-clickhouse

Tableau connector to ClickHouse using ODBC or JDBC driver
JavaScript
27
star
14

clickhouse-regression

Altinity ClickHouse Regression Tests
Python
23
star
15

clickhouse-cluster-manager

Python
22
star
16

clickhouse-sts

ClickHouse stress tests suite
Shell
20
star
17

altinity-datasets

Altinity Datasets for ClickHouse
Python
17
star
18

clickhouse-benchmarks

PHP
12
star
19

clickhouse-sql-examples

Shell
12
star
20

terraform-aws-eks-clickhouse

Terraform module for creating EKS clusters optimized for ClickHouse with EBS and autoscaling ☁️
HCL
11
star
21

clickhouse-demo-dataset-migration

10
star
22

ebs-params-controller

EBS Params Controller allows to manage AWS EBS volume parameters via PVC annotations
Shell
9
star
23

argocd-examples-clickhouse

Shell
8
star
24

ch-stable-versions

The list of known stable versions of ClickHouse
8
star
25

terraform-provider-altinitycloud

Terraform provider for Altinity.Cloud ☁️
Go
7
star
26

altinitycloud-connect

Altinity.Cloud tunneling daemon; part of Altinity.Cloud Anywhere
5
star
27

terraform-altinitycloud-connect

Terraform module for connecting Kubernetes clusters to Altinity.Cloud
HCL
5
star
28

kubernetes-blueprints-for-clickhouse

Shell
5
star
29

clickhouse-graphite-integration

4
star
30

ssb

Shell
4
star
31

hetzner-docker-registry

Scripts to setup a local docker registry or provision a Hetzner Cloud instance and create a docker registry there.
Python
3
star
32

clickhouse-graphouse-integration

Integrate ClickHouse and Graphouse as a storage for Graphite
3
star
33

kioperf

Golang utility to measure I/O performance on block storage and object storage
Go
3
star
34

altinity-demo-appliance

PHP
3
star
35

documentationIssues

Used to log issues with the Altinity Documentation Site.
2
star
36

blog

1
star
37

clickhouse-odbc-rpm

RPM build script for clickhouse-odbc
Shell
1
star
38

terraform-altinitycloud-connect-aws

Terraform module for connecting AWS accounts to Altinity.Cloud ☁️
HCL
1
star