• Stars
    star
    2,853
  • Rank 15,921 (Top 0.4 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created almost 8 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Golang driver for ClickHouse

ClickHouse run-tests Go Reference

Golang SQL database client for ClickHouse.

Versions

There are two version of this client, v1 and v2, available as separate branches.

v1 is now in a state of a maintenance - we will only accept PRs for bug and security fixes.

Users should use v2 which is production ready and significantly faster than v1.

v2 has breaking changes for users migrating from v1. These were not properly tracked prior to this client being officially supported. We endeavour to track known differences here and resolve where possible.

Supported ClickHouse Versions

The client is tested against the currently supported versions of ClickHouse

Supported Golang Versions

Client Version Golang Versions
=> 2.0 <= 2.2 1.17, 1.18
>= 2.3 1.18.4+, 1.19
>= 2.14 1.20, 1.21

Key features

Support for the ClickHouse protocol advanced features using Context:

  • Query ID
  • Quota Key
  • Settings
  • Query parameters
  • OpenTelemetry
  • Execution events:
    • Logs
    • Progress
    • Profile info
    • Profile events

Documentation

https://clickhouse.com/docs/en/integrations/go

clickhouse interface (formally native interface)

	conn, err := clickhouse.Open(&clickhouse.Options{
		Addr: []string{"127.0.0.1:9000"},
		Auth: clickhouse.Auth{
			Database: "default",
			Username: "default",
			Password: "",
		},
		DialContext: func(ctx context.Context, addr string) (net.Conn, error) {
			dialCount++
			var d net.Dialer
			return d.DialContext(ctx, "tcp", addr)
		},
		Debug: true,
		Debugf: func(format string, v ...any) {
			fmt.Printf(format+"\n", v...)
		},
		Settings: clickhouse.Settings{
			"max_execution_time": 60,
		},
		Compression: &clickhouse.Compression{
			Method: clickhouse.CompressionLZ4,
		},
		DialTimeout:      time.Second * 30,
		MaxOpenConns:     5,
		MaxIdleConns:     5,
		ConnMaxLifetime:  time.Duration(10) * time.Minute,
		ConnOpenStrategy: clickhouse.ConnOpenInOrder,
		BlockBufferSize: 10,
		MaxCompressionBuffer: 10240,
		ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
			Products: []struct {
				Name    string
				Version string
			}{
				{Name: "my-app", Version: "0.1"},
			},
		},
	})
	if err != nil {
		return err
	}
	return conn.Ping(context.Background())

database/sql interface

OpenDB

conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:9999"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	TLS: &tls.Config{
		InsecureSkipVerify: true,
	},
	Settings: clickhouse.Settings{
		"max_execution_time": 60,
	},
	DialTimeout: time.Second * 30,
	Compression: &clickhouse.Compression{
		Method: clickhouse.CompressionLZ4,
	},
	Debug: true,
	BlockBufferSize: 10,
	MaxCompressionBuffer: 10240,
	ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
		Products: []struct {
			Name    string
			Version string
		}{
			{Name: "my-app", Version: "0.1"},
		},
	},
})
conn.SetMaxIdleConns(5)
conn.SetMaxOpenConns(10)
conn.SetConnMaxLifetime(time.Hour)

DSN

  • hosts - comma-separated list of single address hosts for load-balancing and failover
  • username/password - auth credentials
  • database - select the current default database
  • dial_timeout - a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix such as "300ms", "1s". Valid time units are "ms", "s", "m". (default 30s)
  • connection_open_strategy - round_robin/in_order (default in_order).
    • round_robin - choose a round-robin server from the set
    • in_order - first live server is chosen in specified order
  • debug - enable debug output (boolean value)
  • compress - compress - specify the compression algorithm - “none” (default), zstd, lz4, gzip, deflate, br. If set to true, lz4 will be used.
  • compress_level - Level of compression (default is 0). This is algorithm specific:
    • gzip - -2 (Best Speed) to 9 (Best Compression)
    • deflate - -2 (Best Speed) to 9 (Best Compression)
    • br - 0 (Best Speed) to 11 (Best Compression)
    • zstd, lz4 - ignored
  • block_buffer_size - size of block buffer (default 2)
  • read_timeout - a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix such as "300ms", "1s". Valid time units are "ms", "s", "m" (default 5m).
  • max_compression_buffer - max size (bytes) of compression buffer during column by column compression (default 10MiB)
  • client_info_product - optional list (comma separated) of product name and version pair separated with /. This value will be pass a part of client info. e.g. client_info_product=my_app/1.0,my_module/0.1 More details in Client info section.

SSL/TLS parameters:

  • secure - establish secure connection (default is false)
  • skip_verify - skip certificate verification (default is false)

Example:

clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60

HTTP Support (Experimental)

The native format can be used over the HTTP protocol. This is useful in scenarios where users need to proxy traffic e.g. using ChProxy or via load balancers.

This can be achieved by modifying the DSN to specify the HTTP protocol.

http://host1:8123,host2:8123/database?dial_timeout=200ms&max_execution_time=60

Alternatively, use OpenDB and specify the interface type.

conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:8123"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	Settings: clickhouse.Settings{
		"max_execution_time": 60,
	},
	DialTimeout: 30 * time.Second,
	Compression: &clickhouse.Compression{
		Method: clickhouse.CompressionLZ4,
	},
	Protocol:  clickhouse.HTTP,
})

Note: using HTTP protocol is possible only with database/sql interface.

Compression

ZSTD/LZ4 compression is supported over native and http protocols. This is performed column by column at a block level and is only used for inserts. Compression buffer size is set as MaxCompressionBuffer option.

If using Open via the std interface and specifying a DSN, compression can be enabled via the compress flag. Currently, this is a boolean flag which enables LZ4 compression.

Other compression methods will be added in future PRs.

TLS/SSL

At a low level all client connect methods (DSN/OpenDB/Open) will use the Go tls package to establish a secure connection. The client knows to use TLS if the Options struct contains a non-nil tls.Config pointer.

Setting secure in the DSN creates a minimal tls.Config struct with only the InsecureSkipVerify field set (either true or false). It is equivalent to this code:

conn := clickhouse.OpenDB(&clickhouse.Options{
	...
    TLS: &tls.Config{
            InsecureSkipVerify: false
	}
	...
    })

This minimal tls.Config is normally all that is necessary to connect to the secure native port (normally 9440) on a ClickHouse server. If the ClickHouse server does not have a valid certificate (expired, wrong host name, not signed by a publicly recognized root Certificate Authority), InsecureSkipVerify can be to true, but that is strongly discouraged.

If additional TLS parameters are necessary the application code should set the desired fields in the tls.Config struct. That can include specific cipher suites, forcing a particular TLS version (like 1.2 or 1.3), adding an internal CA certificate chain, adding a client certificate (and private key) if required by the ClickHouse server, and most of the other options that come with a more specialized security setup.

HTTPS (Experimental)

To connect using HTTPS either:

  • Use https in your dsn string e.g.

    https://host1:8443,host2:8443/database?dial_timeout=200ms&max_execution_time=60
  • Specify the interface type as HttpsInterface e.g.

conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:8443"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	Protocol:  clickhouse.HTTP,
})

Client info

Clickhouse-go implements client info as a part of language client specification. client_name for native protocol and HTTP User-Agent header values are provided with the exact client info string.

Users can extend client options with additional product information included in client info. This might be useful for analysis on a server side.

Order is the highest abstraction to the lowest level implementation left to right.

Usage examples for native API and database/sql are provided.

Async insert

Asynchronous insert is supported via dedicated AsyncInsert method. This allows to insert data with a non-blocking call. Effectively, it controls a async_insert setting for the query.

Using with batch API

Using native protocol, asynchronous insert does not support batching. It means, only inline query data is supported. Please see an example here.

HTTP protocol supports batching. It can be enabled by setting async_insert when using standard Prepare method.

For more details please see asynchronous inserts documentation.

PrepareBatch options

Available options:

  • WithReleaseConnection - after PrepareBatch connection will be returned to the pool. It can help you make a long-lived batch.

Benchmark

V1 (READ) V2 (READ) std V2 (READ) clickhouse API
1.218s 924.390ms 675.721ms
V1 (WRITE) V2 (WRITE) std V2 (WRITE) clickhouse API V2 (WRITE) by column
1.899s 1.177s 699.203ms 661.973ms

Install

go get -u github.com/ClickHouse/clickhouse-go/v2

Examples

native interface

std database/sql interface

ClickHouse alternatives - ch-go

Versions of this client >=2.3.x utilise ch-go for their low level encoding/decoding. This low lever client provides a high performance columnar interface and should be used in performance critical use cases. This client provides more familar row orientated and database/sql semantics at the cost of some performance.

Both clients are supported by ClickHouse.

Third-party alternatives

More Repositories

1

ClickHouse

ClickHouse® is a real-time analytics DBMS
C++
37,339
star
2

clickhouse-java

Java client and JDBC driver for ClickHouse
Java
1,324
star
3

clickhouse-presentations

Presentations, meetups and talks about ClickHouse
HTML
981
star
4

ClickBench

ClickBench: a Benchmark For Analytical Databases
HTML
642
star
5

metabase-clickhouse-driver

ClickHouse database driver for the Metabase business intelligence front-end
Clojure
439
star
6

clickhouse_exporter

This is a simple server that periodically scrapes ClickHouse stats and exports them via HTTP for Prometheus(https://prometheus.io/) consumption.
Go
361
star
7

ch-go

Low-level Go Client for ClickHouse
Go
314
star
8

clickhouse-connect

Python driver/sqlalchemy/superset connectors
Python
308
star
9

clickhouse-cpp

C++ client library for ClickHouse
C
300
star
10

NoiSQL

NoiSQL — Generating Music With SQL Queries
SQL
276
star
11

clickhouse-rs

Official pure Rust typed client for ClickHouse DB
Rust
270
star
12

graphouse

Graphouse allows you to use ClickHouse as a Graphite storage.
Java
259
star
13

clickhouse-odbc

ODBC driver for ClickHouse
C
246
star
14

dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Python
245
star
15

adsb.exposed

Interactive visualization and analytics on ADS-B data with ClickHouse
HTML
223
star
16

clickhouse-js

Official JS client for ClickHouse DB
TypeScript
212
star
17

spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
Scala
180
star
18

clickhouse-jdbc-bridge

A JDBC proxy from ClickHouse to external databases
Java
166
star
19

clickhouse-kafka-connect

ClickHouse Kafka Connector
Java
145
star
20

github-explorer

Everything You Always Wanted To Know About GitHub (But Were Afraid To Ask)
HTML
141
star
21

examples

ClickHouse Examples
Jupyter Notebook
118
star
22

clickhouse-docs

Official documentation for ClickHouse
JavaScript
110
star
23

click-ui

The home of the ClickHouse design system and component library.
TypeScript
73
star
24

pastila

Paste toy-service on top of ClickHouse
HTML
62
star
25

homebrew-clickhouse

ClickHouse Homebrew tap (old repository, unused)
57
star
26

clickhouse-tableau-connector-jdbc

Tableau connector to ClickHouse using JDBC driver
JavaScript
57
star
27

clickpy

PyPI analytics powered by ClickHouse
JavaScript
52
star
28

power-bi-clickhouse

This connector allows you to retrieve data from ClickHouse directly into Power BI for analysis and visualization
45
star
29

reversedns.space

https://reversedns.space/
HTML
44
star
30

clickhouse-academy

ClickHouse Academy training and certification
Python
39
star
31

libhdfs3

HDFS file read access for ClickHouse
C++
33
star
32

ch-bench

Benchmarks for ch
Go
30
star
33

sysroot

Files for cross-compilation
C
27
star
34

CryptoHouse

Artifacts including queries and materialized views used for CryptoHouse
TypeScript
26
star
35

HouseClick

House prices app
JavaScript
23
star
36

ch2rs

Generate Rust structs from ClickHouse rows
Rust
21
star
37

terraform-provider-clickhouse

Terraform Provider for ClickHouse Cloud
Go
21
star
38

web-tables-demo

15
star
39

clickhub

Github analytics powered by the world's fastest real-time analytics database
Python
13
star
40

icudata

Pregenerated data for ICU library
Assembly
11
star
41

clickhouse-website-worker

TypeScript
9
star
42

keeper-extend-cluster

Experiment on how to upgrade single-node clickhouse-keeper to a cluster
Makefile
7
star
43

copier

clickhouse-copier (obsolete)
C++
7
star
44

1trc

1 trillion rows
Python
7
star
45

checkout

Wrapper around actions/checkout for flexible tuning
6
star
46

laion

Supporting code for inserting and searching laion in ClickHouse
Python
5
star
47

bedrock_rag

A simple RAG pipeline for Google Analytics with ClickHouse and Bedrock
Python
5
star
48

aretestsgreenyet

A single-page website to display the status of the open-source ClickHouse CI system.
HTML
4
star
49

fuzz-corpus

Corpuses for libFuzzer-type fuzzers
4
star
50

clickhouse-playground-old

4
star
51

ch-async-inserts-demo

Demo on how to create a Node API that sends data to CH via Async inserts
TypeScript
3
star
52

grpc

Stripped version of grpc
C++
3
star
53

clickhouse-website-content

JavaScript
3
star
54

clickhouse-recipes

Sample code for solving common problems with ClickHouse
Python
3
star
55

clickhouse_vs_snowflake

HTML
2
star
56

clickhouse-com-content

HTML
2
star
57

clickhouse.github.io

HTML
2
star
58

antlr4-runtime

Subtree of antlr4 original repo
C++
2
star
59

kafka-samples

Sample datasets for Kafka
Python
2
star
60

ssl

Minimized libressl
C
2
star
61

libpq

Copy of https://github.com/postgres/postgres/tree/master/src/interfaces/libpq with some files from root
C
2
star
62

llvm

Stripped version of LLVM for use in ClickHouse for runtime code generation.
C++
2
star
63

perspective-forex

Perspective with ClickHouse example using Apache Arrow
JavaScript
2
star
64

clickhouse-typescript-schema

TypeScript
2
star
65

protobuf

add protobuf for libhdfs3
C++
1
star
66

hive-metastore

For files generated with https://github.com/apache/thrift
Thrift
1
star
67

boost-extra

extra boost libs for libhdfs3
C++
1
star
68

UnixODBC

Mirror of http://www.unixodbc.org/
C
1
star
69

doc-pr-preview-test

Testing workflow to build Docusaurus previews for pull requests.
JavaScript
1
star
70

clickhouse-docs-content

1
star
71

clickhouse-blog-images

HTML
1
star
72

bzip2

Forked from https://gitlab.com/federicomenaquintero/bzip2
C
1
star
73

libgsasl

https://www.gnu.org/software/gsasl/
C
1
star
74

readthedocs-stub

HTML
1
star
75

boost

Minimized boost lib
C++
1
star
76

clickhouse-repos-manager

a config and artifacts for packages.clickhouse.com
Python
1
star
77

clickhouse-kafka-transforms

This is meant to hold Clickhouse created kafka transforms.
Java
1
star
78

clickhouse-fivetran-destination

ClickHouse Cloud Fivetran Destination
Go
1
star
79

clickhouse-test.github.io

HTML
1
star
80

rust_vendor

Vendor files from rust dependencies
Rust
1
star
81

simple-logging-benchmark

A simple ClickHouse benchmark for the logging usecase
Python
1
star