• Stars
    star
    270
  • Rank 152,189 (Top 3 %)
  • Language
    Rust
  • License
    Apache License 2.0
  • Created about 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Official pure Rust typed client for ClickHouse DB

clickhouse.rs

A typed client for ClickHouse.

Crates.io Documentation MIT licensed Build Status

  • Uses serde for encoding/decoding rows.
  • Uses RowBinary encoding.
  • Supports HTTP and HTTPS.
  • Provides API for selecting.
  • Provides API for inserting.
  • Provides API for infinite transactional (see below) inserting.
  • Provides API for watching live views.
  • Compression and decompression (LZ4).
  • Provides mocks for unit testing.

Usage

To use the crate, add this to your Cargo.toml:

[dependencies]
clickhouse = "0.11.5"

[dev-dependencies]
clickhouse = { version = "0.11.5", features = ["test-util"] }

Note about ClickHouse prior to v22.6

CH server older than v22.6 (2022-06-16) handles RowBinary incorrectly in some rare cases. Enable wa-37420 feature to solve this problem. Don't use it for newer versions.

Create a client

use clickhouse::Client;

let client = Client::default()
    .with_url("http://localhost:8123")
    .with_user("name")
    .with_password("123")
    .with_database("test");
  • Reuse created clients or clone them in order to reuse a connection pool.

Select rows

use serde::Deserialize;
use clickhouse::Row;

#[derive(Row, Deserialize)]
struct MyRow<'a> {
    no: u32,
    name: &'a str,
}

let mut cursor = client
    .query("SELECT ?fields FROM some WHERE no BETWEEN ? AND ?")
    .bind(500)
    .bind(504)
    .fetch::<MyRow<'_>>()?;

while let Some(row) = cursor.next().await? { .. }
  • Placeholder ?fields is replaced with no, name (fields of Row).
  • Placeholder ? is replaced with values in following bind() calls.
  • Convenient fetch_one::<Row>() and fetch_all::<Row>() can be used to get a first row or all rows correspondingly.
  • sql::Identifier can be used to bind table names.

Note that cursors can return an error even after producing some rows. To avoid this, use client.with_option("wait_end_of_query", "1") in order to enable buffering on the server-side. More details. The buffer_size option can be useful too.

Insert a batch

use serde::Serialize;
use clickhouse::Row;

#[derive(Row, Serialize)]
struct MyRow {
    no: u32,
    name: String,
}

let mut insert = client.insert("some")?;
insert.write(&MyRow { no: 0, name: "foo".into() }).await?;
insert.write(&MyRow { no: 1, name: "bar".into() }).await?;
insert.end().await?;
  • If end() isn't called, the INSERT is aborted.
  • Rows are being sent progressively to spread network load.
  • ClickHouse inserts batches atomically only if all rows fit in the same partition and their number is less max_insert_block_size.
  • ch2rs is useful to generate a row type from ClickHouse.

Infinite inserting

let mut inserter = client.inserter("some")?
    .with_timeouts(Some(Duration::from_secs(5)), Some(Duration::from_secs(20)))
    .with_max_entries(750_000)
    .with_period(Some(Duration::from_secs(15)));

inserter.write(&MyRow { no: 0, name: "foo".into() }).await?;
inserter.write(&MyRow { no: 1, name: "bar".into() }).await?;
let stats = inserter.commit().await?;
if stats.entries > 0 {
    println!(
        "{} entries ({} transactions) have been inserted",
        stats.entries, stats.transactions,
    );
}
  • Inserter ends an active insert in commit() if thresholds (max_entries, period) are reached.
  • The interval between ending active INSERTs can be biased by using with_period_bias to avoid load spikes by parallel inserters.
  • All rows between commit() calls are inserted in the same INSERT statement.
  • Do not forget to flush if you want to terminate inserting:
inserter.end().await?;

Perform DDL

client.query("DROP TABLE IF EXISTS some").execute().await?;

Live views

Requires the watch feature.

let mut cursor = client
    .watch("SELECT max(no), argMax(name, no) FROM some")
    .fetch::<Row<'_>>()?;

let (version, row) = cursor.next().await?.unwrap();
println!("live view updated: version={}, row={:?}", version, row);

// Use `only_events()` to iterate over versions only.
let mut cursor = client.watch("some_live_view").limit(20).only_events().fetch()?;
println!("live view updated: version={:?}", cursor.next().await?);
  • Use carefully.
  • This code uses or creates if not exists a temporary live view named lv_{sha1(query)} to reuse the same live view by parallel watchers.
  • You can specify a name instead of a query.
  • This API uses JSONEachRowWithProgress under the hood because of the issue.
  • Only struct rows can be used. Avoid fetch::<u64>() and other without specified names.

See examples.

Feature Flags

  • lz4 (enabled by default) — enables Compression::Lz4 and Compression::Lz4Hc(_) variants. If enabled, Compression::Lz4 is used by default for all queries except for WATCH.
  • tls (enabled by default) — supports urls with the HTTPS schema.
  • test-util — adds mocks. See the example. Use it only in dev-dependencies.
  • watch — enables client.watch functionality. See the corresponding section for details.
  • uuid — adds serde::uuid to work with uuid crate.
  • time — adds serde::time to work with time crate.
  • wa-37420 — implements a workaround for CH versions prior to v22.6. See the corresponding section for details.

Data Types

  • (U)Int(8|16|32|64|128) maps to/from corresponding (u|i)(8|16|32|64|128) types or newtypes around them.

  • (U)Int256 aren't supported directly, but there is a workaround for it.

  • Float(32|64) maps to/from corresponding f(32|64) or newtypes around them.

  • Decimal(32|64|128) maps to/from corresponding i(32|64|128) or newtypes around them. It's more convenient to use fixnum or another implementation of signed fixed-point numbers.

  • Boolean maps to/from bool or newtypes around it.

  • String maps to/from any string or bytes types, e.g. &str, &[u8], String, Vec<u8> or SmartString. Newtypes are also supported. To store bytes, consider using serde_bytes, because it's more efficient.

    Example
    #[derive(Debug, Serialize, Deserialize)]
    struct MyRow<'a> {
        str: &'a str,
        string: String,
        #[serde(with = "serde_bytes")]
        bytes: Vec<u8>,
        #[serde(with = "serde_bytes")]
        byte_slice: &'a [u8],
    }
  • FixedString(_) isn't supported yet.

  • Enum(8|16) are supported using serde_repr.

    Example
    use serde_repr::{Deserialize_repr, Serialize_repr};
    
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        level: Level,
    }
    
    #[derive(Debug, Serialize_repr, Deserialize_repr)]
    #[repr(u8)]
    enum Level {
        Debug = 1,
        Info = 2,
        Warn = 3,
        Error = 4,
    }
  • UUID maps to/from uuid::Uuid by using serde::uuid. Requires the uuid feature.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        #[serde(with = "clickhouse::serde::uuid")]
        uuid: uuid::Uuid,
    }
  • IPv6 maps to/from std::net::Ipv6Addr.

  • IPv4 maps to/from std::net::Ipv4Addr by using serde::ipv4.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        #[serde(with = "clickhouse::serde::ipv4")]
        ipv4: std::net::Ipv4Addr,
    }
  • Date maps to/from u16 or a newtype around it and represents a number of days elapsed since 1970-01-01. Also, time::Date is supported by using serde::time::date, that requires the time feature.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        days: u16,
        #[serde(with = "clickhouse::serde::time::date")]
        date: Date,
    }
  • Date32 maps to/from i32 or a newtype around it and represents a number of days elapsed since 1970-01-01. Also, time::Date is supported by using serde::time::date32, that requires the time feature.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        days: i32,
        #[serde(with = "clickhouse::serde::time::date32")]
        date: Date,
    }
  • DateTime maps to/from u32 or a newtype around it and represents a number of seconds elapsed since UNIX epoch. Also, time::OffsetDateTime is supported by using serde::time::datetime, that requires the time feature.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        ts: u32,
        #[serde(with = "clickhouse::serde::time::datetime")]
        dt: OffsetDateTime,
    }
  • DateTime64(_) maps to/from i32 or a newtype around it and represents a time elapsed since UNIX epoch. Also, time::OffsetDateTime is supported by using serde::time::datetime64::*, that requires the time feature.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        ts: i64, // elapsed s/us/ms/ns depending on `DateTime64(X)`
        #[serde(with = "clickhouse::serde::time::datetime64::secs")]
        dt64s: OffsetDateTime,  // `DateTime64(0)`
        #[serde(with = "clickhouse::serde::time::datetime64::millis")]
        dt64ms: OffsetDateTime, // `DateTime64(3)`
        #[serde(with = "clickhouse::serde::time::datetime64::micros")]
        dt64us: OffsetDateTime, // `DateTime64(6)`
        #[serde(with = "clickhouse::serde::time::datetime64::nanos")]
        dt64ns: OffsetDateTime, // `DateTime64(9)`
    }
  • Typle(A, B, ...) maps to/from (A, B, ...) or a newtype around it.

  • Array(_) maps to/from any slice, e.g. Vec<_>, &[_]. Newtypes are also supported.

  • Map(K, V) behaves like Array((K, V)).

  • LowCardinality(_) is supported seamlessly.

  • Nullable(_) maps to/from Option<_>. For clickhouse::serde::* helpers add ::option.

    Example
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        #[serde(with = "clickhouse::serde::ipv4::option")]
        ipv4_opt: Option<Ipv4Addr>,
    }
  • Nested is supported by providing multiple arrays with renaming.

    Example
    // CREATE TABLE test(items Nested(name String, count UInt32))
    #[derive(Row, Serialize, Deserialize)]
    struct MyRow {
        #[serde(rename = "items.name")]
        items_name: Vec<String>,
        #[serde(rename = "items.count")]
        items_count: Vec<u32>,
    }
  • JSON and Geo aren't supported for now.

Mocking

The crate provides utils for mocking CH server and testing DDL, SELECT, INSERT and WATCH queries.

The functionality can be enabled with the test-util feature. Use it only in dev-dependencies.

See the example.

More Repositories

1

ClickHouse

ClickHouse® is a real-time analytics DBMS
C++
37,339
star
2

clickhouse-go

Golang driver for ClickHouse
Go
2,853
star
3

clickhouse-java

Java client and JDBC driver for ClickHouse
Java
1,324
star
4

clickhouse-presentations

Presentations, meetups and talks about ClickHouse
HTML
981
star
5

ClickBench

ClickBench: a Benchmark For Analytical Databases
HTML
642
star
6

metabase-clickhouse-driver

ClickHouse database driver for the Metabase business intelligence front-end
Clojure
439
star
7

clickhouse_exporter

This is a simple server that periodically scrapes ClickHouse stats and exports them via HTTP for Prometheus(https://prometheus.io/) consumption.
Go
361
star
8

ch-go

Low-level Go Client for ClickHouse
Go
314
star
9

clickhouse-connect

Python driver/sqlalchemy/superset connectors
Python
308
star
10

clickhouse-cpp

C++ client library for ClickHouse
C
300
star
11

NoiSQL

NoiSQL — Generating Music With SQL Queries
SQL
276
star
12

graphouse

Graphouse allows you to use ClickHouse as a Graphite storage.
Java
259
star
13

clickhouse-odbc

ODBC driver for ClickHouse
C
246
star
14

dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Python
245
star
15

adsb.exposed

Interactive visualization and analytics on ADS-B data with ClickHouse
HTML
223
star
16

clickhouse-js

Official JS client for ClickHouse DB
TypeScript
212
star
17

spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
Scala
180
star
18

clickhouse-jdbc-bridge

A JDBC proxy from ClickHouse to external databases
Java
166
star
19

clickhouse-kafka-connect

ClickHouse Kafka Connector
Java
145
star
20

github-explorer

Everything You Always Wanted To Know About GitHub (But Were Afraid To Ask)
HTML
141
star
21

examples

ClickHouse Examples
Jupyter Notebook
118
star
22

clickhouse-docs

Official documentation for ClickHouse
JavaScript
110
star
23

click-ui

The home of the ClickHouse design system and component library.
TypeScript
73
star
24

pastila

Paste toy-service on top of ClickHouse
HTML
62
star
25

homebrew-clickhouse

ClickHouse Homebrew tap (old repository, unused)
57
star
26

clickhouse-tableau-connector-jdbc

Tableau connector to ClickHouse using JDBC driver
JavaScript
57
star
27

clickpy

PyPI analytics powered by ClickHouse
JavaScript
52
star
28

power-bi-clickhouse

This connector allows you to retrieve data from ClickHouse directly into Power BI for analysis and visualization
45
star
29

reversedns.space

https://reversedns.space/
HTML
44
star
30

clickhouse-academy

ClickHouse Academy training and certification
Python
39
star
31

libhdfs3

HDFS file read access for ClickHouse
C++
33
star
32

ch-bench

Benchmarks for ch
Go
30
star
33

sysroot

Files for cross-compilation
C
27
star
34

CryptoHouse

Artifacts including queries and materialized views used for CryptoHouse
TypeScript
26
star
35

HouseClick

House prices app
JavaScript
23
star
36

ch2rs

Generate Rust structs from ClickHouse rows
Rust
21
star
37

terraform-provider-clickhouse

Terraform Provider for ClickHouse Cloud
Go
21
star
38

web-tables-demo

15
star
39

clickhub

Github analytics powered by the world's fastest real-time analytics database
Python
13
star
40

icudata

Pregenerated data for ICU library
Assembly
11
star
41

clickhouse-website-worker

TypeScript
9
star
42

keeper-extend-cluster

Experiment on how to upgrade single-node clickhouse-keeper to a cluster
Makefile
7
star
43

copier

clickhouse-copier (obsolete)
C++
7
star
44

1trc

1 trillion rows
Python
7
star
45

checkout

Wrapper around actions/checkout for flexible tuning
6
star
46

laion

Supporting code for inserting and searching laion in ClickHouse
Python
5
star
47

bedrock_rag

A simple RAG pipeline for Google Analytics with ClickHouse and Bedrock
Python
5
star
48

aretestsgreenyet

A single-page website to display the status of the open-source ClickHouse CI system.
HTML
4
star
49

fuzz-corpus

Corpuses for libFuzzer-type fuzzers
4
star
50

clickhouse-playground-old

4
star
51

ch-async-inserts-demo

Demo on how to create a Node API that sends data to CH via Async inserts
TypeScript
3
star
52

grpc

Stripped version of grpc
C++
3
star
53

clickhouse-website-content

JavaScript
3
star
54

clickhouse-recipes

Sample code for solving common problems with ClickHouse
Python
3
star
55

clickhouse_vs_snowflake

HTML
2
star
56

clickhouse-com-content

HTML
2
star
57

clickhouse.github.io

HTML
2
star
58

antlr4-runtime

Subtree of antlr4 original repo
C++
2
star
59

kafka-samples

Sample datasets for Kafka
Python
2
star
60

ssl

Minimized libressl
C
2
star
61

libpq

Copy of https://github.com/postgres/postgres/tree/master/src/interfaces/libpq with some files from root
C
2
star
62

llvm

Stripped version of LLVM for use in ClickHouse for runtime code generation.
C++
2
star
63

perspective-forex

Perspective with ClickHouse example using Apache Arrow
JavaScript
2
star
64

clickhouse-typescript-schema

TypeScript
2
star
65

protobuf

add protobuf for libhdfs3
C++
1
star
66

hive-metastore

For files generated with https://github.com/apache/thrift
Thrift
1
star
67

boost-extra

extra boost libs for libhdfs3
C++
1
star
68

UnixODBC

Mirror of http://www.unixodbc.org/
C
1
star
69

doc-pr-preview-test

Testing workflow to build Docusaurus previews for pull requests.
JavaScript
1
star
70

clickhouse-docs-content

1
star
71

clickhouse-blog-images

HTML
1
star
72

bzip2

Forked from https://gitlab.com/federicomenaquintero/bzip2
C
1
star
73

libgsasl

https://www.gnu.org/software/gsasl/
C
1
star
74

readthedocs-stub

HTML
1
star
75

boost

Minimized boost lib
C++
1
star
76

clickhouse-repos-manager

a config and artifacts for packages.clickhouse.com
Python
1
star
77

clickhouse-kafka-transforms

This is meant to hold Clickhouse created kafka transforms.
Java
1
star
78

clickhouse-fivetran-destination

ClickHouse Cloud Fivetran Destination
Go
1
star
79

clickhouse-test.github.io

HTML
1
star
80

rust_vendor

Vendor files from rust dependencies
Rust
1
star
81

simple-logging-benchmark

A simple ClickHouse benchmark for the logging usecase
Python
1
star