• Stars
    star
    944
  • Rank 48,415 (Top 1.0 %)
  • Language
    C++
  • License
    Apache License 2.0
  • Created almost 6 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial

TiFlash

tiflash-architecture

TiFlash is a columnar storage component of TiDB and TiDB Cloud, the fully-managed service of TiDB. It mainly plays the role of Analytical Processing (AP) in the Hybrid Transactional/Analytical Processing (HTAP) architecture of TiDB.

TiFlash stores data in columnar format and synchronizes data updates in real-time from TiKV by Raft logs with sub-second latency. Reads in TiFlash are guaranteed transactionally consistent with Snapshot Isolation level. TiFlash utilizes Massively Parallel Processing (MPP) computing architecture to accelerate the analytical workloads.

TiFlash repository is based on ClickHouse. We appreciate the excellent work of the ClickHouse team.

Quick Start

Start with TiDB Cloud

Quickly explore TiFlash with a free trial of TiDB Cloud.

See TiDB Cloud Quick Start Guide.

Start with TiDB

See Quick Start with HTAP and Use TiFlash.

Build TiFlash

TiFlash can be built on the following hardware architectures:

  • x86-64 / amd64
  • aarch64

And the following operating systems:

  • Linux
  • MacOS

1. Prepare Prerequisites

The following packages are required:

  • CMake 3.21.0+
  • Clang 14.0.0+
  • Rust
  • Python 3.0+
  • Ninja-Build or GNU Make
  • Ccache (not necessary but highly recommended to reduce rebuild time)

Detailed steps for each platform are listed below.

Ubuntu / Debian
sudo apt update

# Install Rust toolchain, see https://rustup.rs for details
curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none
source $HOME/.cargo/env

# Install LLVM, see https://apt.llvm.org for details
# Clang will be available as /usr/bin/clang++-15
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 15 all

# Install other dependencies
sudo apt install -y cmake ninja-build zlib1g-dev libcurl4-openssl-dev ccache

Note for Ubuntu 18.04 and Ubuntu 20.04:

The default installed cmake may be not recent enough. You can install a newer cmake from the Kitware APT Repository:

sudo apt install -y software-properties-common lsb-release
wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/kitware.gpg >/dev/null
sudo apt-add-repository "deb https://apt.kitware.com/ubuntu/ $(lsb_release -cs) main"
sudo apt update
sudo apt install -y cmake

If you are facing "ld.lld: error: duplicate symbol: ssl3_cbc_digest_record":

It is likely because you have a pre-installed libssl3 where TiFlash prefers libssl1. TiFlash has vendored libssl1, so that you can simply remove the one in the system to make compiling work:

sudo apt remove libssl-dev

If this doesn't work, please file an issue.

Archlinux
# Install Rust toolchain, see https://rustup.rs for details
curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none
source $HOME/.cargo/env

# Install compilers and dependencies
sudo pacman -S clang lld libc++ libc++abi compiler-rt openmp lcov cmake ninja curl openssl zlib llvm ccache
CentOS 7

Please refer to release-centos7-llvm/env/prepare-sysroot.sh

MacOS
# Install Rust toolchain, see https://rustup.rs for details
curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none
source $HOME/.cargo/env

# Install compilers
xcode-select --install

# Install other dependencies
brew install ninja cmake [email protected] ccache

If your MacOS is higher or equal to 13.0, it should work out of the box because by default Apple clang is 14.0.0. But if your MacOS is lower than 13.0, you should install llvm clang manually.

brew install llvm@15

# check llvm version
clang --version # should be 15.0.0 or higher

2. Checkout Source Code

git clone https://github.com/pingcap/tiflash.git --recursive -j 20
cd tiflash

3. Build

To build TiFlash for development:

# In the TiFlash repository root:
mkdir cmake-build-debug  # The directory name can be customized
cd cmake-build-debug

cmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG

ninja tiflash

Note: In Linux, usually you need to explicitly specify to use LLVM.

# In cmake-build-debug directory:
cmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG \
  -DCMAKE_C_COMPILER=/usr/bin/clang-14 \
  -DCMAKE_CXX_COMPILER=/usr/bin/clang++-14

In MacOS, if you install llvm clang, you need to explicitly specify to use llvm clang.

Add the following lines to your shell environment, e.g. ~/.bash_profile.

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export CC="/opt/homebrew/opt/llvm/bin/clang"
export CXX="/opt/homebrew/opt/llvm/bin/clang++"

Or use CMAKE_C_COMPILER and CMAKE_CXX_COMPILER to specify the compiler, like this:

cmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++

After building, you can get TiFlash binary in dbms/src/Server/tiflash in the cmake-build-debug directory.

Build Options

TiFlash has several CMake build options to tweak for development purposes. These options SHOULD NOT be changed for production usage, as they may introduce unexpected build errors and unpredictable runtime behaviors.

To tweak options, pass one or multiple -D...=... args when invoking CMake, for example:

cd cmake-build-debug
cmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG -DFOO=BAR
                                          ^^^^^^^^^
  • Build Type:

    • -DCMAKE_BUILD_TYPE=RELWITHDEBINFO: Release build with debug info (default)

    • -DCMAKE_BUILD_TYPE=DEBUG: Debug build

    • -DCMAKE_BUILD_TYPE=RELEASE: Release build

    Usually you may want to use different build directories for different build types, e.g. a new build directory named cmake-build-release for the release build, so that compile unit cache will not be invalidated when you switch between different build types.

  • Build with Unit Tests:

    • -DENABLE_TESTS=ON: Enable unit tests (enabled by default in debug profile)

    • -DENABLE_TESTS=OFF: Disable unit tests (default in release profile)

  • Build using GNU Make instead of ninja-build:

    Click to expand instructions

    To use GNU Make, simply don't pass -GNinja to cmake:

    cd cmake-build-debug
    cmake .. -DCMAKE_BUILD_TYPE=DEBUG
    make tiflash -j

    NOTE: Option -j (defaults to your system CPU core count, otherwise you can optionally specify a number) is used to control the build parallelism. Higher parallelism consumes more memory. If you encounter compiler OOM or hang, try to lower the parallelism by specifying a reasonable number, e.g., half of your system CPU core count or even smaller, after -j, depending on the available memory in your system.

  • Build with System Libraries:

    Click to expand instructions

    For local development, it is sometimes handy to use pre-installed third-party libraries in the system, rather than to compile them from sources of the bundled (internal) submodules.

    Options are supplied to control whether to use internal third-party libraries (bundled in TiFlash) or to try using the pre-installed system ones.

    WARNING: It is NOT guaranteed that TiFlash would still build if any of the system libraries are used. Build errors are very likely to happen, almost all the time.

    You can view these options along with their descriptions by running:

    cd cmake-build-debug
    cmake -LH | grep "USE_INTERNAL" -A3

    All of these options are default as ON, as the names tell, using the internal libraries and build from sources.

    There is another option to append extra paths for CMake to find system libraries:

    • PREBUILT_LIBS_ROOT: Default as empty, can be specified with multiple values, seperated by ;
  • Build for AMD64 Architecture:

    Click to expand instructions

    To deploy TiFlash under the Linux AMD64 architecture, the CPU must support the AVX2 instruction set. Ensure that cat /proc/cpuinfo | grep avx2 has output.

    If need to build TiFlash for AMD64 architecture without such instruction set, please use cmake option -DNO_AVX_OR_HIGHER=ON.

Run Unit Tests

Unit tests are automatically enabled in debug profile. To build these unit tests:

cd cmake-build-debug
cmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG
ninja gtests_dbms       # Most TiFlash unit tests
ninja gtests_libdaemon  # Settings related tests
ninja gtests_libcommon

Then, to run these unit tests:

cd cmake-build-debug
./dbms/gtests_dbms
./libs/libdaemon/src/tests/gtests_libdaemon
./libs/libcommon/src/tests/gtests_libcommon

More usages are available via ./dbms/gtests_dbms --help.

Run Sanitizer Tests

TiFlash supports testing with thread sanitizer and address sanitizer.

To build unit test executables with sanitizer enabled:

# In the TiFlash repository root:
mkdir cmake-build-sanitizer
cd cmake-build-sanitizer
cmake .. -GNinja -DENABLE_TESTS=ON -DCMAKE_BUILD_TYPE=ASan # or TSan
ninja gtests_dbms
ninja gtests_libdaemon
ninja gtests_libcommon

There are known false positives reported from leak sanitizer (which is included in address sanitizer). To suppress these errors, set the following environment variables before running the executables:

LSAN_OPTIONS=suppressions=test/sanitize/asan.suppression

Run Integration Tests

  1. Build your own TiFlash binary using debug profile:

    cd cmake-build-debug
    cmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG
    ninja tiflash
  2. Start a local TiDB cluster with your own TiFlash binary using TiUP:

    cd cmake-build-debug
    tiup playground nightly --tiflash.binpath ./dbms/src/Server/tiflash
    
    # Or using a more stable cluster version:
    # tiup playground v6.1.0 --tiflash.binpath ./dbms/src/Server/tiflash

    TiUP is the TiDB component manager. If you don't have one, you can install it via:

    curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh

    If you are not running the cluster using the default port (for example, you run multiple clusters), make sure that the port and build directory in tests/_env.sh are correct.

  3. Run integration tests:

    # In the TiFlash repository root:
    cd tests
    ./run-test.sh
    
    # Or run specific integration test:
    # ./run-test.sh fullstack-test2/ddl

Note: some integration tests (namely, tests under delta-merge-test) requires a standalone TiFlash service without a TiDB cluster, otherwise they will fail. To run these integration tests: TBD

Run MicroBenchmark Tests

To build micro benchmark tests, you need release profile and tests enabled:

# In the TiFlash repository root:
mkdir cmake-build-release
cd cmake-build-release
cmake .. -GNinja -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_TESTS=ON
ninja bench_dbms

Then, to run these micro benchmarks:

cd cmake-build-release
./dbms/bench_dbms

# Or run with filter:
# ./dbms/bench_dbms --benchmark_filter=xxx

More usages are available via ./dbms/bench_dbms --help.

Generate LLVM Coverage Report

TBD.

Contributing

Here is the overview of TiFlash architecture The architecture of TiFlash's distributed storage engine and transaction layer.

See TiFlash Development Guide and TiFlash Design documents.

Before submitting a pull request, please resolve clang-tidy errors and use format-diff.py to format source code, otherwise CI build may raise error.

NOTE: It is required to use clang-format 12.0.0+.

# In the TiFlash repository root:
merge_base=$(git merge-base upstream/master HEAD)
python3 release-centos7-llvm/scripts/run-clang-tidy.py -p cmake-build-debug -j 20 --files `git diff $merge_base --name-only`
# if there are too much errors, you can try to run the script again with `-fix`
python3 format-diff.py --diff_from $merge_base

License

TiFlash is under the Apache 2.0 license. See the LICENSE file for details.

More Repositories

1

tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
Go
37,200
star
2

talent-plan

open source training courses about distributed database and distributed systems
Rust
10,112
star
3

awesome-database-learning

A list of learning materials to understand databases internals
9,339
star
4

docs-cn

TiDB/TiKV/PD ไธญๆ–‡ๆ–‡ๆกฃ
Shell
1,811
star
5

ossinsight

Analysis, Comparison, Trends, Rankings of Open Source Software, you can also get insight from more than 7 billion with natural language (powered by OpenAI). Follow us on Twitter: https://twitter.com/ossinsight
TypeScript
1,744
star
6

parser

A MySQL Compatible SQL Parser
Go
1,409
star
7

tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
Go
1,221
star
8

tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV
Scala
883
star
9

failpoint

An implementation of failpoints for Golang.
Go
816
star
10

go-ycsb

A Go port of Yahoo! Cloud Serving Benchmark (YCSB)
Go
594
star
11

docs

TiDB database documentation. TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
Python
584
star
12

dm

Data Migration Platform
Go
456
star
13

tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Go
426
star
14

tiup

A component manager for TiDB
Go
417
star
15

ossinsight-lite

๐Ÿšง[WIP] Yet another customizable free GitHub stats dashboard based on TiDB Serverless: https://ossinsight-lite.vercel.app, hand-drawn style.
TypeScript
383
star
16

presentations

367
star
17

tidb-docker-compose

Python
352
star
18

tidb-ansible

Python
326
star
19

tidb-binlog

A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.
Go
292
star
20

tla-plus

TLA
292
star
21

tidb-tools

tidb-tools are some useful tool collections for TiDB.
Go
286
star
22

dumpling

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).
Go
281
star
23

community

TiDB community content
260
star
24

chaos

A test framework for linearizability check with Go
Go
206
star
25

tidb.ai

https://TiDB.AI is a Graph RAG based and conversational knowledge base tool built with TiDB Serverless Vector Storage and LlamaIndex. Open source and free to use.
TypeScript
186
star
26

go-tpc

A toolbox to benchmark TPC workloads in Go
Go
177
star
27

tidb-dashboard

A Web UI for monitoring, diagnosing and managing the TiDB cluster.
TypeScript
175
star
28

kvproto

Protocol buffer files for TiKV
CMake
152
star
29

tidb-lightning

This repository has been moved to https://github.com/pingcap/br
Go
142
star
30

tipocket

A toolkit for testing TiDB
Go
141
star
31

blog-cn

Shell
126
star
32

br

A command-line tool for distributed backup and restoration of the TiDB cluster data
Go
123
star
33

tidb-dev-guide

A comprehensive development guide to help you be more and more familiar with the TiDB community and become an expert finally.
118
star
34

tidb-bench

A Simple Benchmark For TiDB
C
108
star
35

gdocwiki

A wiki based on Google Doc / Drive
TypeScript
102
star
36

tidb-map

A series of maps to help users and contributors
95
star
37

tipb

TiDB protobuf
CMake
92
star
38

style-guide

Style guide for PingCAP and TiKV code
80
star
39

benchmarksql

Unofficial mirror of benchmarksql on github
Java
79
star
40

go-randgen

a QA tool to random generate sql by bnf pattern
Go
75
star
41

mysql-tester

A Golang implementation of MySQL Test Framework
Go
63
star
42

weekly

57
star
43

tidb-prisma-vercel-demo

Virtual online bookstore application demo which you can find books of various categories and rate the books.
TypeScript
56
star
44

tiproxy

Go
56
star
45

advanced-statefulset

Go
55
star
46

blog

Python
49
star
47

docs-tidb-operator

Documentation for TiDB on Kubernetes. TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
Python
47
star
48

tikv-client-lib-java

TiKV Java client library
Java
44
star
49

tidiff

A toolset to improve efficiency
Go
41
star
50

meetup

37
star
51

fn

Go
35
star
52

tidb-vector-python

TiDB Vector SDK for Python, including code examples. Join our Discord: https://discord.gg/XzSW23Jg9p
Python
35
star
53

tiunimanager

TiUniManager
Go
34
star
54

thirdparty-ops

This repo is used for the operation and maintenance of third party tools.
Python
32
star
55

dead-mans-switch

A bypass monitoring prober
Go
32
star
56

ng-monitoring

Go
31
star
57

django-tidb

TiDB dialect for Django
Python
30
star
58

tidb-course-201-lab

Lab scripts for the PingCAP training course: TiDB SQL for Developers.
Python
30
star
59

tidb-vision

TiDB data visualization
JavaScript
28
star
60

tidb-inspect-tools

Python
27
star
61

monitoring

Shell
26
star
62

activerecord-tidb-adapter

TiDB adapter for ActiveRecord, allows the use of TiDB as a backend for ActiveRecord and Rails apps.
Ruby
24
star
63

diag

A tool to collect diagnostic data from TiDB Clusters
Go
24
star
64

docs-dm

Documentation for the TiDB Data Migration (DM) tool in both English and Chinese.
Python
23
star
65

LinguFlow

LinguFlow, a low-code tool designed for LLM application development, simplifies the building, debugging, and deployment process for developers.
TypeScript
23
star
66

website-docs

The next generation of PingCAP Docs. Powered by Gatsby โš›๏ธ.
TypeScript
22
star
67

book.tidb.net

JavaScript
22
star
68

kdt

Kernel Debug Toolkit
Shell
20
star
69

log

Go
17
star
70

octopus

A toolkit including many powerful distributed test tools
Go
15
star
71

Auto-GPT-TiDB-Serverless-Plugin

Python
15
star
72

errcode

Go
14
star
73

dbt-tidb

A dbt adapter for TiDB
Python
14
star
74

tidb_workload_analysis

Go
14
star
75

k8s-fluent-bit-stackdriver

Shell
11
star
76

website

The website of PingCAP. Powered by Gatsby โš›๏ธ and Rocket ๐Ÿš€.
JavaScript
11
star
77

tpcc-mysql

forked from https://code.launchpad.net/~percona-dev/perconatools/tpcc-mysql
C
11
star
78

tidb-insight

Python
11
star
79

tidb-loadbalance

Java
10
star
80

tso

Timestamp Oracle
Go
9
star
81

tiunimanager-ui

A web UI for TiUniManager
TypeScript
9
star
82

tidb-ctl

TiDB Controller
Go
9
star
83

hackernews-insight

Chat to query Hacker News database, based on Auto-GPT and TiDB Cloud Serverless Database
TypeScript
9
star
84

tidb-cloud-backup

Go
8
star
85

wordpress-tidb-plugin

PHP
8
star
86

docs-appdev

Python
7
star
87

wordpress-tidb-docker

WordPress x TiDB Serverless Tier Cluster
Shell
7
star
88

tidb-academy-labs

6
star
89

etcdv3-gateway

Gateway for etcdv3
Go
6
star
90

tispark-test

C
6
star
91

sysutil

sysutil is a library which implementats the gRPC service Diagnostics and shares the diagnostics functions between TiDB and PD.
Go
6
star
92

sqlalchemy-tidb

Python
5
star
93

oasis

Python
5
star
94

homebrew-brew

Homebrew taps for TiDB
Ruby
5
star
95

mysqlrelay

Go
4
star
96

tidb-lmdb

lmdb as storage engine for tidb
Go
4
star
97

cloud-assets-utils

Cloud assets utils by PingCAP FE.
OCaml
4
star
98

mpdriver

MySQL Protocol Driver, used to record MySQL query commands..
Go
4
star
99

tidb-helper

Shell
3
star
100

vldb-boss-2018

Slides and links for VLDB BOSS 2018
3
star