• Stars
    star
    11,640
  • Rank 2,747 (Top 0.06 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created almost 7 years ago
  • Updated 15 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Apache Doris is an easy-to-use, high performance and unified analytics database.

Apache Doris

License GitHub release Jenkins Vec Total Lines Join the Doris Community at Slack Join the chat at https://gitter.im/apache-doris/Lobby EN doc CN doc Twitter

Apache Doris is an easy-to-use, high-performance and real-time analytical database based on MPP architecture, known for its extreme speed and ease of use. It only requires a sub-second response time to return query results under massive data and can support not only high-concurrent point query scenarios but also high-throughput complex analysis scenarios.

All this makes Apache Doris an ideal tool for scenarios including report analysis, ad-hoc query, unified data warehouse, and data lake query acceleration. On Apache Doris, users can build various applications, such as user behavior analysis, AB test platform, log retrieval analysis, user portrait analysis, and order analysis.

๐ŸŽ‰ Version 2.0.0 version released now. The 2.0.0 version has achieved over 10x performance improvements on standard Benchmark, comprehensive enhancement in log analysis and lakehouse scenarios, more efficient and stable data update and write efficiency, support for more comprehensive multi-tenant and resource isolation mechanisms, and take a new step in the direction of resource elasticity and storage computing separation. It has also been added a series of usability features for enterprise users. We welcome all users who have requirements for the new features of the 2.0 version to deploy and upgrade. Check out the ๐Ÿ”—Release Notes here.

๐ŸŽ‰ Version 1.2.6 released now! It is fully evolved release and all users are encouraged to upgrade to this release. Check out the ๐Ÿ”—Release Notes here.

๐ŸŽ‰ Version 1.1.5 released now. It is a stability improvement and bugfix release based on version 1.1. Check out the ๐Ÿ”—Release Notes here.

๐Ÿ‘€ Have a look at the ๐Ÿ”—Official Website for a comprehensive list of Apache Doris's core features, blogs and user cases.

๐Ÿ“ˆ Usage Scenarios

As shown in the figure below, after various data integration and processing, the data sources are usually stored in the real-time data warehouse Apache Doris and the offline data lake or data warehouse (in Apache Hive, Apache Iceberg or Apache Hudi).

Apache Doris is widely used in the following scenarios:

  • Reporting Analysis

    • Real-time dashboards
    • Reports for in-house analysts and managers
    • Highly concurrent user-oriented or customer-oriented report analysis: such as website analysis and ad reporting that usually require thousands of QPS and quick response times measured in milliseconds. A successful user case is that Doris has been used by the Chinese e-commerce giant JD.com in ad reporting, where it receives 10 billion rows of data per day, handles over 10,000 QPS, and delivers a 99 percentile query latency of 150 ms.
  • Ad-Hoc Query. Analyst-oriented self-service analytics with irregular query patterns and high throughput requirements. XiaoMi has built a growth analytics platform (Growth Analytics, GA) based on Doris, using user behavior data for business growth analysis, with an average query latency of 10 seconds and a 95th percentile query latency of 30 seconds or less, and tens of thousands of SQL queries per day.

  • Unified Data Warehouse Construction. Apache Doris allows users to build a unified data warehouse via one single platform and save the trouble of handling complicated software stacks. Chinese hot pot chain Haidilao has built a unified data warehouse with Doris to replace its old complex architecture consisting of Apache Spark, Apache Hive, Apache Kudu, Apache HBase, and Apache Phoenix.

  • Data Lake Query. Apache Doris avoids data copying by federating the data in Apache Hive, Apache Iceberg, and Apache Hudi using external tables, and thus achieves outstanding query performance.

๐Ÿ–ฅ๏ธ Core Concepts

๐Ÿ“‚ Architecture of Apache Doris

The overall architecture of Apache Doris is shown in the following figure. The Doris architecture is very simple, with only two types of processes.

  • Frontend (FE): user request access, query parsing and planning, metadata management, node management, etc.

  • Backend (BE): data storage and query plan execution

Both types of processes are horizontally scalable, and a single cluster can support up to hundreds of machines and tens of petabytes of storage capacity. And these two types of processes guarantee high availability of services and high reliability of data through consistency protocols. This highly integrated architecture design greatly reduces the operation and maintenance cost of a distributed system.

The overall architecture of Apache Doris

In terms of interfaces, Apache Doris adopts MySQL protocol, supports standard SQL, and is highly compatible with MySQL dialect. Users can access Doris through various client tools and it supports seamless connection with BI tools.

๐Ÿ’พ Storage Engine

Doris uses a columnar storage engine, which encodes, compresses, and reads data by column. This enables a very high compression ratio and largely reduces irrelavant data scans, thus making more efficient use of IO and CPU resources. Doris supports various index structures to minimize data scans:

  • Sorted Compound Key Index: Users can specify three columns at most to form a compound sort key. This can effectively prune data to better support highly concurrent reporting scenarios.
  • Z-order Index: This allows users to efficiently run range queries on any combination of fields in their schema.
  • MIN/MAX Indexing: This enables effective filtering of equivalence and range queries for numeric types.
  • Bloom Filter: very effective in equivalence filtering and pruning of high cardinality columns
  • Invert Index: This enables fast search for any field.

๐Ÿ’ฟ Storage Models

Doris supports a variety of storage models and has optimized them for different scenarios:

  • Aggregate Key Model: able to merge the value columns with the same keys and significantly improve performance

  • Unique Key Model: Keys are unique in this model and data with the same key will be overwritten to achieve row-level data updates.

  • Duplicate Key Model: This is a detailed data model capable of detailed storage of fact tables.

Doris also supports strongly consistent materialized views. Materialized views are automatically selected and updated, which greatly reduces maintenance costs for users.

๐Ÿ” Query Engine

Doris adopts the MPP model in its query engine to realize parallel execution between and within nodes. It also supports distributed shuffle join for multiple large tables so as to handle complex queries.

The Doris query engine is vectorized, with all memory structures laid out in a columnar format. This can largely reduce virtual function calls, improve cache hit rates, and make efficient use of SIMD instructions. Doris delivers a 5โ€“10 times higher performance in wide table aggregation scenarios than non-vectorized engines.

Apache Doris uses Adaptive Query Execution technology to dynamically adjust the execution plan based on runtime statistics. For example, it can generate runtime filter, push it to the probe side, and automatically penetrate it to the Scan node at the bottom, which drastically reduces the amount of data in the probe and increases join performance. The runtime filter in Doris supports In/Min/Max/Bloom filter.

๐Ÿš… Query Optimizer

In terms of optimizers, Doris uses a combination of CBO and RBO. RBO supports constant folding, subquery rewriting, predicate pushdown and CBO supports Join Reorder. The Doris CBO is under continuous optimization for more accurate statistical information collection and derivation, and more accurate cost model prediction.

Technical Overview: ๐Ÿ”—Introduction to Apache Doris

๐ŸŽ† Why choose Apache Doris?

  • ๐ŸŽฏ Easy to Use: Two processes, no other dependencies; online cluster scaling, automatic replica recovery; compatible with MySQL protocol, and using standard SQL.

  • ๐Ÿš€ High Performance: Extremely fast performance for low-latency and high-throughput queries with columnar storage engine, modern MPP architecture, vectorized query engine, pre-aggregated materialized view and data index.

  • ๐Ÿ–ฅ๏ธ Single Unified: A single system can support real-time data serving, interactive data analysis and offline data processing scenarios.

  • โš›๏ธ Federated Querying: Supports federated querying of data lakes such as Hive, Iceberg, Hudi, and databases such as MySQL and Elasticsearch.

  • โฉ Various Data Import Methods: Supports batch import from HDFS/S3 and stream import from MySQL Binlog/Kafka; supports micro-batch writing through HTTP interface and real-time writing using Insert in JDBC.

  • ๐Ÿš™ Rich Ecology: Spark uses Spark-Doris-Connector to read and write Doris; Flink-Doris-Connector enables Flink CDC to implement exactly-once data writing to Doris; DBT Doris Adapter is provided to transform data in Doris with DBT.

๐Ÿ™Œ Contributors

Apache Doris has graduated from Apache incubator successfully and become a Top-Level Project in June 2022.

Currently, the Apache Doris community has gathered more than 400 contributors from nearly 200 companies in different industries, and the number of active contributors is close to 100 per month.

Monthly Active Contributors

Contributor over time

We deeply appreciate ๐Ÿ”—community contributors for their contribution to Apache Doris.

๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ Users

Apache Doris now has a wide user base in China and around the world, and as of today, Apache Doris is used in production environments in thousands of companies worldwide. More than 80% of the top 50 Internet companies in China in terms of market capitalization or valuation have been using Apache Doris for a long time, including Baidu, Meituan, Xiaomi, Jingdong, Bytedance, Tencent, NetEase, Kwai, Sina, 360, Mihoyo, and Ke Holdings. It is also widely used in some traditional industries such as finance, energy, manufacturing, and telecommunications.

The users of Apache Doris: ๐Ÿ”—Users

Add your company logo at Apache Doris Website: ๐Ÿ”—Add Your Company

๐Ÿ‘ฃ Get Started

๐Ÿ“š Docs

All Documentation ๐Ÿ”—Docs

โฌ‡๏ธ Download

All release and binary version ๐Ÿ”—Download

๐Ÿ—„๏ธ Compile

See how to compile ๐Ÿ”—Compilation

๐Ÿ“ฎ Install

See how to install and deploy ๐Ÿ”—Installation and deployment

๐Ÿงฉ Components

๐Ÿ“ Doris Connector

Doris provides support for Spark/Flink to read data stored in Doris through Connector, and also supports to write data to Doris through Connector.

๐Ÿ”—apache/doris-flink-connector

๐Ÿ”—apache/doris-spark-connector

๐ŸŒˆ Community and Support

๐Ÿ“ค Subscribe Mailing Lists

Mail List is the most recognized form of communication in Apache community. See how to ๐Ÿ”—Subscribe Mailing Lists

๐Ÿ™‹ Report Issues or Submit Pull Request

If you meet any questions, feel free to file a ๐Ÿ”—GitHub Issue or post it in ๐Ÿ”—GitHub Discussion and fix it by submitting a ๐Ÿ”—Pull Request

๐Ÿป How to Contribute

We welcome your suggestions, comments (including criticisms), comments and contributions. See ๐Ÿ”—How to Contribute and ๐Ÿ”—Code Submission Guide

โŒจ๏ธ Doris Improvement Proposals (DSIP)

๐Ÿ”—Doris Improvement Proposal (DSIP) can be thought of as A Collection of Design Documents for all Major Feature Updates or Improvements.

๐Ÿ”‘ Backend C++ Coding Specification

๐Ÿ”— Backend C++ Coding Specification should be strictly followed, which will help us achieve better code quality.

๐Ÿ’ฌ Contact Us

Contact us through the following mailing list.

Name Scope
[email protected] Development-related discussions Subscribe Unsubscribe Archives

๐Ÿงฐ Links

๐Ÿ“œ License

Apache License, Version 2.0

Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to be complied with Apache 2.0 License. For details, refer to the thirdparty/LICENSE.txt

More Repositories

1

echarts

Apache ECharts is a powerful, interactive charting and data visualization library for browser
TypeScript
56,321
star
2

superset

Apache Superset is a Data Visualization and Data Exploration Platform
TypeScript
55,051
star
3

dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
Java
40,162
star
4

spark

Apache Spark - A unified analytics engine for large-scale data processing
Scala
36,719
star
5

airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Python
35,085
star
6

kafka

Mirror of Apache Kafka
Java
27,688
star
7

incubator-seata

๐Ÿ”ฅ Seata is an easy-to-use, high-performance, open source distributed transaction solution.
Java
24,602
star
8

skywalking

APM, Application Performance Monitoring System
Java
22,610
star
9

flink

Apache Flink
Java
22,197
star
10

mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
C++
20,572
star
11

shardingsphere

Distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database.
Java
19,505
star
12

rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
Java
18,578
star
13

brpc

brpc is an Industrial-grade RPC framework using C++ Language, which is often used in high performance system such as Search, Storage, Machine learning, Advertisement, Recommendation etc. "brpc" means "better RPC".
C++
14,261
star
14

incubator-weex

Apache Weex (Incubating)
C++
13,884
star
15

hadoop

Apache Hadoop
Java
13,855
star
16

arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
C++
13,780
star
17

pulsar

Apache Pulsar - distributed pub-sub messaging system
Java
13,062
star
18

druid

Apache Druid: a high performance real-time analytics database.
Java
12,843
star
19

predictionio

PredictionIO, a machine learning server for developers and ML engineers.
Scala
12,556
star
20

zookeeper

Apache ZooKeeper
Java
11,532
star
21

incubator-answer

A Q&A platform software for teams at any scales. Whether it's a community forum, help center, or knowledge management platform, you can always count on Apache Answer.
Go
11,487
star
22

apisix

The Cloud-Native API Gateway
Lua
11,031
star
23

thrift

Apache Thrift
C++
9,900
star
24

dolphinscheduler

Apache DolphinScheduler is the modern data workflow orchestration platform with powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available `out of the box`
Java
9,619
star
25

cassandra

Mirror of Apache Cassandra
Java
8,187
star
26

shardingsphere-elasticjob

Distributed scheduled job
Java
7,965
star
27

tvm

Open deep learning compiler stack for cpu, gpu and specialized accelerators
Python
7,828
star
28

beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
Java
7,597
star
29

shenyu

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance.
Java
7,541
star
30

jmeter

Apache JMeter open-source load testing tool for analyzing and measuring the performance of a variety of services
Java
7,335
star
31

tomcat

Apache Tomcat
Java
6,926
star
32

storm

Apache Storm
Java
6,480
star
33

zeppelin

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Java
6,162
star
34

openwhisk

Apache OpenWhisk is an open source serverless cloud platform
Scala
6,130
star
35

couchdb

Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability
Erlang
5,810
star
36

iceberg

Apache Iceberg
Java
5,719
star
37

incubator-kie-drools

Drools is a rule engine, DMN engine and complex event processing (CEP) engine for Java.
Java
5,542
star
38

dubbo-spring-boot-project

Spring Boot Project for Apache Dubbo
Java
5,355
star
39

pinot

Apache Pinot - A realtime distributed OLAP datastore
Java
5,220
star
40

mesos

Apache Mesos
Java
5,111
star
41

hive

Apache Hive
Java
5,053
star
42

camel

Apache Camel is an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.
Java
5,042
star
43

groovy

Apache Groovy: A powerful multi-faceted programming language for the JVM platform
Java
4,977
star
44

hbase

Apache HBase
Java
4,971
star
45

incubator-weex-ui

๐Ÿ„ A rich interaction, lightweight, high performance UI library based on Weex.
Vue
4,788
star
46

ignite

Apache Ignite
Java
4,548
star
47

rocketmq-externals

Mirror of Apache RocketMQ (Incubating)
Java
4,474
star
48

seatunnel

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
Java
4,459
star
49

incubator-pagespeed-ngx

Automatic PageSpeed optimization module for Nginx
C++
4,381
star
50

lucene-solr

Apache Lucene and Solr open-source search software
4,349
star
51

dubbo-go

Go Implementation For Apache Dubbo
Go
4,315
star
52

shiro

Apache Shiro
Java
4,164
star
53

calcite

Apache Calcite
Java
4,028
star
54

nifi

Apache NiFi
Java
4,006
star
55

maven

Apache Maven core
Java
3,836
star
56

hudi

Upserts, Deletes And Incremental Processing on Big Data.
Java
3,804
star
57

dubbo-admin

The ops and reference implementation for Apache Dubbo
Java
3,707
star
58

incubator-heron

Apache Heron (Incubating) is a realtime, distributed, fault-tolerant stream processing engine from Twitter
Java
3,646
star
59

cordova-android

Apache Cordova Android
JavaScript
3,539
star
60

kylin

Apache Kylin
Java
3,526
star
61

httpd

Mirror of Apache HTTP Server. Issues: http://issues.apache.org
C
3,441
star
62

linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Java
3,258
star
63

incubator-kie-optaplanner

AI constraint solver in Java to optimize the vehicle routing problem, employee rostering, task assignment, maintenance scheduling, conference scheduling and other planning problems.
Java
3,152
star
64

logging-log4j2

Apache Log4j 2 is a versatile, feature-rich, efficient logging API and backend for Java.
Java
3,151
star
65

iotdb

Apache IoTDB
Java
3,000
star
66

arrow-datafusion

Apache Arrow DataFusion SQL Query Engine
Rust
2,998
star
67

curator

Apache Curator
Java
2,997
star
68

singa

a distributed deep learning platform
C++
2,968
star
69

incubator-streampark

StreamPark, Make stream processing easier! easy-to-use streaming application development framework and operation platform
Scala
2,820
star
70

avro

Apache Avro is a data serialization system.
Java
2,798
star
71

nutch

Apache Nutch is an extensible and scalable web crawler
Java
2,653
star
72

incubator-fury

A blazingly fast multi-language serialization framework powered by JIT and zero-copy.
Java
2,649
star
73

guacamole-server

Mirror of Apache Guacamole Server
C
2,601
star
74

netbeans

Apache NetBeans
Java
2,553
star
75

commons-lang

Apache Commons Lang
Java
2,539
star
76

flume

Mirror of Apache Flume
Java
2,458
star
77

incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Go
2,440
star
78

geode

Apache Geode
Java
2,244
star
79

incubator-seata-samples

seata-samples
Java
2,196
star
80

gobblin

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
Java
2,196
star
81

maven-mvnd

Apache Maven Daemon
Java
2,191
star
82

activemq

Mirror of Apache ActiveMQ
Java
2,184
star
83

incubator-hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
Java
2,156
star
84

parquet-mr

Apache Parquet
Java
2,153
star
85

kvrocks

Kvrocks is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol.
C++
2,147
star
86

pdfbox

Mirror of Apache PDFBox
Java
2,131
star
87

cordova-ios

Apache Cordova iOS
JavaScript
2,130
star
88

mahout

Mirror of Apache Mahout
Java
2,095
star
89

lucenenet

Apache Lucene.NET
C#
2,031
star
90

ambari

Apache Ambari simplifies provisioning, managing, and monitoring of Apache Hadoop clusters.
Java
1,990
star
91

libcloud

Apache Libcloud is a Python library which hides differences between different cloud provider APIs and allows you to manage different cloud resources through a unified and easy to use API.
Python
1,956
star
92

incubator-pegasus

Apache Pegasus - A horizontally scalable, strongly consistent and high-performance key-value store
C++
1,946
star
93

tika

The Apache Tika toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF).
Java
1,860
star
94

drill

Apache Drill is a distributed MPP query layer for self describing data
Java
1,841
star
95

dubbo-samples

samples for Apache Dubbo
Java
1,839
star
96

servicecomb-java-chassis

ServiceComb Java Chassis is a Software Development Kit (SDK) for rapid development of microservices in Java, providing service registration, service discovery, dynamic routing, and service management features
Java
1,829
star
97

tinkerpop

Apache TinkerPop - a graph computing framework
Java
1,825
star
98

cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
Java
1,794
star
99

bookkeeper

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads
Java
1,785
star
100

rocketmq-spring

Apache RocketMQ Spring Integration
Java
1,775
star