E-MapReduce:Doris

After data from various sources is integrated and processed, it is typically loaded into a real-time data warehouse, such as Doris, and an offline data lakehouse, such as Hive, Iceberg, and Hudi. The following figure shows this process.

Apache Doris is widely used in the following scenarios.

• Report analysis

  • Real-time dashboards.

  • Reports for internal analysts and managers.

  • High-concurrency report analysis (Customer Facing Analytics), such as site analysis for website owners and ad reports for advertisers. This typically requires thousands of queries per second (QPS) with millisecond-level latency.

• Ad hoc queries: Self-service analytics for analysts with variable query patterns that require high throughput.

• Unified data warehouse construction: Doris serves as a single platform to build a unified data warehouse, replacing legacy architectures that consist of Spark, Hive, Kudu, Hbase, and Phoenix and greatly simplifying the big data software stack.

• Data lake federated queries: External tables enable federated analysis on data in Hive, Iceberg, and Hudi without data copying, which significantly improves query performance.

The following figure shows the Doris architecture, which consists of only two types of processes.

• Frontend (FE): Handles client requests, query parsing and planning, metadata, and node management.

• Backend (BE): Manages data storage and executes query plans.

Both process types scale horizontally, supporting hundreds of machines and tens of petabytes of storage per cluster. A consistency protocol ensures high availability (HA) for services and high reliability for data, while the integrated architecture reduces the operations and maintenance (O&M) costs of a distributed system.

Doris technology has five key aspects:

• Doris uses the MySQL protocol, is highly compatible with MySQL syntax, and supports standard SQL. This lets you access Doris with various client tools and seamlessly integrate it with Business Intelligence (BI) tools.

• Doris uses a columnar storage engine that encodes, compresses, and reads data by column. This achieves a very high compression ratio and reduces scans of irrelevant data for more efficient I/O and CPU usage.

  Doris also supports a rich set of index structures to reduce data scanning:

  • Sorted Compound Key Index: Supports up to three columns as a compound sort key, enabling effective data clipping for high-concurrency reporting scenarios.

  • Z-order Index: Efficiently performs range queries on any combination of fields in the data model.

  • Min/Max: Effectively filters equality and range queries on numeric types.

  • Bloom Filter: Very effective for equality filter clipping on high-cardinality columns.

  • Inverted Index: Enables fast retrieval on any field.

• Doris supports multiple storage models that are optimized for different scenarios:

  • Aggregate Key model: Merges value columns with the same key. This model greatly improves performance through pre-aggregation.

  • Unique Key model: Keys are unique. Data with the same key is overwritten, which enables row-level data updates.

  • Duplicate Key model: A detailed data model that stores the details of a fact table.

  Doris also supports strongly consistent materialized views that are automatically updated and selected without manual intervention, which greatly reduces maintenance costs.

• Doris uses a Massively Parallel Processing (MPP) model. Execution is parallel between nodes and within each node. It also supports distributed shuffle joins for multiple large tables to better handle complex queries. The following figure shows the query engine.

  The Doris query engine is a vectorized engine. All memory structures are arranged in a columnar layout. This design greatly reduces virtual function invocations, improves the cache hit rate, and makes efficient use of SIMD instructions. In wide table aggregation scenarios, its performance is 5 to 10 times better than that of a non-vectorized engine.

• Doris uses a hybrid optimization strategy that combines a Rule-Based Optimizer (RBO) and a Cost-Based Optimizer (CBO). The RBO supports constant folding, subquery rewrites, and predicate pushdown. The CBO supports join reordering. The CBO is continuously optimized, focusing on more accurate statistics collection and derivation, and more precise cost model estimation.

Doris uses Adaptive Query Execution technology to dynamically adjust execution plans based on runtime statistics. For example, the Runtime Filter generates a filter at runtime and pushes it to the probe side. The filter can also be automatically pushed down to the lowest-level scan node on the probe side, which greatly reduces data volume and accelerates join performance, as shown in the following flowchart.The Doris Runtime Filter supports In, Min, Max, and Bloom filters.