The Pipeline execution engine is a new core feature of ApsaraDB for SelectDB, designed to replace the legacy execution engine. In mixed-workload environments, the legacy volcano-model engine cannot fully exploit multi-core CPUs, suffers from thread pool exhaustion, and lets large queries starve small ones. The Pipeline execution engine replaces this legacy engine with a data-driven, push-based architecture that eliminates thread bloat and reduces resource contention between concurrent queries.
For design details, see Support Pipeline Exec Engine.
How it works
The legacy SQL execution engine is based on the volcano model, which has three fundamental problems in multi-core environments:
Manual parallelism tuning required. The engine cannot automatically exploit multi-core CPU capacity. To improve query performance, you must manually set the degree of parallelism—something that is difficult to tune correctly in production.
Thread pool exhaustion and deadlocks. Each query instance occupies one thread in the thread pool. When the pool reaches capacity, the query engine enters a pseudo-deadlock and stops responding to new queries. Logical deadlocks can also occur—for example, when all threads are executing a probe task on an instance. Even under normal load, blocking operators hold threads that cannot be reassigned to runnable instances, keeping overall CPU utilization low.
High thread-switching overhead. Blocking operators depend on the OS thread scheduler. In mixed-workload scenarios, this causes significant context-switch overhead that degrades throughput.
The Pipeline execution engine resolves these problems through three architectural changes:
Pull to push execution. Queries run in push mode instead of the traditional pull model. Data flows downstream through pipelines without threads waiting to pull results.
Blocking to asynchronous operations. Operators that previously blocked threads are redesigned as asynchronous operations. This eliminates thread-switching overhead and keeps CPU utilization high.
Bounded thread count with time-slice scheduling. The engine caps the number of execution threads and uses time-slice switching to arbitrate between concurrent pipelines. Large queries can no longer monopolize threads and starve small queries.
Configure the Pipeline execution engine
Set session variables to control how the Pipeline execution engine runs for a given query session.
Session variables
| Variable | Default | Description |
|---|---|---|
enable_pipeline_engine | true | Enables the Pipeline execution engine on backend (BE) nodes. When set to true, the BE node uses the Pipeline execution engine for query execution. |
parallel_pipeline_task_num | 0 | Number of pipeline tasks to run concurrently per SQL query. When set to 0, ApsaraDB for SelectDB automatically sets this to half the number of CPU cores on the BE node. |
max_instance_num | 64 | Maximum concurrency that ApsaraDB for SelectDB sets automatically. By default, this session variable is set to 64. |
Enable pipeline execution
The Pipeline execution engine is enabled by default. To explicitly enable it:
SET enable_pipeline_engine = true;Set pipeline concurrency
By default, ApsaraDB for SelectDB automatically sets pipeline task concurrency to half the number of CPU cores (parallel_pipeline_task_num = 0). To override this:
SET parallel_pipeline_task_num = <n>;Whenparallel_pipeline_task_numis0, the auto-computed concurrency is bounded bymax_instance_num(default:64). You can adjustmax_instance_numto change the ceiling on automatic concurrency.