Append Delta Tables use Range Clustering for data organization. By default, Row_ID is the cluster key, and buckets are allocated dynamically as data grows. Once you specify a cluster key, a background clustering job performs incremental reclustering — keeping data ordered without blocking writes.
When to use Append Delta Tables
Append Delta Tables are well-suited for these scenarios:
-
High-throughput append workloads: Tables in the Operational Data Store (ODS) layer that receive continuous streaming writes benefit from write-time decoupling — data lands immediately without requiring upfront sorting.
-
High-cardinality filter columns: Workloads that filter on columns with many distinct values gain clustering benefits without the write-cost penalty of traditional Range/Hash Cluster tables.
-
Rapidly growing data: Tables that scale from terabytes to exabytes need a bucketing strategy that adapts automatically — no manual reconfiguration as data volume changes.
-
Data skew-prone tables: Uneven data distributions make static bucket counts unreliable. Dynamic bucketing eliminates that guesswork.
-
ODS-layer query acceleration: When both query performance and data freshness matter, incremental reclustering delivers millisecond-level freshness while background tasks handle clustering.
How it works
Append Delta Tables combine two mechanisms to keep data organized efficiently:
-
Dynamic bucketing — automatically adjusts the number of buckets as data volume grows, eliminating the need to predict bucket counts at table creation time.
-
Incremental reclustering — a background data service asynchronously reclusters newly written data, decoupling write performance from clustering overhead.
Together, these mechanisms maintain a dynamic balance among storage efficiency, data freshness, and query performance through three background tasks: Merge, Compaction, and Reclustering. Dynamic bucketing also supports seamless scaling from terabytes to exabytes through Auto-Split/Merge policies.
Dynamic bucketing
The problem with static bucket counts
Traditional Range/Hash Cluster tables require you to specify a bucket count at table creation time. MaxCompute then routes data to buckets based on the cluster key. Getting this number wrong in either direction causes problems:
-
Data skew: Too few buckets relative to data volume causes individual buckets to grow oversized, which reduces the effectiveness of data pruning during queries.
-
Data fragmentation: Too many buckets relative to data volume leaves each bucket with very little data, producing many small fragmented files that harm query performance.
Choosing the right bucket count requires detailed knowledge of expected data volume and MaxCompute's internal table format. For large-scale data migrations involving thousands of tables, estimating the appropriate bucket count per table is not feasible. Even when the initial estimate is accurate, business data volumes change over time, making a previously correct configuration incorrect.
How dynamic bucketing works
Append Delta Tables allocate buckets automatically. Each bucket is a logically contiguous storage unit that holds approximately 500 MB of data. As you write data continuously, new buckets are created as needed — with no configuration required. The bucket count always reflects the actual data volume, eliminating skew and fragmentation caused by over- or under-provisioning.
The following diagram shows the workflow:
Incremental reclustering
The problem with synchronous clustering
Clustering accelerates queries by sorting and co-locating data by a specified cluster key. When a query uses the cluster key, MaxCompute can apply pushdown and pruning to narrow the data scan range.
Traditional Range Clustering and Hash Clustering achieve this by bucketing and sorting data during the write process to produce a globally sorted state. The following diagram illustrates the Range Clustering / Hash Clustering pruning mechanism:
This write-time sorting creates two problems:
Problem 1: High cost of appending data
Write-time sorting restricts how you can load data. The initial write must complete in a single operation using INSERT INTO or INSERT OVERWRITE. To append more data afterward, you must read all existing table data, combine it with new data using UNION, and rewrite the entire dataset. This makes append operations expensive and throughput-limited.
ODS-layer tables typically receive continuous writes from external collection pipelines, which require low-latency, high-throughput ingestion. The write-amplification cost of traditional clustered tables prevents applying clustering at the ODS layer.
Problem 2: Data freshness latency in the data warehouse layer
To avoid write amplification, clustering is typically applied only at the data warehouse (DW) layer, where data from ODS is cleaned and loaded in stable batches. This introduces a freshness lag: DW-layer data always trails the ODS layer by at least one processing cycle.
Some workloads require both clustered query performance and real-time data freshness simultaneously. Synchronous clustering at write time cannot meet this requirement.
How incremental reclustering works
Append Delta Tables decouple clustering from writes. Data is written directly to disk unsorted and allocated to buckets, maximizing write throughput and minimizing latency. Because newly written data is not yet clustered, the data ranges of new buckets overlap with those of existing clustered buckets. The SQL engine handles this transparently: it prunes clustered buckets and scans incremental buckets during query execution.
The following diagram illustrates the process:
A background data service continuously monitors the Bucket Overlap Depth. When overlap reaches a specific threshold, it triggers incremental reclustering on the newly written buckets. The bulk of data remains ordered at all times, providing stable overall query performance.
This approach achieves an optimal balance: writes land with minimal latency, queries run against predominantly ordered data, and the gap between ODS ingestion and clustered query performance is eliminated. The result is millisecond-level data freshness in the ODS layer with clustered query acceleration.