When multiple jobs write to the same Delta table concurrently, you need to understand which operation combinations conflict, how MaxCompute resolves those conflicts, and how to design your pipelines to avoid them. MaxCompute uses MetaService to manage all data modification operations on Delta tables as transactions. Multiversion concurrency control (MVCC) isolates concurrent read and write operations on snapshots, while optimistic concurrency control (OCC) handles transaction concurrency.
Conflict detection rules
The following table shows how MaxCompute resolves conflicts between concurrent jobs on the same non-partitioned table or partition. The job that finishes first is the earlier job; the job that finishes later is the later job.
| Earlier job | Later job | Outcome |
|---|---|---|
| INSERT OVERWRITE / TRUNCATE | INSERT OVERWRITE / TRUNCATE | Both succeed. The later job's result overwrites the earlier job's result. |
| INSERT OVERWRITE / TRUNCATE | INSERT INTO | Earlier job succeeds. Later job fails. |
| INSERT OVERWRITE / TRUNCATE | UPDATE / DELETE | Earlier job succeeds. Later job fails. |
| INSERT OVERWRITE / TRUNCATE | MINOR COMPACT | Earlier job succeeds. Later job fails. |
| INSERT OVERWRITE / TRUNCATE | MAJOR COMPACT | Earlier job succeeds. Later job fails. |
| INSERT INTO | INSERT OVERWRITE / TRUNCATE | Both succeed. The later job's result overwrites the earlier job's result. |
| INSERT INTO | INSERT INTO | Earlier job succeeds. Later job fails. |
| INSERT INTO | UPDATE / DELETE | Earlier job succeeds. Later job fails. |
| INSERT INTO | MINOR COMPACT | Both succeed. The compaction from the later job completes. |
| INSERT INTO | MAJOR COMPACT | Earlier job succeeds. Later job fails. |
| UPDATE / DELETE | INSERT OVERWRITE / TRUNCATE | Both succeed. The later job's result overwrites the earlier job's result. |
| UPDATE / DELETE | INSERT INTO | Earlier job succeeds. Later job fails. |
| UPDATE / DELETE | UPDATE / DELETE | Earlier job succeeds. Later job fails. |
| UPDATE / DELETE | MINOR COMPACT | Both succeed. The compaction from the later job completes. |
| UPDATE / DELETE | MAJOR COMPACT | Earlier job succeeds. Later job fails. |
| MINOR COMPACT | INSERT OVERWRITE / TRUNCATE | Both succeed. The later job's result overwrites the earlier job's result. |
| MINOR COMPACT | INSERT INTO | Both succeed. New data from the later job is written. |
| MINOR COMPACT | UPDATE / DELETE | Both succeed. New data from the later job is written. |
| MINOR COMPACT | MINOR COMPACT | Earlier job succeeds. Later job fails. |
| MINOR COMPACT | MAJOR COMPACT | Both succeed. The compaction from the later job completes. |
| MAJOR COMPACT | INSERT OVERWRITE / TRUNCATE | Both succeed. The later job's result overwrites the earlier job's result. |
| MAJOR COMPACT | INSERT INTO | Both succeed. New data from the later job is written. |
| MAJOR COMPACT | UPDATE / DELETE | Both succeed. New data from the later job is written. |
| MAJOR COMPACT | MINOR COMPACT | Earlier job succeeds. Later job fails. |
| MAJOR COMPACT | MAJOR COMPACT | Earlier job succeeds. Later job fails. |
These rules apply at the transaction level, not at the row or file level.
Concurrency conflict optimization
MaxCompute optimizes conflict handling for common operation combinations, reducing unnecessary failures while preserving data correctness.
Concurrency conflict optimization applies to single-table transaction management only.
Smart conflict resolution
Not all overlapping transactions conflict. For example, when a clustering operation and an INSERT INTO operation overlap in start and commit time, both succeed. Clustering reorganizes the physical layout of data without changing data status, so it cannot cause status inconsistency with a concurrent write. MaxCompute detects this and allows both to proceed. The transaction conflict handling logic will continue to be optimized to adapt to more scenarios.
Metadata-level retry
When conflict detection fails, MaxCompute retries at the metadata level. No data needs to be re-read or re-written. This reduces resource consumption compared to rerunning the full job.
Atomic metadata updates
Metadata is updated atomically to guarantee data consistency.
Data file version management
Each transaction generates a batch of new data files associated with two version types:
| Version type | Data type | Generated when | Used for |
|---|---|---|---|
| Time version | TIMESTAMP | User-triggered operations that make logical data changes only. Clustering and compaction do not generate a new time version. | Incremental queries by time. Files from clustering and compaction are excluded. |
| ID version | Auto-increment integer | Every data operation in a transaction, including clustering and compaction. | Internal transaction management (primary). Also available for incremental queries. |