MaxCompute incremental computing processes only the data that changed since the last run, rather than reprocessing your entire dataset on every run. This lets you build near-real-time pipelines from existing SQL logic without rewriting your transformation code. The feature relies on Delta Tables for incremental data storage and access.
How it works
Dynamic Materialized Views (Delta Live MVs) are the core of incremental computing. Define a Delta Live MV with standard SQL, and it refreshes automatically whenever its Base Tables receive incremental changes. Because each refresh processes only the changes — not the full table — runs are faster and cheaper than a full recalculation.
Delta Live MVs support unified SQL logic for both stream and batch processing. Adjust the auto-refresh frequency to balance data freshness against compute cost: lower the interval for near-real-time pipelines, raise it to reduce spend during off-peak periods.
As an alternative, combine Stream objects with Periodic Tasks to detect changes in input data and trigger downstream SQL on a defined schedule.
Use cases
Batch-to-streaming migration: Make minor changes to existing MaxCompute SQL jobs to produce near-real-time results. Delta Live MVs add continuous incremental refresh, so you migrate incrementally rather than rebuilding from scratch. This gradually upgrades your MaxCompute system from an offline-only system to a unified stream-batch data warehouse architecture that unifies stream and batch business traces.
New unified stream-batch pipeline: Build a pipeline directly with Delta Live MVs to handle both incremental and full data in a single architecture that supports stream and batch processing.
Event-driven incremental processing: Combine Stream objects and Periodic Tasks to detect changes in source tables and trigger downstream SQL on a defined schedule.
When to use which approach
| Approach | Use when | Refresh control |
|---|---|---|
| Dynamic Materialized Views (Delta Live MVs) | You want declarative, SQL-only pipelines that refresh automatically without managing schedules or dependencies | Adjustable auto-refresh frequency |
| Stream objects + Periodic Tasks | You need explicit control over when downstream SQL fires, or your pipeline requires imperative transformation logic | You define the schedule; Stream detects data changes |
Both approaches rely on Delta Tables for incremental data storage.
Usage notes
The Stream object feature is generally available.
The Dynamic Materialized View (Delta Live MV) and Periodic Task features are in invitational preview. To try these features, submit a ticket.