MaxCompute incremental computing leverages the storage and read/write capabilities of Delta Tables to perform complex calculations on incremental data and dynamically capture data changes.
Background
As the core of incremental computing, the Dynamic Materialized View (Delta Live MV) enables complex calculations on incremental data. A Dynamic Materialized View uses incremental computing technology to support unified SQL logic for both streaming and batch processing. It automatically captures changes from a source table and continuously performs efficient, low-cost incremental calculations, unifying your stream and batch data pipelines. You can adjust the automatic refresh frequency of a Dynamic Materialized View (Delta Live MV) to achieve the optimal balance between data freshness and computational cost for various scenarios.
If you already use MaxCompute for your offline data warehouse, you can easily transform your existing batch services into near-real-time operations with minimal changes. This lets you gradually upgrade your MaxCompute system from a batch-only platform to a unified stream-batch data warehouse architecture. For new MaxCompute users, use Dynamic Materialized View from the start. This helps you build a unified data warehouse architecture that handles both incremental and full data, as well as stream and batch processing, based on your business's real-time requirements.
You can also dynamically capture upstream data changes using a Stream and a periodic task.
Use cases
Use a Dynamic Materialized View (Delta Live MV) to build automated data pipelines for both incremental and full data.
Use a Stream and a periodic task (invitation-only preview) to dynamically capture data changes.
Usage
The Stream and Dynamic Materialized View (Delta Live MV) features are now generally available. The periodic task feature is in invitation-only preview. To request access, please submit a ticket.