MaxCompute Delta Table is an incremental data table format. It enables minute-level data imports, uses AliORC as its base file format, and supports UPSERT operations. It provides a Change Data Capture (CDC) format for reading and writing incremental data. The Delta Table automatically manages data based on the storage services and metadata services of MaxCompute. This topic describes the key functionalities of the MaxCompute Delta Table.
Delta Table format description
Table data format supports efficient storage and access of both incremental and full data. When a primary key is defined, the format enables real-time data updates.
Near-real-time incremental import
Full import and near-real-time incremental import for various data sources: MaxCompute is used together with related services to provide various custom data import tools, such as Use Flink to write data to a Delta table of MaxCompute and Data Integration of DataWorks. These tools support efficient near-real-time incremental import. These tools can be connected to the Near real-time incremental import of the Tunnel service of MaxCompute to support high-concurrency minute-level incremental write.
Full write and incremental batch write for various data sources: MaxCompute SQL and other interfaces can be used to support efficient full write and incremental batch write.
Data organization optimization
The storage service of MaxCompute manages incremental data files in an intelligent manner. The management of incremental data files includes optimization operations, such as Clustering of small files, data COMPACTION, and data sorting. The storage service automatically executes data optimization tasks for some operations based on a comprehensive evaluation of multiple dimensions, such as data characteristics and time series, to help maintain stable and efficient data storage and computing.
Transaction management
This module is used for Manage transactions, data version management, time travel management, metadata updates, and metadata analysis.
Support for querying incremental data by using SQL syntax.
Support for querying historical snapshots by using SQL syntax, which helps in tracing back the historical state of business data and recovering erroneous data.
Basic operations
DDL operation - Data Definition Language
Write data