Overview
MaxCompute stores Apache Iceberg tables in Object Storage Service (OSS) with unified metadata, permission, and lifecycle management. Iceberg tables are compatible with open-source engines such as Spark, Flink, Trino, and Presto, enabling multi-engine data sharing for a Lakehouse architecture.
The open Iceberg format eliminates data silos and enables seamless data sharing with Spark, Flink, and Hive. You can choose the right compute engine for each workload without vendor lock-in. The open standard also ensures long-term data portability and supports cross-organizational collaboration in multi-cloud scenarios.
MaxCompute-managed Iceberg tables offer the following features:
Iceberg lake table data management:
Create, read, and write Iceberg tables on OSS using MaxCompute SQL. Supports DDL, DML, time travel, schema evolution, and partition evolution.
Automated background maintenance:
Built-in optimization handles compaction, snapshot expiration, and metadata archiving automatically, eliminating manual maintenance.
Metadata consistency:
MaxCompute Metastore (MaxMeta) manages table metadata with ACID-compliant writes. External engines must use the MaxCompute Storage Write API or a MaxCompute-native engine, preventing metadata conflicts from multiple write sources.
Open ecosystem integration:
Trino, Presto, Spark, Flink, and Doris can access managed Iceberg tables natively, enabling low-latency, high-concurrency multi-engine queries.
Unified permission management: Define permissions for your data lake and data warehouse in a single system. Integrates RAM permissions with data warehouse access control for enterprise-grade Lakehouse security.
Usage notes
The following operations on a MaxCompute-managed Iceberg table can cause data loss or make data unreadable:
Forbidden operation | Consequence | Correct practice |
Modifying Iceberg table data through any interface other than MaxCompute write APIs. | The table may fail consistency checks and become unreadable. | Modify data only through MaxCompute SQL. |
Uploading files to the managed OSS path. | The background service treats uploaded files as orphan files and deletes them. | Do not upload data to the managed path. |
Sharing the same OSS path across multiple managed Iceberg tables. | Each table's garbage collection process deletes the other table's files. | Each table uses a separate OSS path. If multiple tables are created in the same directory, recreate the table using a clean path and copy the data to the new table to avoid data loss. |
Using OSS paths with a parent-child directory relationship for different managed Iceberg tables. | Each table's garbage collection treats the other table's files as orphan data and deletes them, causing data loss. | Use a unique OSS path for each managed Iceberg table. Avoid paths with parent-child relationships. |
Create a table
Syntax
CREATE ICEBERG TABLE [IF NOT EXISTS] <table name> (
<col_name> <data_type>,
...
)
[PARTITIONED BY (<partitionExpression>)]
WITH CONNECTION <connection name>
OPTIONS(
location='<oss_location>'
)
;Parameters
Example
CREATE ICEBERG TABLE mc_iceberg_table (
id bigint COMMENT 'Unique user ID',
name string COMMENT 'User name',
age bigint COMMENT 'User age',
gender string COMMENT 'User gender',
height float COMMENT 'User height',
birthday date COMMENT 'User date of birth',
phone_number string COMMENT 'User phone number',
email string COMMENT 'User email address',
address string COMMENT 'User address',
salary decimal(18, 2) COMMENT 'User salary',
create_time timestamp COMMENT 'Time when user information was created',
update_time timestamp COMMENT 'Time when user information was last updated',
is_deleted boolean COMMENT 'Flag indicating if the user information has been deleted',
dt string COMMENT 'Partition field'
)
PARTITIONED BY (dt)
WITH CONNECTION <connection name>
OPTIONS(
location='oss://<oss bucket>/Demo-iceberg/',
comment = '测试表'
);Writing data
MaxCompute write syntax: Syntax description.
Example:
SET odps.sql.type.system.odps2=true; SET odps.sql.decimal.odps2=true; INSERT INTO mc_iceberg_table VALUES (1, 'Zhang San', 18, 'Male', cast (178.56 as float), DATE '1990-01-01', '13800000000', 'zhangsan@example.com', 'Haidian District, Beijing', 5000.00, TIMESTAMP '2023-04-19 11:32:00', TIMESTAMP '2023-04-19 11:32:00', false,'20260402'), (2, 'Li Si', 20, 'Female', cast (162.70 as float), DATE '1992-02-02', '13900000000', 'lisi@example.com', 'Pudong New Area, Shanghai', 6000.00, TIMESTAMP '2023-04-19 11:32:00', TIMESTAMP '2023-04-19 11:32:00',false,'20260401'), (3, 'Wang Wu', 22, 'Male', cast (185.21 as float), DATE '1994-03-03', '14000000000', 'wangwu@example.com', 'Nanshan District, Shenzhen', 7000.00, TIMESTAMP '2023-04-19 11:32:00', TIMESTAMP '2023-04-19 11:32:00', false,'20260403') ;
Querying and analyzing data
Basic queries
SELECT syntax: Syntax description.
Example:
SELECT * FROM mc_iceberg_table; -- The following result is returned: +------------+------+------------+--------+--------+----------+--------------+-------+---------+--------+-------------+-------------+------------+----+ | id | name | age | gender | height | birthday | phone_number | email | address | salary | create_time | update_time | is_deleted | dt | +------------+------+------------+--------+--------+----------+--------------+-------+---------+--------+-------------+-------------+------------+----+ | 1 | Zhang San | 18 | Male | 178.56 | 1990-01-01 | 13800000000 | zhangsan@example.com | Haidian District, Beijing | 5000 | 2023-04-19 03:32:00 | 2023-04-19 03:32:00 | false | 20260402 | | 2 | Li Si | 20 | Female | 162.7 | 1992-02-02 | 13900000000 | lisi@example.com | Pudong New Area, Shanghai | 6000 | 2023-04-19 03:32:00 | 2023-04-19 03:32:00 | false | 20260401 | | 3 | Wang Wu | 22 | Male | 185.21 | 1994-03-03 | 14000000000 | wangwu@example.com | Nanshan District, Shenzhen| 7000 | 2023-04-19 03:32:00 | 2023-04-19 03:32:00 | false | 20260403 | +------------+------+------------+--------+--------+----------+--------------+-------+---------+--------+-------------+-------------+------------+----+
Time Travel queries for historical versions
Time Travel queries on MaxCompute-managed Iceberg tables retrieve historical snapshots of the source table at a specified time or version. The [TIMESTAMP | VERSION AS OF expr] clause specifies the query method.
SHOW HISTORY
SHOW HISTORY lists snapshot_id and committed_at.
Syntax
SHOW HISTORY FOR TABLE <table_name>;Parameters
table_name: The table name.
Example
SHOW HISTORY FOR TABLE user_info_test; -- The results are as follows committed_at snapshot_id parent_id operation manifest_list summary 1793701639053000 9223372035110209012 null append oss://bucket/manage_iceberg_table/user_info_test/metadata/snap-9223372035110209012-1-55508b08-676a-48c6-9923-efdd2ea423cb.avro [{"key":"odps.task.instance.name","value":"2026110310270359gv6arkmh5ze5"},{"key":"added-files-size","value":"2316"},{"key":"total-equality-deletes","value":"0"},{"key":"total-records","value":"2"},{"key":"iceberg-version","value":"Apache Iceberg 1.6.1 (commit 8e9d59d299be42b0bca9461457cd1e95dbaad086)"},{"key":"changed-partition-count","value":"1"},{"key":"added-data-files","value":"1"},{"key":"total-files-size","value":"2316"},{"key":"total-position-deletes","value":"0"},{"key":"total-data-files","value":"1"},{"key":"total-delete-files","value":"0"},{"key":"added-records","value":"2"}] 1793701689430000 9104544753594776504 9223372035110209012 append oss://bucket/manage_iceberg_table/user_info_test/metadata/snap-9104544753594776504-1-5bed9def-2147-4070-9421-da287d5b1dc8.avro [{"key":"total-files-size","value":"4584"},{"key":"total-equality-deletes","value":"0"},{"key":"total-records","value":"3"},{"key":"iceberg-version","value":"Apache Iceberg 1.6.1 (commit 8e9d59d299be42b0bca9461457cd1e95dbaad086)"},{"key":"changed-partition-count","value":"1"},{"key":"added-data-files","value":"1"},{"key":"total-position-deletes","value":"0"},{"key":"total-delete-files","value":"0"},{"key":"total-data-files","value":"2"},{"key":"added-records","value":"1"},{"key":"added-files-size","value":"2268"},{"key":"odps.task.instance.name","value":"20261103102754425gbx1wpz04ti"}] 1793758383410000 9223372036507144701 9104544753594776504 append oss://bucket/manage_iceberg_table/user_info_test/metadata/snap-9223372036507144701-1-521017a8-0c05-4844-bb74-75663d02b294.avro [{"key":"total-position-deletes","value":"0"},{"key":"added-data-files","value":"1"},{"key":"changed-partition-count","value":"1"},{"key":"iceberg-version","value":"Apache Iceberg 1.6.1 (commit 8e9d59d299be42b0bca9461457cd1e95dbaad086)"},{"key":"total-equality-deletes","value":"0"},{"key":"total-records","value":"4"},{"key":"added-records","value":"1"},{"key":"added-files-size","value":"2303"},{"key":"odps.task.instance.name","value":"20261104021242439gag3a4kh5ze5"},{"key":"total-data-files","value":"3"},{"key":"total-delete-files","value":"0"},{"key":"total-files-size","value":"6887"}] 1793758503143000 9223372036251690789 9223372036507144701 append oss://bucket/manage_iceberg_table/user_info_test/metadata/snap-9223372036251690789-1-ba5e5d25-4844-4624-b6e9-63e19e558238.avro [{"key":"total-equality-deletes","value":"0"},{"key":"added-data-files","value":"1"},{"key":"iceberg-version","value":"Apache Iceberg 1.6.1 (commit 8e9d59d299be42b0bca9461457cd1e95dbaad086)"},{"key":"changed-partition-count","value":"1"},{"key":"added-records","value":"1"},{"key":"added-files-size","value":"2300"},{"key":"odps.task.instance.name","value":"2026110402143563g69g9k2a4385"},{"key":"total-records","value":"5"},{"key":"total-files-size","value":"9187"},{"key":"total-delete-files","value":"0"},{"key":"total-data-files","value":"4"},{"key":"total-position-deletes","value":"0"}] 1793759378679000 5293593429568553902 9223372036251690789 append oss://bucket/manage_iceberg_table/user_info_test/metadata/snap-5293593429568553902-1-636398c5-cc19-4554-bcdf-8e4654924c7f.avro [{"key":"odps.task.instance.name","value":"20261104022908984goqofj5ywhj"},{"key":"added-data-files","value":"1"},{"key":"added-records","value":"1"},{"key":"added-files-size","value":"2329"},{"key":"changed-partition-count","value":"1"},{"key":"total-records","value":"6"},{"key":"total-files-size","value":"11516"},{"key":"total-data-files","value":"5"},{"key":"total-delete-files","value":"0"},{"key":"total-position-deletes","value":"0"},{"key":"total-equality-deletes","value":"0"},{"key":"iceberg-version","value":"Apache Iceberg 1.6.1 (commit 8e9d59d299be42b0bca9461457cd1e95dbaad086)"}]
TIMESTAMP AS OF expr
Query the source table as of a specified historical time.
Syntax
SELECT * FROM <table_name> TIMESTAMP AS OF <expr>;Parameters
table_name: The table name.
TIMESTAMP AS OFis a fixed syntax format for querying historical snapshot data based on a historical time.exprhas the standard MaxCompute TIMESTAMP type, such as2026-11-04 10:00:00.123. The committed_at field in the SHOW HISTORY results can be converted to a TIMESTAMP value for a historical query.
Example
SET odps.sql.type.system.odps2=true; SELECT FROM_UNIXTIME(1793701689); -- Remove the last 6 digits of 1793701639053000 -- -- TIMESTAMP AS OF expr: Query by timestamp (session time zone, millisecond precision) SELECT * FROM user_info_test TIMESTAMP AS OF '2026-11-03 18:28:09'; -- The results are as follows. The current table contains 6 rows. The SQL statement below queries the snapshot after the second data insertion. +------------+----------+------------+------------+-------+--------+---------------+ | user_id | username | age | salary | email | status | register_time | +------------+----------+------------+------------+-------+--------+---------------+ | 1001 | NULL | 25 | 8000.0 | zhangsan@email.com | ACTIVE | 2026-11-01 10:30:00 | | 1002 | NULL | 30 | 12000.0 | lisi@email.com | ACTIVE | 2026-11-02 14:20:00 | +------------+----------+------------+------------+-------+--------+---------------+ -- Current contents of the queried table +------------+----------+------------+------------+-------+--------+---------------+ | user_id | username | age | salary | email | status | register_time | +------------+----------+------------+------------+-------+--------+---------------+ | 1001 | NULL | 25 | 8000.0 | zhangsan@email.com | ACTIVE | 2026-11-01 10:30:00 | | 1002 | NULL | 30 | 12000.0 | lisi@email.com | ACTIVE | 2026-11-02 14:20:00 | | 1004 | 赵六 | 32 | 11000.0 | zhaoliu@email.com | INACTIVE | 2026-11-03 16:40:00 | | 1003 | 王五 | 28 | 9500.0 | wangwu@email.com | ACTIVE | 2026-11-03 09:15:00 | | 1004 | 赵六 | 32 | 11000.0 | zhaoliu@email.com | INACTIVE | 2026-11-03 16:40:00 | | 1005 | 孙七 | 35 | 15000.0 | sunqi@email.com | ACTIVE | 2026-11-04 08:20:00 | +------------+----------+------------+------------+-------+--------+---------------+
VERSION AS OF expr
Query the source table as of a specified version.
Syntax
SELECT * FROM <table_name> VERSION AS OF <expr>;Parameters
table_name: The table name.
VERSION AS OFis a fixed syntax format for querying historical snapshot data based on the version number (version) of a historical data operation.exprhas the MaxCompute bigint integer return type. Specify the snapshot_id field value returned bySHOW HISTORY FOR TABLE <table_name>;.
Example
SELECT * FROM t VERSION AS OF 9223372035110209012; -- The results are as follows. The current table contains 6 rows. The SQL statement below queries the snapshot after the first data insertion. +------------+----------+------------+------------+-------+--------+---------------+ | user_id | username | age | salary | email | status | register_time | +------------+----------+------------+------------+-------+--------+---------------+ | 1001 | NULL | 25 | 8000.0 | zhangsan@email.com | ACTIVE | 2026-11-01 10:30:00 | | 1002 | NULL | 30 | 12000.0 | lisi@email.com | ACTIVE | 2026-11-02 14:20:00 | +------------+----------+------------+------------+-------+--------+---------------+ -- Current contents of the queried table +------------+----------+------------+------------+-------+--------+---------------+ | user_id | username | age | salary | email | status | register_time | +------------+----------+------------+------------+-------+--------+---------------+ | 1001 | NULL | 25 | 8000.0 | zhangsan@email.com | ACTIVE | 2026-11-01 10:30:00 | | 1002 | NULL | 30 | 12000.0 | lisi@email.com | ACTIVE | 2026-11-02 14:20:00 | | 1004 | 赵六 | 32 | 11000.0 | zhaoliu@email.com | INACTIVE | 2026-11-03 16:40:00 | | 1003 | 王五 | 28 | 9500.0 | wangwu@email.com | ACTIVE | 2026-11-03 09:15:00 | | 1004 | 赵六 | 32 | 11000.0 | zhaoliu@email.com | INACTIVE | 2026-11-03 16:40:00 | | 1005 | 孙七 | 35 | 15000.0 | sunqi@email.com | ACTIVE | 2026-11-04 08:20:00 | +------------+----------+------------+------------+-------+--------+---------------+
Iceberg external vs. managed tables
Dimension | Iceberg external table | MaxCompute-managed Iceberg table |
CREATE TABLE statement | | |
Table lifecycle | MaxCompute only maps the table. | MaxCompute manages the full lifecycle of metadata and OSS data. The specified OSS directory must be empty at creation. |
Metadata and data governance | Same as standard external tables. | High-performance metadata caching. |
Data openness (read/write by open-source engines) | Consistency guaranteed by the Iceberg format. | MaxCompute guarantees read/write consistency. Open-source engines read from OSS directly or use the MaxCompute Storage API. |
Lake table maintenance | Maintained by the user. | Automatic background maintenance:
|
Supported data types
Iceberg data type | MaxCompute data type | Read/write support |
Types.BooleanType | BOOLEAN | |
Types.IntegerType | INT | |
Types.LongType | BIGINT | |
Types.FloatType | FLOAT | |
Types.DoubleType | DOUBLE | |
Types.DecimalType | DECIMAL(precision, scale)
| |
Types.DateType | DATE | |
Types.TimeType | BIGINT
| |
Types.TimestampType | TIMESTAMP_NTZ | |
Types.TimestampType_z | TIMESTAMP | |
Types.StringType | STRING | |
Types.UUIDType | BINARY
| |
Types.FixedType | BINARY | |
Types.BinaryType | BINARY | |
TypeID.STRUCT | STRUCT | |
TypeID.LIST | ARRAY | |
TypeID.MAP | MAP | |
N/A | TINYINT, SMALLINT, VARCHAR(n), CHAR(n), DATETIME, JSON | N/A |
UPDATE/DELETE support
MaxCompute-managed Iceberg tables now support row-level updates and deletes through standard DML statements:
The optimizer splits an UPDATE into DELETE and INSERT sub-processes and commits them atomically within the same transaction. That is, data manifests and delete manifests are committed together in the same snapshot.
Partitioned tables are supported.
If an UPDATE or DELETE matches no rows, no empty snapshot is created.
Usage notes
Merging delete files during reads currently still requires the Java path. Therefore, run the following command before querying:
SET odps.sql.iceberg.native.enabled=false;UPDATE operations are not currently supported on partitioned tables whose partition definitions contain time functions.
Example
Create a test table and insert data.
CREATE ICEBERG TABLE data_process_log_nopt2 ( file_id BIGINT, file_path STRING, file_size BIGINT, process_status STRING, error_message STRING, retry_count INT, process_time TIMESTAMP, create_time TIMESTAMP ) WITH CONNECTION iceberg_connection OPTIONS( location='oss://oss-cn-<region>-internal.aliyuncs.com/<bucket path>/' ); -- Insert processing records INSERT INTO data_process_log_nopt2 VALUES (4001, 'oss://bucket/data/file1.csv', 1048576, 'PENDING', NULL, 0, NULL, TIMESTAMP '2026-11-03 08:00:00'), (4002, 'oss://bucket/data/file2.csv', 2097152, 'PENDING', NULL, 0, NULL, TIMESTAMP '2026-11-03 08:05:00'); -- Query the test table SELECT * FROM data_process_log_nopt2; -- The following result is returned: +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+ | file_id | file_path | file_size | process_status | error_message | retry_count | process_time | create_time | +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+ | 4001 | oss://bucket/data/file1.csv | 1048576 | PENDING | NULL | 0 | NULL | 2026-11-03 08:00:00 | | 4002 | oss://bucket/data/file2.csv | 2097152 | PENDING | NULL | 0 | NULL | 2026-11-03 08:05:00 | +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+Mark the data.
-- Mark processing as completed UPDATE data_process_log_nopt2 SET process_status = 'COMPLETED', error_message = NULL WHERE file_id = 4001 AND process_status = 'PENDING'; -- Query the test table SELECT * FROM data_process_log_nopt2; -- The following result is returned: +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+ | file_id | file_path | file_size | process_status | error_message | retry_count | process_time | create_time | +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+ | 4002 | oss://bucket/data/file2.csv | 2097152 | PENDING | NULL | 0 | NULL | 2026-11-03 08:05:00 | | 4001 | oss://bucket/data/file1.csv | 1048576 | COMPLETED | NULL | 0 | NULL | 2026-11-03 08:00:00 | +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+Delete data.
-- Delete records that failed and exceeded the retry limit DELETE FROM data_process_log_nopt2 WHERE process_status = 'PENDING'; -- Query the test table SELECT * FROM data_process_log_nopt2; -- The following result is returned: +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+ | file_id | file_path | file_size | process_status | error_message | retry_count | process_time | create_time | +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+ | 4001 | oss://bucket/data/file1.csv | 1048576 | COMPLETED | NULL | 0 | NULL | 2026-11-03 08:00:00 | +------------+-----------+------------+----------------+---------------+-------------+--------------+-------------+
Schema evolution support
Data type changes support only conversions between primitive types that are allowed by the type promotion matrix in the Iceberg specification. Incompatible type changes return an error. For more information, see Column operations.
Operation type | Supported |
Add columns | |
Drop columns | |
Change column order | |
Change column data types | |
Rename columns | |
Change column comments | |
Change column nullability |
Partition evolution support
Iceberg partition evolution is supported. Example:
ALTER TABLE t ADD PARTITION FIELDS c1; -- identity
ALTER TABLE t ADD PARTITION FIELDS bucket(10, id) AS id_bucket;
ALTER TABLE t REPLACE PARTITION FIELDS c1 WITH bucket(5, c3) AS bucket_c3;
ALTER TABLE t REPLACE PARTITION FIELDS year(ts) WITH month(ts);
ALTER TABLE t DROP PARTITION FIELDS region;After partition evolution is complete, the changes can be viewed in the data directory in OSS.
Limitations
Version limitations
Only features compatible with Iceberg SDK 1.6.1 are supported.
Reading and writing are supported only for Iceberg table format v2.
Features of the Iceberg v3 format are not supported.
Supported operations
MaxCompute data types: Data types (Version 1.0) and Data types (Version 2.0).
Partition pruning is supported. If not enabled, submit feedback through the Application Link or join the MaxCompute Developer Community DingTalk group (group ID: 11782920).
Unsupported operations
For MaxCompute-managed Iceberg tables:
The Rename Table statement is not supported.
The XCOPY Cross-Region Replication statement is not supported.
The CLONE TABLE and RENAME TABLE statements are not supported.
Statements to rename table snapshots are not supported.
Setting default values for columns is not supported.
Materialized view operations are not supported.
Row-level access control is not supported.
The TRUNCATE statement is not supported.
CDC (private preview) is not supported.
Local backup is not supported.
The
CREATE OR REPLACEstatement is not supported.The
LOADandUNLOADcommands are not supported.
Console queries and API responses display the storage size of managed Iceberg tables as 0 bytes.