Iceberg data lake table format - EMR - E-MapReduce - Alibaba Cloud Documentation Center

Core capabilities

Apache Iceberg started as a solution for migrating Hive data warehouses to the cloud and has since become the standard table format for cloud-based data lake services. Visit the official Apache Iceberg website for details.

Iceberg provides the following capabilities:

Low-cost data lake storage on HDFS or Object Storage Service (OSS).
Seamless integration with open source compute engines for ingestion and analysis.
Full ACID semantics.
Row-level data changes.
Historical version rollbacks.
Efficient data filtering.
Schema evolution.
Partition layout evolution.
Hidden partitioning.

The following table compares three architectures: open source ClickHouse (real-time data warehouse), open source Hive (offline data warehouse), and Alibaba Cloud Iceberg (data lake).

Comparison Criteria	Subitem	Open source ClickHouse real-time data warehouse	Open source Hive offline data warehouse	Alibaba Cloud Iceberg data lake
System architecture	Architecture	Coupled compute and storage	Decoupled compute and storage	Decoupled compute and storage
	Support for multiple compute engines	Not supported	Supported	Supported
	Data storage on object storage	Not supported	Partially supported	Supported
	Data storage on HDFS	Not supported	Supported	Supported
	Openness of storage format	Disabled	Open	Open
Business value	Timeliness	Seconds	Hours/Days	Minutes
	Compute flexibility	Low	Strong	Strong
	Transactional	Not supported	Incomplete	Supported
	Universality of table-level semantics	Poor	Poor	Excellent
	Row-level data changes	Not supported	Weakly supported	Supported
	Data quality	Very high	High	High
Maintenance cost	Query performance	High	High	High
	Storage cost	Very high	Medium	Low
	Self-service	Not supported	Not supported	Supported
	Resource elasticity	General	General	Excellent

Comparison with open source Iceberg

The following table compares Alibaba Cloud Iceberg with open source Iceberg across basic features, data changes, and compute engines.

Note

√ indicates that the feature is supported. x indicates that the feature is not yet supported.

Category	Item	Subitem	Open source Iceberg	Iceberg Commercial Edition (Alibaba Cloud)
Basic features	ACID	N/A	√	√
	Historical version rollbacks	N/A	√	√
	Source and Sink integration	Batch	√	√
	Source and Sink integration	Streaming	√	√
	Efficient data filtering	N/A	√	√
Data changes	Schema evolution	N/A	√	√
	Partition evolution	N/A	√	√
	Copy-on-write updates	N/A	√	√
	Merge-on-read updates	Read	√	√
		Write	√	√
		Compaction	x	x
Compute engines	Apache Spark	Read	√	√
	Apache Spark	Write	√	√
	Apache Hive	Read	√	√
	Apache Hive	Write	√	√
	Apache Flink	Read	√	√
	Apache Flink	Write	√	√
	PrestoDB or Trino	Read	√	√
	PrestoDB or Trino	Write	√	√
Programming languages	Java	N/A	√	√
Programming languages	Python	N/A	√	√
Advanced features	Native integration with Alibaba Cloud OSS	N/A	x	√
	Native integration with Alibaba Cloud DLF	N/A	x	√
	Local data cache acceleration	N/A	x	√
	Automatic small file compaction	N/A	x	√

Note

This comparison was created in September 2021 based on an analysis of open source Iceberg and the commercial edition of Iceberg at that time. Feature support may change with future version upgrades.

Scenarios

Iceberg is a core component in data lake solutions, suited for the following scenarios.

Scenario	Description
Real-time data import and query	Upstream data streams into the Iceberg data lake in real time and is immediately queryable. For example, in a logging scenario, start an Iceberg or Spark streaming job to write logs into an Iceberg table. Then query the data with Hive, Spark, Iceberg, or Presto. ACID support ensures isolation between ingestion and queries, preventing dirty reads.
Delete or update data	Traditional data warehouses struggle to perform row-level deletes or updates efficiently. These operations typically require running an offline job that reads the entire original table, modifies the data, and writes it back. Iceberg narrows changes from the table level to the file level, enabling efficient partial updates. In an Iceberg data lake, you can directly change data in a table by running a command such as `DELETE FROM test_table WHERE id > 10`.
Data quality control	Use Iceberg schema validation to discard or further process abnormal data during import.
Data schema evolution	Iceberg supports schema changes through Spark SQL DDL statements without rewriting historical data, making evolution fast. ACID support isolates schema changes from existing read jobs, ensuring consistent reads throughout the process.
Real-time machine learning	In machine learning workflows, data processing (cleansing, transformation, feature extraction) dominates time. Iceberg unifies historical and real-time data into a single reliable stream where each processing step is a node in the pipeline. This eliminates separate batch and streaming paths. Iceberg also provides a native Python SDK for algorithm developers.