Data Lake Formation (DLF) integrates with major big data compute engines for real-time and offline lakehouse workloads and Online Analytical Processing (OLAP), with native support for Flink (VVP), EMR Serverless Spark, EMR Serverless StarRocks, and EMR on ECS.
Supported engines and connection methods
The following table lists the supported engines and tools for each integration method:
|
Integration method |
Supported engines/tools |
|
Paimon REST |
|
|
Iceberg REST |
|
|
File access |
|
|
Lance |
Self-managed Spark clusters are not supported. To use Spark with DLF, choose Alibaba Cloud's EMR Serverless Spark or EMR on ECS.
Integration methods
DLF provides four integration methods, each designed for a different engine type and access pattern:
-
Paimon REST: A RESTful metadata service interface that follows Apache Paimon community standards. Choose this method when your compute engine is built on Apache Paimon. It handles table schema management and snapshot queries.
-
Iceberg REST: A RESTful metadata service interface that follows Apache Iceberg community standards. Choose this method when your compute engine is built on Apache Iceberg. It handles table schema management and snapshot queries.
-
File access: Exposes table data as standard file paths through the Paimon Virtual File System (PVFS), giving you direct access to underlying data files and metadata without a full compute engine. This method is well suited for scripted exploration, debugging, and lightweight data processing.
-
Lance: Connects to DLF Catalog through the Python SDK (lance-dlf) to read and write Lance tables. It supports query filtering and batch processing with the Daft DataFrame engine.