This topic describes how to configure Data Lake Formation (DLF) metadata for Iceberg tables.
The following configurations are supported:
Spark configuration
This configuration uses Object Storage Service (OSS) as the file system. The default catalog name and required parameters vary by E-MapReduce (EMR) cluster version. The following sections describe the configurations for specific versions.
-
EMR-3.40 and later, and EMR-5.6.0 and later
NoteThe default catalog name is
iceberg.Parameter
Description
Remarks
spark.sql.extensions
The Spark SQL extension module.
Set this parameter to org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.
NoteThis parameter was introduced in Iceberg 0.11.0 and is supported only in Spark 3.x.
spark.sql.catalog.<catalog-name>
The catalog name.
Set this parameter to org.apache.iceberg.spark.SparkCatalog.
spark.sql.catalog.<catalog-name>.catalog-impl
The catalog's implementation class.
Set this parameter to org.apache.iceberg.aliyun.dlf.hive.DlfCatalog.
-
EMR-3.39.x and EMR-5.5.x
NoteThe default catalog name is
dlf.Parameter
Description
Remarks
spark.sql.extensions
The Spark SQL extension module.
Set this parameter to org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.
NoteThis parameter was introduced in Iceberg 0.11.0 and is supported only in Spark 3.x.
spark.sql.catalog.<catalog-name>
The catalog name.
Set this parameter to org.apache.iceberg.spark.SparkCatalog.
spark.sql.catalog.<catalog-name>.catalog-impl
The catalog's implementation class.
Set this parameter to org.apache.iceberg.aliyun.dlf.hive.DlfCatalog.
-
EMR-3.38.x, and EMR-5.3.x to EMR-5.4.x (inclusive)
NoteThe default catalog name is
dlf_catalog.Parameter
Description
Remarks
spark.sql.extensions
The Spark SQL extension module.
Set this parameter to org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.
NoteThis parameter was introduced in Iceberg 0.11.0 and is supported only in Spark 3.x.
spark.sql.catalog.<catalog-name>
The catalog name.
Set this parameter to org.apache.iceberg.spark.SparkCatalog.
spark.sql.catalog.<catalog-name>.catalog-impl
The catalog's implementation class.
Set this parameter to org.apache.iceberg.aliyun.dlf.DlfCatalog.
spark.sql.catalog.<catalog-name>.io-impl
The I/O implementation class.
Set this parameter to org.apache.iceberg.hadoop.HadoopFileIO.
spark.sql.catalog.<catalog-name>.oss.endpoint
The Object Storage Service (OSS) endpoint.
For more information, see Regions and endpoints.
We recommend that you set oss.endpoint to the OSS VPC endpoint. For example, if your region is China (Hangzhou), set oss.endpoint to
oss-cn-hangzhou-internal.aliyuncs.com.NoteIf you access OSS across VPCs, set oss.endpoint to the OSS public endpoint.
spark.sql.catalog.<catalog-name>.warehouse
The OSS path where table data is stored.
None.
spark.sql.catalog.<catalog-name>.access.key.id
The AccessKey ID for your Alibaba Cloud account.
For more information, see Obtain an AccessKey pair.
spark.sql.catalog.<catalog-name>.access.key.secret
The AccessKey secret for your Alibaba Cloud account.
For more information, see Obtain an AccessKey pair.
spark.sql.catalog.<catalog-name>.dlf.catalog-id
Your Alibaba Cloud account ID.
To find your account ID, go to the Security Settings page. You can also log on to the Alibaba Cloud console and find your Account ID on the Account Management page.
spark.sql.catalog.<catalog-name>.dlf.endpoint
The DLF service endpoint.
We recommend that you set dlf.endpoint to the DLF VPC endpoint. For example, if your region is China (Hangzhou), set dlf.endpoint to
dlf-vpc.cn-hangzhou.aliyuncs.com.NoteYou can also use the DLF public endpoint. For example, if your region is China (Hangzhou), set dlf.endpoint to
dlf.cn-hangzhou.aliyuncs.com.spark.sql.catalog.<catalog-name>.dlf.region-id
The DLF service region ID.
Ensure this region matches the one specified for dlf.endpoint.
Hive configuration
Configure the parameters based on your cluster's version.
-
EMR-3.39.0 and later, and EMR-5.5.0 and later
NoteThe default catalog name is
dlf.Parameter
Description
Remarks
iceberg.catalog.<catalog-name>.catalog-impl
The catalog's implementation class.
Set this parameter to org.apache.iceberg.aliyun.dlf.hive.DlfCatalog.
-
EMR-3.38.x, and EMR-5.3.x to EMR-5.4.x (inclusive)
NoteThe default catalog name is
dlf_catalog.Parameter
Description
Remarks
iceberg.catalog
The catalog name.
Specify a custom name.
iceberg.catalog.<catalog-name>.type
The catalog type.
Set this parameter to custom.
iceberg.catalog.<catalog-name>.catalog-impl
The catalog's implementation class.
Set this parameter to org.apache.iceberg.aliyun.dlf.DlfCatalog.
iceberg.catalog.<catalog-name>.io-impl
The I/O implementation class.
Set this parameter to org.apache.iceberg.hadoop.HadoopFileIO.
iceberg.catalog.<catalog-name>.warehouse
The warehouse path where table data is stored.
You can specify an HDFS path or an OSS path.
iceberg.catalog.<catalog-name>.access.key.id
The AccessKey ID for your Alibaba Cloud account.
For more information, see Obtain an AccessKey pair.
iceberg.catalog.<catalog-name>.access.key.secret
The AccessKey secret for your Alibaba Cloud account.
For more information, see Obtain an AccessKey pair.
iceberg.catalog.<catalog-name>.dlf.catalog-id
Your Alibaba Cloud account ID.
To find your account ID, go to the Security Settings page.
iceberg.catalog.<catalog-name>.dlf.endpoint
The DLF service endpoint.
We recommend that you set dlf.endpoint to the DLF VPC endpoint. For example, if your region is China (Hangzhou), set dlf.endpoint to
dlf-vpc.cn-hangzhou.aliyuncs.com.NoteYou can also use the DLF public endpoint. For example, if your region is China (Hangzhou), set dlf.endpoint to
dlf.cn-hangzhou.aliyuncs.com.iceberg.catalog.<catalog-name>.dlf.region-id
The DLF service region ID.
Ensure this region matches the one specified for dlf.endpoint.