Query Iceberg Tables on EMR Spark with DLF REST Catalog - Data Lake Formation - Alibaba Cloud - Data Lake Formation

Prerequisites

Version requirements: An EMR cluster of version 5.12.0 or later with the Spark 3 component. Spark must use JDK 17 (required by Iceberg 1.11.0 — configure it in the same way as described in Use JDK 11 in Spark 3).
Region requirements: The EMR cluster and DLF must be in the same region, and the VPC of the cluster must be added to the DLF allowlist.
Permission requirements: You must have an AccessKey pair with permissions to access DLF, and the corresponding RAM user must be granted data permissions on the target catalog. For more information, see Manage data permissions.

Dependencies

Only the following two Apache Iceberg community artifacts from Maven Central are required:

iceberg-spark-runtime-3.5_2.12 (1.11.0 and later)
iceberg-aws-bundle (1.11.0 and later)

You can use the --packages flag to automatically pull the dependencies when submitting a job (see the following example). For offline clusters, you can also download the JAR files and place them in $SPARK_HOME/jars.

Usage example

Configure a Catalog connection

Run the spark-sql command in a terminal. Replace the placeholders with your actual values.

spark-sql \
  --master local \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.iceberg_catalog=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.iceberg_catalog.catalog-impl=org.apache.iceberg.rest.RESTCatalog \
  --conf spark.sql.catalog.iceberg_catalog.uri=http://${regionID}-vpc.dlf.aliyuncs.com/iceberg \
  --conf spark.sql.catalog.iceberg_catalog.warehouse=${catalogName} \
  --conf spark.sql.catalog.iceberg_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  --conf spark.sql.catalog.iceberg_catalog.rest.auth.type=sigv4 \
  --conf spark.sql.catalog.iceberg_catalog.rest.auth.sigv4.delegate-auth-type=none \
  --conf spark.sql.catalog.iceberg_catalog.rest.signing-region=${regionID} \
  --conf spark.sql.catalog.iceberg_catalog.rest.signing-name=DlfNext \
  --conf spark.sql.catalog.iceberg_catalog.rest.access-key-id=${AccessKeyId} \
  --conf spark.sql.catalog.iceberg_catalog.rest.secret-access-key=${AccessKeySecret}

The following table describes the configuration items:

Configuration item	Description	Example
`uri`	The VPC endpoint of the DLF Iceberg REST service, in the format `http://<regionID>-vpc.dlf.aliyuncs.com/iceberg`. For more information, see Service access point	`http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg`
`warehouse`	The catalog name	`iceberg_test`
`catalog-impl`	Fixed value: `org.apache.iceberg.rest.RESTCatalog`	`org.apache.iceberg.rest.RESTCatalog`
`io-impl`	The standard Iceberg community implementation. Fixed value: `org.apache.iceberg.aws.s3.S3FileIO`	`org.apache.iceberg.aws.s3.S3FileIO`
`rest.auth.type`	The auth type. Fixed value: `sigv4`	`sigv4`
`rest.auth.sigv4.delegate-auth-type`	Fixed value: `none`	`none`
`rest.signing-region`	The DLF region ID	`cn-hangzhou`
`rest.signing-name`	Fixed value: `DlfNext`	`DlfNext`
`rest.access-key-id`	The AccessKey ID used to access DLF
`rest.secret-access-key`	The AccessKeySecret used to access DLF

Read and write DLF Iceberg tables

After startup, you can use standard Spark SQL to read and write data:

CREATE DATABASE IF NOT EXISTS iceberg_catalog.db;
CREATE TABLE iceberg_catalog.db.iceberg_tbl (id BIGINT, name STRING) USING iceberg;
INSERT INTO iceberg_catalog.db.iceberg_tbl VALUES (1, 'hello'), (2, 'world');
SELECT * FROM iceberg_catalog.db.iceberg_tbl;