Access DLF using the Java API - Data Lake Formation - Alibaba Cloud Documentation Center

This topic describes how to access the DLF Iceberg REST Catalog using the open-source Apache Iceberg Java API, Spark, and Flink. You can manage table metadata and read and write Iceberg tables through the standard Iceberg REST protocol with AWS SigV4 signing. The data plane uses the community-standard S3FileIO to connect directly to OSS S3-compatible endpoints.

Prerequisites

Your runtime environment must use JDK 17 or later (required by Iceberg 1.11.0).
Add the following Apache Iceberg community dependencies (version 1.11.0) to your Maven project. All dependencies are available from Maven Central:

<dependencies>
    <dependency>
        <groupId>org.apache.iceberg</groupId>
        <artifactId>iceberg-core</artifactId>
        <version>1.11.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.iceberg</groupId>
        <artifactId>iceberg-aws</artifactId>
        <version>1.11.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.iceberg</groupId>
        <artifactId>iceberg-aws-bundle</artifactId>
        <version>1.11.0</version>
    </dependency>
</dependencies>

Note

If you need to read and write Parquet data directly using the Java API (without a compute engine), also add the iceberg-data, iceberg-parquet, iceberg-orc, parquet-hadoop, and hadoop-common dependencies.

Connect using the Java API

Use the community-standard RESTCatalog class to connect to the DLF Iceberg REST Catalog. Pass the following configuration parameters when initializing:

Map<String, String> props = new HashMap<>();
props.put("uri", "http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg");
props.put("warehouse", "iceberg_table_test");
props.put("rest.auth.type", "sigv4");
props.put("rest.auth.sigv4.delegate-auth-type", "none");
props.put("rest.signing-region", "cn-hangzhou");
props.put("rest.signing-name", "DlfNext");
props.put("rest.access-key-id", "xxx");
props.put("rest.secret-access-key", "yyy");
props.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO");
RESTCatalog icebergCatalog = new RESTCatalog();
icebergCatalog.initialize(catalogName, props);

Configuration parameters

Parameter	Description	Example
`uri`	DLF Iceberg REST Catalog endpoint (accessible within the VPC). The format is `http://<region>-vpc.dlf.aliyuncs.com/iceberg`. For a complete list of service endpoints by region, see Endpoints and public network access.	`http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg`
`warehouse`	DLF catalog name.	`iceberg_test`
`rest.auth.type`	Authentication type. Set to `sigv4`.	`sigv4`
`rest.auth.sigv4.delegate-auth-type`	Delegated authentication type. Set to `none`.	`none`
`rest.signing-region`	Region ID of the DLF instance.	`cn-hangzhou`
`rest.signing-name`	Signing service name. Set to `DlfNext`.	`DlfNext`
`rest.access-key-id`	AccessKey ID for accessing DLF.
`rest.secret-access-key`	AccessKey Secret for accessing DLF.
`io-impl`	File IO implementation class. Uses the community-standard S3FileIO. Set to `org.apache.iceberg.aws.s3.S3FileIO`.	`org.apache.iceberg.aws.s3.S3FileIO`

Access with Apache Spark

Open-source Apache Spark can read and write DLF Iceberg tables directly using the Iceberg community runtime. The catalog communicates through the standard Iceberg REST protocol with AWS SigV4 signing, while the data plane uses the community S3FileIO to connect to OSS S3-compatible endpoints over the S3-compatible API. All dependencies come from Maven Central.

Start Spark SQL

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.dlf=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.dlf.type=rest \
  --conf spark.sql.catalog.dlf.uri=http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg \
  --conf spark.sql.catalog.dlf.warehouse=iceberg_table_test \
  --conf spark.sql.catalog.dlf.rest.auth.type=sigv4 \
  --conf spark.sql.catalog.dlf.rest.auth.sigv4.delegate-auth-type=none \
  --conf spark.sql.catalog.dlf.rest.signing-region=cn-hangzhou \
  --conf spark.sql.catalog.dlf.rest.signing-name=DlfNext \
  --conf spark.sql.catalog.dlf.rest.access-key-id=xxx \
  --conf spark.sql.catalog.dlf.rest.secret-access-key=yyy \
  --conf spark.sql.catalog.dlf.io-impl=org.apache.iceberg.aws.s3.S3FileIO

Read and write with Spark SQL

Once configured, use standard Spark SQL to read and write Iceberg tables:

CREATE DATABASE IF NOT EXISTS dlf.demo_db;
CREATE TABLE dlf.demo_db.demo_tbl (id BIGINT, name STRING) USING iceberg;
INSERT INTO dlf.demo_db.demo_tbl VALUES (1, 'hello'), (2, 'world');
SELECT * FROM dlf.demo_db.demo_tbl;

Access with Apache Flink

Open-source Apache Flink uses the Iceberg community runtime to access DLF Iceberg tables. Place iceberg-flink-runtime-1.20-1.11.0.jar and iceberg-aws-bundle-1.11.0.jar (both community artifacts from Maven Central) in the lib/ directory of your Flink installation, then register the catalog in the SQL Client.

Register the catalog

CREATE CATALOG dlf WITH (
  'type' = 'iceberg',
  'catalog-type' = 'rest',
  'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg',
  'warehouse' = 'iceberg_table_test',
  'rest.auth.type' = 'sigv4',
  'rest.auth.sigv4.delegate-auth-type' = 'none',
  'rest.signing-region' = 'cn-hangzhou',
  'rest.signing-name' = 'DlfNext',
  'rest.access-key-id' = 'xxx',
  'rest.secret-access-key' = 'yyy',
  'io-impl' = 'org.apache.iceberg.aws.s3.S3FileIO'
);

Read and write with Flink SQL

USE CATALOG dlf;
CREATE DATABASE IF NOT EXISTS demo_db;
CREATE TABLE demo_db.demo_tbl (id BIGINT, name STRING);
INSERT INTO demo_db.demo_tbl VALUES (1, 'hello'), (2, 'world');
SELECT * FROM demo_db.demo_tbl;