Access Iceberg catalogs in DLF using Flink SQL - Data Lake Formation

This topic describes how to access a Data Lake Formation (DLF) catalog using Flink SQL via Iceberg REST.

Prerequisites

You have created a Realtime Compute for Apache Flink workspace. For instructions, see Activate Realtime Compute for Apache Flink.
You have created a session cluster with an engine version of Ververica Runtime (VVR) 11.2.0 or later. See Create a session cluster.

Engine requirements

Only VVR 11.2.0+ versions are supported.

Register the DLF catalog in Flink

Note

This operation creates a mapping to your DLF catalog. Creating or deleting the catalog in Flink does not affect actual data in DLF.
All tables created in the DLF catalog via Iceberg REST are Iceberg tables.

Log on to Realtime Compute for Apache Flink's Management Console.
In the Actions column of your workspace, click Console.
In the left navigation menu, click Development > Scripts.

Create a new script. In the SQL editor, copy and paste the following SQL statement. In the lower-right corner, click Environment, select a session cluster of VVR 11.2.0+, and run the SQL statement to register a DLF catalog via Iceberg REST.

CREATE CATALOG `catalog_name` 
 WITH (
    'type' = 'iceberg',
    'catalog-type' = 'rest',
    'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg',
    'warehouse' = 'iceberg_test',
    'rest.signing-region' = 'cn-hangzhou',
    'io-impl' = 'org.apache.iceberg.rest.DlfFileIO'
);

The following table describes the options.

Option	Description	Required	Example
type	The type. Set this to `iceberg`.	Yes	iceberg
catalog-type	The catalog type. Set this to `rest`.	Yes	rest
token.provider	The token provider. Set this to dlf.	Yes	dlf
uri	The URI used to access the DLF catalog via Iceberg REST. For more information, see Regions and endpoints.	Yes	http://ap-southeast-1-vpc.dlf.aliyuncs.com/iceberg
warehouse	The name of your DLF catalog.	Yes	iceberg_test
rest.signing-region	The region ID of DLF. For more information, see Regions and endpoints.	Yes	ap-southeast-1
io-impl	Set this to `org.apache.iceberg.rest.DlfFileIO`.	Yes	org.apache.iceberg.rest.DlfFileIO

Query data in the catalog

To query databases and tables in the registered catalog, run the following SQL statement in Scripts:

SELECT * FROM catalog_name.database_name.table_name;