DLF Catalog - E-MapReduce - Alibaba Cloud Documentation Center

Background information

Alibaba Cloud Data Lake Formation (DLF) is a fully managed platform for unified metadata and data storage and management.

Use DLF

Prerequisites

A Serverless StarRocks instance is created. For more information, see Create an instance.

The instance must be version 3.3 or later, and the Minor Version must be 3.3.8-1.99 or later.

Note
You can view the minor version information in the Version Information area on the Instance Details page. If the minor version is earlier than 3.3.8-1.99, you must upgrade it. For more information, see Upgrade a minor version.
A data catalog is created in DLF.

Example of using a DLF Catalog

Step 1: Prepare a user with an Alibaba Cloud identity in Serverless StarRocks

Important

Access control for DLF is managed at the Resource Access Management (RAM) level. By default, StarRocks users do not have permissions on DLF. You must ensure that the StarRocks user who needs to access DLF has an Alibaba Cloud identity (RAM user or RAM role) and grant the required permissions. If you have not created a RAM user, see Create a RAM user.

You can assign an Alibaba Cloud identity to a StarRocks user in either of the following ways. Both methods allow the user to access DLF Catalogs. Choose the one that best fits your scenario:

Method 1: Select a RAM user as the source when creating a new user. Use this method when you want to create a dedicated StarRocks user for accessing DLF.
Method 2: Associate an existing user with a RAM user or RAM role. Use this method when you already have a StarRocks user (regular user) that needs to access DLF, or when you need to access DLF as a RAM role (including STS temporary identities).

Navigate to the user management page

Go to the E-MapReduce Serverless StarRocks instance list page.
1. Log on to the E-MapReduce console.
2. In the navigation pane on the left, choose EMR Serverless > StarRocks.
3. In the top menu bar, select the required region.
On the Instances page, click Connect in the Actions column for the created instance. For more information, see Connect to a StarRocks instance using EMR StarRocks Manager.

Connect to the StarRocks instance as the admin user or a super administrator.
In the left navigation pane, choose Security Center > Users.

Method 1: Select a RAM user when creating a new user

On the Users page, click Add User.
In the Add User dialog box, configure the following parameters and click OK.
- User Source: Select RAM User.
- Username: Select the RAM user to authorize (for example, dlf-test).
- Password and Confirm Password: Set a custom password.
- Role: The default value public is sufficient.

Method 2: Associate an existing user with a RAM user or RAM role

On the Users page, find the target StarRocks user and click Associate RAM User/Role in the Actions column.

In the association dialog box, select the RAM user or RAM role to associate, and then click OK.

The selectable range depends on the identity you used to log on to the console:

Current logon identity	Associable objects
Alibaba Cloud account (primary account)	Any RAM user or RAM role under the account
RAM user (sub-account)	Only the RAM user currently logged on
STS temporary identity (assumed RAM role)	Only the RAM role currently assumed

After the association is successful, the User Type of the user in the Users list will automatically change to RAM User or RAM Role.

Note

The association can be revoked at any time. On the Users page, click Dissociate in the Actions column for the user. After dissociation, the User Type reverts to Regular User, and the user loses the ability to access DLF and other cloud services through the Alibaba Cloud identity.

When associated with a RAM role, StarRocks uses the STS temporary credentials of that role to access DLF. Therefore, you must grant permissions in DLF to the RAM Role (not to the assumable identity). See Step 2 for details.

Step 2: Grant permissions to the catalog in DLF

Log on to the Data Lake Formation console.
On the Catalogs page, click the name of the target catalog.
Click the Permission tab, and then click Authorize.
From the Authorization Object drop-down list, select the Alibaba Cloud identity used in Step 1:
- If the StarRocks user is associated with a RAM user, select that RAM user (for example, dlf-test).
- If the StarRocks user is associated with a RAM role, select that RAM role.
For Preset Permission Type, select Custom, and grant the user the ALL permission on the current data catalog and all resources within it.
Click OK.

Step 3: Create a DLF Catalog in Serverless StarRocks

Paimon Catalog

Connect to the instance. For more information, see Connect to a StarRocks instance using EMR StarRocks Manager.

Important
Reconnect to the StarRocks instance as the StarRocks user with an Alibaba Cloud identity from Step 1 (i.e., a user whose User Type is RAM User or RAM Role, such as dlf-test). You must use this user to create SQL queries that access the DLF external catalog. Regular users without an Alibaba Cloud identity cannot access DLF.
Create an SQL query. On the Queries page of the SQL Editor, click the icon to create a query task.

Create a Paimon Catalog. Enter the following SQL statement and click Run.

CREATE EXTERNAL CATALOG `dlf_catalog`
PROPERTIES (
'type' = 'paimon',
'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com',
'paimon.catalog.type' = 'rest',
'paimon.catalog.warehouse' = 'StarRocks_test',
'token.provider' = 'dlf'
);

Read and write data.

Create a database.

CREATE DATABASE IF NOT EXISTS dlf_catalog.sr_dlf_db;

Create a table.

CREATE TABLE dlf_catalog.sr_dlf_db.ads_age_pvalue_analytics(
    final_gender_code STRING COMMENT 'Gender',
    age_level STRING COMMENT 'Age Level',
    pvalue_level STRING COMMENT 'Consumption Level',
    clicks INT COMMENT 'Clicks',
    total_behaviors INT COMMENT 'Total Behaviors'
);

Insert data.

INSERT INTO dlf_catalog.sr_dlf_db.ads_age_pvalue_analytics (final_gender_code, age_level, pvalue_level, clicks, total_behaviors)
VALUES 
('M', '18-24', 'Low', 1500, 2500),
('F', '25-34', 'Medium', 2200, 3300),
('M', '35-44', 'High', 2800, 4000);

Query data.
```
SELECT * FROM dlf_catalog.sr_dlf_db.ads_age_pvalue_analytics;
```
The query result contains five columns: final_gender_code, age_level, pvalue_level, clicks, and total_behaviors, confirming that the DLF Catalog can access the user behavior analysis data in the target table.

Iceberg Catalog

Connect to the instance. For more information, see Connect to a StarRocks instance using EMR StarRocks Manager.

Important
Reconnect to the StarRocks instance as the StarRocks user with an Alibaba Cloud identity from Step 1 (i.e., a user whose User Type is RAM User or RAM Role, such as dlf-test). You must use this user to create SQL queries that access the DLF external catalog. Regular users without an Alibaba Cloud identity cannot access DLF.
Create an SQL query task. On the Queries page of the SQL Editor, click the icon to create a query task.

Create an Iceberg Catalog. Enter the following SQL statement and click Run.

CREATE EXTERNAL CATALOG `iceberg_catalog`
PROPERTIES
( 
    'type' = 'iceberg',
    'iceberg.catalog.type' = 'dlf_rest',
    'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg',
    'warehouse' = 'iceberg_test',
    'rest.signing-region' = 'cn-hangzhou'
);

Query data.

Note
Iceberg external tables are read-only in StarRocks. You can run SELECT queries but cannot write data to Iceberg tables from StarRocks.
```
select * from  iceberg_catalog.`default`.test_iceberg;
```
The query returns two columns, id and name, with three records: (1, Alice), (2, Bob), and (3, Charlie), confirming that the Iceberg Catalog can read data from the target table.

Use DLF 1.0 (Legacy)

Prerequisites

A Serverless StarRocks instance is created. For more information, see Create an instance.
A data catalog is created in DLF 1.0 (Legacy). For more information, see Data Catalog.

Create a catalog

Create a Hive Catalog

Syntax

CREATE EXTERNAL CATALOG <catalog_name>
[COMMENT <comment>]
PROPERTIES
(
    "type" = "hive",                                    -- required
    GeneralParams,
    MetastoreParams
)

Parameters

General parameters

Parameter	Required	Description
`catalog_name`	Yes	Catalog name. Must start with a letter and can contain letters (a-z, A-Z), numbers (0-9), and underscores (_). Maximum 64 characters.
`comment`	No	Catalog description.
`type`	Yes	Data source type. Set to `hive`.
`enable_recursive_listing`	No	Specifies whether to recursively read files in subdirectories of a table or partition directory. Default: `true`. Set to `false` to read only the current directory level.

Metastore parameters

Choose the parameter set that matches your metadata service.

Use HMS

Parameter	Description
`hive.metastore.type`	Metastore type. Leave blank or omit this parameter to use HMS. Default: empty.
`hive.metastore.uris`	HMS URI in the format `thrift://<IP_address>:<port>`. Default port: `9083`.

Use DLF 1.0 (Legacy)

Parameter	Description
`hive.metastore.type`	Metastore type. Set to `dlf`.
`dlf.catalog.id`	ID of an existing data catalog in DLF 1.0. If not specified, the default DLF Catalog is used.

Example

CREATE EXTERNAL CATALOG hive_catalog
PROPERTIES
(
    "type" = "hive",
    "hive.metastore.type" = "dlf",
    "dlf.catalog.id" = "sr_dlf"
);

For more information, see Hive Catalog.

Create an Iceberg Catalog

Syntax

CREATE EXTERNAL CATALOG <catalog_name>
[COMMENT <comment>]
PROPERTIES
(
    "type" = "iceberg",
    MetastoreParams
)

Parameters

catalog_name: Required. The name of the Iceberg Catalog. The naming conventions are as follows:
- It must start with a letter and can contain only letters (a-z or A-Z), digits (0-9), or underscores (_).
- The name cannot exceed 64 characters.
- The catalog name is case-sensitive.
comment: Optional. The description of the Iceberg Catalog.
type: The type of the data source. Set to iceberg.

MetastoreParams: Parameters for StarRocks to access the metadata service of the Iceberg cluster.

Property	Description
iceberg.catalog.type	The Iceberg catalog type. Set to `dlf`.
dlf.catalog.id	The ID of an existing DLF data catalog. If `dlf.catalog.id` is not set, StarRocks uses the default DLF catalog.

Example

CREATE EXTERNAL CATALOG iceberg_catalog_hms
PROPERTIES
(
    "type" = "iceberg",
    "iceberg.catalog.type" = "dlf",
    "dlf.catalog.id" = "sr_dlf"
);

For more information, see Iceberg Catalog.

Create a Paimon Catalog

Syntax

CREATE EXTERNAL CATALOG <catalog_name>
[COMMENT <comment>]
PROPERTIES
(
    "type" = "paimon",
    CatalogParams,
    StorageCredentialParams
);

Parameters

Common parameters

Parameter	Required	Description
`catalog_name`	Yes	The name of the Paimon catalog. Must start with a letter and contain only letters (a–z, A–Z), digits (0–9), or underscores (\_). Maximum 64 characters.
`comment`	No	A description of the Paimon catalog.
`type`	Yes	The data source type. Set to `paimon`.

CatalogParams

CatalogParams specifies how StarRocks connects to the Paimon metadata service. Configuration varies by metadata type.

Use HMS

Property	Required	Description
`paimon.catalog.type`	Yes	Set to `hive`.
`hive.metastore.uris`	Yes	The HMS URI. Format: `thrift://<HMS IP address>:<port>`. Default port: 9083. For high-availability mode, specify multiple addresses separated by commas: `"thrift://<ip1>:<port1>,thrift://<ip2>:<port2>"`.

Use filesystem

Property	Required	Description
`paimon.catalog.type`	Yes	Set to `filesystem`.
`paimon.catalog.warehouse`	Yes	The storage path of the warehouse where Paimon data is stored. Supported formats: `oss://<yourBucketName>/<yourPath>` for OSS or OSS-HDFS, or an HDFS path. If you use OSS or OSS-HDFS, also configure `StorageCredentialParams`.

Use DLF 1.0 (Legacy)

Property	Required	Description
`paimon.catalog.type`	Yes	Set to `dlf`.
`paimon.catalog.warehouse`	Yes	The storage path of the warehouse. Format: `oss://<yourBucketName>/<yourPath>` for OSS or OSS-HDFS. If you use OSS or OSS-HDFS, also configure `StorageCredentialParams`.
`dlf.catalog.id`	No	The ID of an existing data catalog in DLF. If omitted, the system uses the default DLF catalog.

Use DLF

Important

If you use DLF, use a configured RAM user to perform operations in StarRocks Manager. For details, see Use a DLF catalog.

Property	Required	Description
`type`	Yes	Set to `paimon`.
`paimon.catalog.type`	Yes	Set to `rest`.
`uri`	Yes	The REST API address of DLF. Format: `http://<VPC endpoint>`. For example, `http://cn-hangzhou-vpc.dlf.aliyuncs.com`.
`paimon.catalog.warehouse`	Yes	The name of the Paimon catalog. Find it on the Catalogs page in the Data Lake Formation console.
`token.provider`	Yes	The REST service provider. Set to `dlf`.

StorageCredentialParams

StorageCredentialParams specifies how StarRocks accesses the Paimon file storage. This parameter group is only required when using OSS or OSS-HDFS as storage. Skip it if you use HDFS.

"aliyun.oss.endpoint" = "<YourAliyunOSSEndpoint>"

Property	Description
`aliyun.oss.endpoint`	The endpoint of your OSS or OSS-HDFS storage. Find it on the Overview page of your bucket, under the Port section. See OSS regions and endpoints for region-specific values. Examples: `oss-cn-hangzhou.aliyuncs.com` (OSS), `cn-hangzhou.oss-dls.aliyuncs.com` (OSS-HDFS).

Important

After setting aliyun.oss.endpoint, go to the Parameter Configuration page in the EMR Serverless StarRocks console and update fs.oss.endpoint in core-site.xml and jindosdk.cfg to the same value.

Example

CREATE EXTERNAL CATALOG paimon_catalog
PROPERTIES
(
    "type" = "paimon",
    "paimon.catalog.type" = "dlf",
    "paimon.catalog.warehouse" = "oss://<yourBucketName>/<yourPath>",
    "dlf.catalog.id" = "paimon_dlf_test"
);

For more information, see Paimon Catalog.

References

For more information about Paimon Catalogs, see Paimon Catalog.

Background information

Use DLF

Prerequisites

Example of using a DLF Catalog

Step 1: Prepare a user with an Alibaba Cloud identity in Serverless StarRocks

Step 2: Grant permissions to the catalog in DLF

Step 3: Create a DLF Catalog in Serverless StarRocks

Paimon Catalog

Iceberg Catalog

Use DLF 1.0 (Legacy)

Prerequisites

Create a catalog

Create a Hive Catalog

Syntax

Parameters

General parameters

Metastore parameters

Use HMS

Use DLF 1.0 (Legacy)

Example

Create an Iceberg Catalog

Syntax​

Parameters

Example

Create a Paimon Catalog

Syntax

Parameters

Common parameters

CatalogParams

Use HMS

Use filesystem

Use DLF 1.0 (Legacy)

Use DLF

StorageCredentialParams

Example

References

Syntax