Data Lake Formation:Learn how compute engines access Data Lake Formation using fine-grained access control

Step 1: Grant DLF permissions to Flink service role

1. Log on to the RAM console with your Alibaba Cloud account or as a RAM administrator.

2. In the navigation pane on the left, choose Identities > > Roles, and then search for AliyunStreamAsiDefaultRole.

3. In the Actions column, click Add Permissions. Search for and select AliyunDLFFullAccess, then click Confirm Add Permissions.

Step 2: Grant data permissions in DLF

1. Log on to the Data Lake Formation console.

2. Go to the target catalog, switch to the Permissions tab, and click Grant.

3. Select RAM User/RAM Role, and select AliyunStreamAsiDefaultRole from the drop-down list.

4. Grant the required data permissions, such as Data Reader, Data Editor, or custom permissions, based on your business needs.

Step 3: Add Flink VPC to DLF trusted list

1. In the navigation pane on the left of the DLF console, click System and Security > > System Security > > Add VPC ID.

2. In the dialog box, select one or more VPCs to add to the trusted list, and then click OK.

For more information, see Configure a trusted VPC.

Step 4: Create a DLF catalog in Flink

Method 1: Create a catalog using the console

1. Log on to the Realtime Compute for Apache Flink console and go to the target workspace.

2. In the navigation pane on the left, click Data Management.

3. On the Catalogs page, click Create Catalog and select the catalog type.

Method 2: Create a catalog using SQL

In the text editor on the Data Exploration page, enter and run the following SQL statement:

CREATE CATALOG `flink_catalog_name`
WITH (
  'type' = 'paimon',
  'metastore' = 'rest',
  'token.provider' = 'dlf',
  'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com',
  'warehouse' = 'dlf_test'
);

CREATE CATALOG `flink_catalog_name`
WITH (
  'type' = 'iceberg',
  'catalog-type' = 'rest',
  'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg',
  'warehouse' = 'iceberg_test',
  'rest.signing-region' = 'cn-hangzhou',
  'io-impl' = 'org.apache.iceberg.rest.DlfFileIO'
);

Step 1: Grant console permissions

1. Log on to the RAM console with your Alibaba Cloud account or as a RAM administrator.

2. Create a new RAM user or select an existing one.

   Note: The name of the RAM user must comply with StarRocks user naming conventions. It can contain only letters, digits, and underscores. Hyphens and periods are not allowed.

3. Attach the following permission policies to the RAM user:

   • EMR console permissions: Attach AliyunEMRStarRocksFullAccess (administrator) or AliyunEMRStarRocksReadOnlyAccess (read-only) to access the EMR StarRocks console.

   • DLF API permissions: Attach AliyunDLFFullAccess or a custom DLF permission policy to call DLF APIs.

For more information, see Grant permissions to a RAM user for EMR Serverless StarRocks.

Step 2: Grant data permissions in DLF

1. Log on to the Data Lake Formation console.

2. Go to the target catalog, switch to the Permissions tab, and click Grant Permissions.

3. Select the RAM user from the previous step and grant the required data permissions, such as Data Reader, Data Editor, or a custom permission.

Step 3: Create a mapped StarRocks user

1. Log on to the EMR Serverless StarRocks console and connect to your StarRocks instance.

2. In security center > user management, click Add User:

   • User Source: Select RAM User.

   • Username: Select the RAM user that you granted permissions to in Step 1.

   • Password: Create an 8- to 30-character password that contains uppercase and lowercase letters, digits, and special characters.

   • Role: Keep the default public role.

3. Grant the user the USAGE permission on the External Catalog. On the User Management page, click Authorize in the Actions column of the target user. In the Add Permission panel, select External Catalog and grant the USAGE permission.

For more information, see Manage users in Serverless StarRocks.

Step 4: Create an external catalog

Log on to StarRocks Manager as the RAM user and run the following SQL statement to create an external catalog:

CREATE EXTERNAL CATALOG `paimon_catalog`
PROPERTIES (
  'type' = 'paimon',
  'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com',
  'paimon.catalog.type' = 'rest',
  'paimon.catalog.warehouse' = 'my_catalog',
  'token.provider' = 'dlf'
 );

CREATE EXTERNAL CATALOG iceberg_catalog
PROPERTIES ( 
    'type' = 'iceberg',
    'uri' = 'http://cn-hangzhou-vpc.dlf.aliyuncs.com/iceberg',
    'iceberg.catalog.type' = 'dlf_rest',
    'warehouse' = 'iceberg_test',
    'rest.signing-region' = 'cn-hangzhou'
);

Step 1: Grant console permissions to a RAM user

1. Log on to the RAM console by using your Alibaba Cloud account or as a RAM administrator.

2. Attach the following permission policies to the RAM user:

   • EMR console permissions: Attach AliyunEMRServerlessSparkFullAccess (for administrators), AliyunEMRServerlessSparkDeveloperAccess (for developers), or AliyunEMRServerlessSparkReadOnlyAccess (for read-only access).

   • DLF API permissions: Attach AliyunDLFFullAccess or a custom DLF permission policy.

For more information, see RAM User Authorization.

Step 2: Grant data permissions in DLF

1. Log on to the Data Lake Formation console.

2. Go to the target catalog and grant the required data permissions, such as Data Reader, Data Editor, or custom permissions, to the RAM user or the RAM role associated with the workspace.

Step 3: Bind a DLF catalog to a Spark workspace

Method 1: Bind during workspace creation

When you create a Serverless Spark workspace, enable Use DLF as metadata service and select the target DLF catalog.

Method 2: Bind to an existing workspace

Go to the Data Catalog page of your Serverless Spark workspace and add the target DLF catalog. For more information, see Manage Data Catalogs.

After you bind a DLF catalog, both Livy Gateway and Kyuubi Gateway natively support it as the default data catalog.

How it works

1. Token generation: A time-limited token is generated in Kyuubi Gateway for a specific RAM user and bound to their identity.

2. Client authentication: A client, such as Beeline, includes the token and RAM username in the JDBC connection string to connect to Kyuubi Gateway.

3. Identity proxy: After Kyuubi Gateway validates the token, the Spark engine impersonates that RAM user.

4. DLF authorization: The Spark engine sends requests to DLF as the impersonated RAM user. DLF then authorizes the request based on that user's permission policies.

Configuration steps

1. In DLF, grant the corresponding table-level or column-level data permissions to each RAM user who requires independent access.

2. On the Token Management page of Kyuubi Gateway, create a token for each RAM user.

3. From your client, use the corresponding token to connect to Kyuubi Gateway:

beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;user=<RAM-username>;httpPath=cliservice/token/<Token>"

Verify permission isolation

• Query an authorized table: SELECT * FROM db.authorized_table LIMIT 10; — Data is returned successfully.

• Query an unauthorized table: SELECT * FROM db.unauthorized_table LIMIT 10; — A permission error is returned, such as emr_test doesn't have privilege SELECT on TABLE.

For more information, see Permission control for DLF data using Kyuubi tokens.

Steps 1 to 3

Follow the procedure from the previous scenario to configure RAM permissions, authorize DLF data access, and bind the catalog.

Step 4: Access files through PVFS

In a notebook or PySpark, use the pvfs:// path to access DLF-managed files:

df = spark.read.option("delimiter", ",").option("header", True) \
    .csv("pvfs://<catalog_name>/default/object_table/employee.csv")
df.show(5)

Step 1: Grant RAM permissions to the ECS role

1. Log on to the RAM console.

2. Search for AliyunECSInstanceForEMRRole.

3. Attach the AliyunDLFFullAccess permission policy to the role.

Step 2: Grant DLF data permissions

1. Log on to the Data Lake Formation console.

2. On the target catalog's Permissions tab, click Authorize.

3. Select RAM User/RAM Role, and then select AliyunECSInstanceForEMRRole from the drop-down list.

4. Select a predefined permission type (Data Reader or Data Editor) or Custom Permissions.

If AliyunECSInstanceForEMRRole does not appear in the drop-down list, click Sync on the User Management page.

Step 3: Upgrade dependencies

Upgrade the Paimon dependency to version 1.1 or later. See the following documents for download and deployment instructions:

• Access DLF from EMR on ECS Spark by using Paimon REST

• Access DLF from EMR on ECS Spark by using Iceberg REST