Data Lake Formation:Access DLF from EMR on ECS Trino

Prerequisites

Before you begin, ensure that you have:

• An EMR cluster running version 5.13.0 or later with the Trino component selected

• (Optional) If you need support for a different version, join the DingTalk group (ID: 106575000021) to contact DLF developers

Create a catalog

See Set up DLF.

Grant DLF permissions to a role

Grant the required permissions to the AliyunECSInstanceForEMRRole role so that EMR nodes can access DLF. This involves two steps: granting a Resource Access Management (RAM) access policy and granting DLF catalog permissions.

You can skip step 1 after the EMR product integration with DLF is complete.

1. Grant RAM permissions to the AliyunECSInstanceForEMRRole role.

   1. Log on to the RAM console using your Alibaba Cloud account or as a RAM administrator.

   2. In the navigation pane, choose Identity Management > Roles. Search for AliyunECSInstanceForEMRRole.

   3. In the Actions column, click Add Permissions.

   4. In the Access Policy section, search for and select AliyunDLFFullAccess, then click Confirm to Add Permissions.

2. Grant DLF permissions to the AliyunECSInstanceForEMRRole role.

   1. Log on to the Data Lake Formation console.

   2. On the Catalogs page, click the name of a catalog to open its details page.

   3. Click the Permissions tab, then click Grant.

   4. On the Grant page, configure the following parameters and click OK.

      • Set User/Role to RAM User/RAM Role.

      • For Select Authorization Object, select AliyunECSInstanceForEMRRole from the drop-down list.

        Note: If AliyunECSInstanceForEMRRole is not in the drop-down list, go to the user management page and click Sync.

      • Set Preset Permission Type to Data Editor.

Upgrade Paimon dependencies for the EMR cluster

EMR is a managed cluster environment, so you must update the Paimon plugin JARs through a script action rather than replacing files directly on each node.

1. Import Paimon dependencies.

   1. Download the two required JAR files and upload them to Object Storage Service (OSS). Set the file permissions to public-read. For upload instructions, see Simple upload.

      • paimon-ali-jindo-1.4.jar

      • paimon-trino-422-1.4-plugin.jar

   2. Create the following upgrade script and upload it to OSS.

      #!/bin/bash
      
      echo 'backup paimon-trino-ali-.*'
      sudo mkdir /tmp/upgrade-trino-dlf2.5
      sudo cp -r /opt/apps/TRINO/trino-current/plugin/paimon /tmp/upgrade-trino-dlf2.5
      echo 'clear old paimon plugin jars'
      sudo rm -rf /opt/apps/TRINO/trino-current/plugin/paimon/*
      
      echo 'download paimon-ali-trino-1.4.jar'
      cd /opt/apps/TRINO/trino-current/plugin/paimon/
      wget ${paimon-ali-jindo-1.4.jar}
      wget ${paimon-trino-422-1.4-plugin.jar}

      Important

      Replace ${paimon-ali-jindo-1.4.jar} and ${paimon-trino-422-1.4-plugin.jar} with the OSS download paths of the corresponding files. For example: https://<bucket>.oss-cn-hangzhou-internal.aliyuncs.com/jars/paimon-ali-jindo-1.4.jar.

2. Run the script in the EMR cluster as a manual script action. For more information, see Manually run a script.

   1. In the EMR cluster, go to Script Operation > Manual Execution, then click Create and Execute.

   2. In the dialog box, set the following parameters and click OK.

      • Name: Enter a name for the script.

      • Script Location: Select the upgrade script that you uploaded to OSS. The path must follow the oss://**/*.sh format.

      • Execution Scope: Select Cluster.

Configure Trino to connect to the catalog

In the EMR cluster, go to Cluster Services, click Configure for the Trino service, and add or modify the following settings in the paimon.properties file.

connector.name=paimon
warehouse=dlf_test
metastore=rest
token.provider=dlf
uri=http://<region-id>-vpc.dlf.aliyuncs.com
dlf.token-loader=ecs

In the REST Catalog context, warehouse is the name of your DLF catalog instance, not a file system path. In traditional Hive or filesystem catalogs, warehouse points to a storage path such as hdfs:///path/to/warehouse. With DLF REST Catalog, it must be the catalog instance name — for example, dlf_test.

The following table describes each parameter.

After saving the configuration, restart the Trino service for the changes to take effect.

Read a Paimon table with Trino

1. Enter the Trino command-line interface (CLI).

   sh /opt/apps/TRINO/trino-current/bin/trino --server master-1-1:9090

2. Query a table from the DLF catalog. The following example reads from users_samples, an existing table in the catalog.

   select * from paimon.default.users_samples;