Data Lake Formation:API usage guide

Choose an access endpoint

The DLF metadata API is available over two endpoints. Choose an endpoint based on where your client runs:

• DLF VPC endpoint (Recommended) — Provides low-latency access from a VPC that is in the DLF whitelist. The examples in this topic use a DLF VPC endpoint.

• Alibaba Cloud OpenAPI endpoint — Provides public network access, which serves scenarios such as cross-region access and local debugging. Signature support for the OpenAPI endpoint requires a Paimon client of version 1.4 or later, which is later than the SDK version used in the Maven dependency in this topic.

Prerequisites

Before you add a VPC to the whitelist, grant permissions to a role, add the Maven dependency, and call the DLF metadata API, make sure that the following requirements are met:

• Network access — By default, the DLF metadata API can be accessed only from VPCs in the whitelist. When you access DLF over a DLF VPC endpoint, your client must run in a whitelisted VPC.

• Runtime environment and role — When you use Elastic Compute Service (ECS) role authentication, your client runs on an ECS instance or an E-MapReduce (EMR) cluster node that is bound to an ECS role.

• DLF catalog — A DLF catalog is available, and the ECS role is granted permissions on that catalog.

• Authorization credentials — You have an Alibaba Cloud account or Resource Access Management (RAM) administrator credentials, which are required to grant RAM permissions to the ECS role.

• Client version — The Paimon client meets the version requirement of the endpoint that you use, as described in the preceding section.

Configure a VPC whitelist

When you activate DLF, the system automatically synchronizes the VPC IDs in your current region to the user-level whitelist. To add a VPC later, add its VPC ID in the DLF console:

1. Log on to the Data Lake Formation (DLF) console.

2. In the left-side navigation pane, click System & Security.

3. Click the System & Security tab, and then click Add VPC.

4. In the dialog box that appears, enter the ID of the VPC from which you access the DLF metadata API, and then click OK.

Grant DLF permissions to a role

Grant the ECS role two sets of permissions: RAM permissions in the RAM console, and catalog permissions in the DLF console. For example, for an EMR cluster, the role is AliyunECSInstanceForEMRRole.

1. Grant RAM permissions to the ECS role.

   1. Log on to the Resource Access Management (RAM) console with your Alibaba Cloud account or as a RAM administrator.

   2. Choose Identities > Roles, and then search for the ECS role.

   3. In the Actions column, click Add Permissions.

   4. In Policies, search for and select AliyunDLFFullAccess, and then click OK.

2. Grant DLF permissions to the ECS role.

   1. Log on to the Data Lake Formation (DLF) console.

   2. On the Catalogs page, click the name of the target catalog.

   3. Click the Permissions tab, and then click Grant Permissions.

   4. On the authorization page, configure the following parameters, and then click OK.

      • Principal: Select a RAM user or RAM role.

      • Select DLF User: Select ECS Role from the drop-down list.

        Note

        If ECS Role does not appear in the user drop-down list, choose System & Security > Access Control > Users, and then click Sync Users.

      • Predefined Permission Type: Select Data Editor.

Create a REST API client

Create a REST API client to call the DLF metadata API from your Java application.

<dependency>
  <groupId>org.apache.paimon</groupId>
  <artifactId>paimon-api</artifactId>
  <version>1.3.0</version>
</dependency>

import org.apache.paimon.options.Options;
import org.apache.paimon.rest.RESTApi;
import static org.apache.paimon.options.CatalogOptions.WAREHOUSE;
import static org.apache.paimon.rest.RESTCatalogOptions.DLF_TOKEN_LOADER;
import static org.apache.paimon.rest.RESTCatalogOptions.TOKEN_PROVIDER;
import static org.apache.paimon.rest.RESTCatalogOptions.URI;
public class RESTApiExample {
    public static void main(String[] args) {
        Options options = new Options();
        options.set(URI, "http://cn-hangzhou-vpc.dlf.aliyuncs.com");
        options.set(WAREHOUSE, "dlf_test");
        options.set(TOKEN_PROVIDER, "dlf");
        options.set(DLF_TOKEN_LOADER, "ecs");
        RESTApi api = new RESTApi(options);
        System.out.println(api.listTables("my_database"));
    }
}