MaxCompute:Purchase and use exclusive data transmission resource groups

Background

Prerequisites

Before purchasing and using an exclusive data transmission service resource group, ensure your environment meets the following network and region requirements:

• Network requirements: Only access from a VPC network is supported. Access from the public internet is not supported.

• Region requirements: Cross-region data transfers are not supported. You must purchase the resource group in the same region as your data transfer task.

• For other limitations, see Data Transmission Service limitations.

Purchase an exclusive data transmission resource group

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Quotas .

3. On the Quotas page, click New Quota.

   On the purchase page, select Data Transmission Service as the specification type.

   Configure the Tunnel Concurrency based on your requirements.

4. After you complete the purchase, return to the console. On the Quotas page, you will find the new quota group with the Quota Type of Data Transmission Service. You can upgrade, downgrade, or renew this quota group. For more information, see Related operations.

Use an exclusive data transmission resource group

You can specify an exclusive data transmission service resource group for data transfer or integration jobs to improve processing efficiency and meet specific business needs.

1. Connect to your MaxCompute project by using the MaxCompute client (odpscmd) or another connection tool, and then run the following command to enable exclusive quota routing:

    setproject odps.tunnel.enable.quota.route.v2=true;

   Note

   Ensure that your MaxCompute SDK is version 0.45 or later.

2. Grant permissions to use the quota feature to all required accounts (including Alibaba Cloud accounts) and roles.

   1. Log in to the MaxCompute console and select a region in the upper-left corner.

   2. In the left-side navigation pane, choose Manage Configurations > Tenants .

   3. On the Tenants page, click the Roles tab.

   4. On the Roles tab, click Add Role. In the Add Role dialog box, enter a custom Role Name, provide the Policy Content, and then click OK.

      The Policy Content is as follows:

      {
          "Statement": [{
                  "Action": [
                      "odps:List",
                      "odps:Usage"],
                  "Effect": "Allow",
                  "Resource": ["acs:odps:*:regions/*/quotas/*"]}],
          "Version": "1"
      }

   5. Assign the role to the accounts that need to specify quotas at the job level.

      1. Grant permissions to an Alibaba Cloud account.

         -- Add an Alibaba Cloud account to the tenant and grant a role to the account.
         ADD tenant USER <Aliyun$xxxx>;
         GRANT tenant role <role_name> TO USER <Aliyun$xxxx>;
         
         -- View the permissions of a role or user in the tenant.
         SHOW grants FOR tenant role <role_name>;
         SHOW grants FOR tenant USER <user_name>;
         SHOW principals FOR tenant [role] <role_name>;

      2. Grant permissions to a RAM user.

         On the Tenants page, click the Users tab.

         On the Users tab, click Add Member. In the Add Member dialog box, select or add the desired accounts. In the Batch Set Roles section, select the roles, click , and then click OK.

3. Manually specify the quota group in the SDK for your Data Transmission Service task.

   -- Use the resource group named xxx for the Data Transmission Service.
   final TableTunnel tunnel = new TableTunnel(odps);
   tunnel.getConfig().setQuotaName("xxx");
   TableTunnel.UploadSession up = tunnel.createUploadSession(projectName, tableName);
   
   -- Get the name of the resource group that the session is actually using.
   up.getQuotaName();

   In the left-side navigation pane, choose Manage Configurations > Quotas .

   In the left-side navigation pane, choose Manage Configurations > Quotas .

   You can view and copy the QuotaName. For example: ot_42854300324****_169821756****_p#ot_42854300324****_169821756****.

View resource group metrics

On the Quotas page, click Configure Resources in the Actions column of the target quota.

You can view the usage of your subscription-based exclusive data transmission service resource group.

Resource group configuration

1. Log in to the MaxCompute console and select a region in the upper-left corner.

2. In the left-side navigation pane, choose Manage Configurations > Projects.

3. On the Projects page, click Manage in the Actions column for the target project.

   In the Basic Properties section, you can view the Data Transmission Service configuration for the current project. The following table describes the parameters.

   Parameter	Description
   Default Tunnel Quota	The default Data Transmission Service resource group quota to use when no specific quota is specified for reading data from or writing data to the project. This is typically Default, which represents the shared Data Transmission Service resource group. You cannot modify this setting in the console.
   Authorized Tunnel Quota	The project authorizes all users and roles to use the configured exclusive resource group for data read and write jobs. Therefore, you can specify the configured exclusive resource group quota in an SDK for data read and write jobs in the project, even without granting manual authorization. You can configure only one exclusive resource group for a project.
   Overlay Tunnel Quota	This enhanced feature for exclusive resource groups allows you to use both an exclusive resource group and the default resource group. After you configure a purchased exclusive resource group, the maximum number of concurrent jobs allowed for the current project is increased to the combined resources of the shared Data Transmission Service resource group and the exclusive resource group. This feature is supported only in some regions. Availability is subject to what is displayed in the console.

   • A project can be configured with only one exclusive resource group, but multiple projects can share one. When you use the overlay feature, leave the quota group setting empty.

   • If you use the Storage API, you must specify the exclusive resource group. The QuotaName format is ot_42854300324****_169821756****_p#ot_42854300324****_169821756****.

4. In the Basic Properties section, click Edit to modify the configuration as needed.