MaxCompute:Purchase and use an exclusive resource group for Data Transmission Service

Background

Usage notes

Before you purchase and use an exclusive resource group for Data Transmission Service, ensure that your network and region meet the following requirements:

• Network limits: Only access over a VPC network is supported. Access over the public internet is not supported.

• Region limits: Only data transmission within the same region is supported. Cross-region data transmission is not supported. You must purchase a resource group in the same region where your data transmission task runs.

• For other limits, see Limits on Data Transmission Service.

Purchase an exclusive resource group for Data Transmission Service

1. Log on to the MaxCompute console and select a region in the top-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Quotas.

3. On the Quotas page, click New Quota.

   On the Purchase page, for Specification Type, select Tunnel.

   Configure the Tunnel Concurrency parameter as required.

4. After you complete the purchase, return to the console. On the Quotas page, a quota group with the Quota Type set to Data Transmission Service is displayed. You can upgrade, downgrade, or renew this quota group. For more information, see Upgrade and downgrade.

Use an exclusive resource group for Data Transmission Service

When you run data transmission or data integration jobs, you can specify an exclusive resource group for Data Transmission Service based on your business needs to improve data processing efficiency.

1. Access your MaxCompute project using the MaxCompute client (odpscmd) or another connection tool, and then enable the exclusive resource group for Data Transmission Service.

    setproject odps.tunnel.enable.quota.route.v2=true;

   Note

   Ensure that your MaxCompute software development kit (SDK) is version 0.45 or later.

2. Grant the permissions to use the quota feature to all required accounts and roles, including Alibaba Cloud accounts.

   1. Log on to the MaxCompute console and select a region in the top-left corner.

   2. In the navigation pane on the left, choose Manage Configurations > Tenants .

   3. In the navigation pane on the left, choose Manage Configurations > Tenants .

   4. On the Roles tab, click Add Role. In the Add Role dialog box, enter a custom Role Name and Policy Content, and then click OK.

      Configure the Policy Content as follows:

      {
          "Statement": [{
                  "Action": [
                      "odps:List",
                      "odps:Usage"],
                  "Effect": "Allow",
                  "Resource": ["acs:odps:*:regions/*/quotas/*"]}],
          "Version": "1"
      }

   5. Grant the role to the accounts that need to specify quotas at the job level.

      1. Grant permissions to an Alibaba Cloud account.

         -- Add an Alibaba Cloud account to the tenant and grant a role to the account.
         ADD tenant USER <Aliyun$xxxx>;
         GRANT tenant role <role_name> TO USER <Aliyun$xxxx>;
         
         -- View the permissions of a role or user in the tenant.
         SHOW grants FOR tenant role <role_name>;
         SHOW grants FOR tenant USER <user_name>;
         SHOW principals FOR tenant [role] <role_name>;

      2. Grant permissions to a Resource Access Management (RAM) user.

         On the Tenants page, click the Users tab.

         On the Users tab, click Add Member. In the Add Member dialog box, select or manually add the accounts for which you want to set roles. In the Batch Set Roles area, select the roles to assign to these accounts, click , and then click OK.

3. Manually specify the quota group in the SDK for the Data Transmission Service task.

   -- Use the resource group named XXX for Data Transmission Service.
   final TableTunnel tunnel = new TableTunnel(odps);
   tunnel.getConfig().setQuotaName("xxx");
   TableTunnel.UploadSession up = tunnel.createUploadSession(projectName, tableName);
   
   -- You can obtain the resource group that is actually used by the session.
   up.getQuotaName();

   In the navigation pane on the left, choose Manage Configurations > Quotas.

   In the navigation pane on the left, choose Manage Configurations > Quotas.

   You can view and copy the QuotaName. For example: ot_42854300324****_169821756****_p#ot_42854300324****_169821756****.

View resource group metrics

On the Quotas page, find the target quota and click Configure Resources in the Actions column.

You can view the usage of the current subscription-based exclusive resource group for Data Transmission Service.

Resource group configuration

1. Log on to the MaxCompute console and select a region in the top-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Projects.

3. On the Projects page, find the target project and click Manage in the Actions column.

   The Basic Properties section displays the Data Transmission Service configuration for the current project. The parameters are described as follows:

   Parameter	Description
   Default Tunnel Quota	The default Data Transmission Service resource group that the project uses for read and write operations when no specific resource group quota is designated. The value is usually Default, which indicates the public Data Transmission Service resource group. You cannot change this parameter in the console.
   Default Tunnel Quota	Authorizes all users and roles to use the configured dedicated resource group to perform data read and write tasks for the project. Therefore, you can specify the configured dedicated resource group quota in the SDK for data read and write tasks without manual authorization. A project can be configured with only one dedicated resource group.
   Overlay Tunnel Quota	An enhanced feature for dedicated Data Transmission Service resource groups that lets you use a dedicated resource group and the Default resource group together. After you configure a purchased dedicated resource group, the maximum concurrency for the project increases to the sum of the resources from the public Data Transmission Service resource group and the dedicated resource group. This feature is available only in some regions. Availability is subject to what is displayed on the console.

   • A project can be associated with only one exclusive resource group, but multiple projects can use the same exclusive resource group. When you use the overlay feature, you must specify Default for the quota group.

   • To use the open storage API operations, you must still specify an exclusive resource group. The QuotaName format is ot_42854300324****_169821756****_p#ot_42854300324****_169821756***.

4. In the Basic Properties section, click Edit to modify the configurations.