MaxCompute:Use of computing resources

Specify a default quota group for a project

Specify a quota group at the job level

You can add a SET command before the SQL statement of a job to specify a quota group for the job. This feature checks whether the job owner has the permissions to use the quota group. The following section describes the procedure.

1. Create a quota group.

   For more information about how to create a quota group, see Configure quota groups.

   The new MaxCompute console provides new quota management interaction. For more information about how to create a quota group in the new MaxCompute console, see Configure quotas.

   Note

   We recommend that you use a combination of letters and digits when you specify a quota group name. The existing Chinese quota group names can still be used.

2. Grant required permissions.

   By default, all Alibaba Cloud accounts, RAM users, and roles do not have permissions to specify a quota group for a job. An account can be used to specify a quota group for a job only after the account is granted the required permissions.

   1. Create a role.

      1. Log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, click Tenants.

      2. On the Tenant Management page, click the Roles tab.

      3. On the Roles tab, click Create Role.

      4. In the Create Role dialog box, configure the Role Name parameter and enter the following policy document:

         {
             "Statement": [{
                     "Action": [
                         "odps:List",
                         "odps:Usage"],
                     "Effect": "Allow",
                     "Resource": ["acs:odps:*:regions/*/quotas/*"]}],
             "Version": "1"
         }

         The following table describes the parameters.

         Parameter	Description
         Action	The action that is allowed or denied. You can specify multiple actions for a single authorization operation based on your business requirements. If you specify multiple actions, separate the actions with commas (,). For more information about the supported actions, see MaxCompute permissions. Note For more information about policy parameters, see Policy elements.
         Resource	The authorized resource range. The value is in the ["acs:odps:Tenant/${tenant_id}:regions/${region_id}/quotas/${quota_name}"] format. ["acs:odps:*:regions/*/quotas/*"] specifies all quota groups in all regions for the current tenant. You can specify a specific region and a specific quota group based on your business requirements.

      5. Click OK.

   2. Assign the role that you created to the account that you want to use to specify the quota group at the job level.

      Note

      By default, the Alibaba Cloud account or a RAM user that is assigned the tenant-level Super_Administrator role can perform the authorization operations.

      The authorization operations vary based on the type of the object to which permissions are granted.

      • Alibaba Cloud account

        To grant permissions to an Alibaba Cloud account, run the following commands:

        -- Add the Alibaba Cloud account to the current tenant and assign the specified role to the Alibaba Cloud account.
        Add tenant user <Aliyun$xxxx>;
        Grant tenant role <role_name> to user <Aliyun$xxxx>;
        -- View the permissions of the tenant-level role or user.
        Show grants for tenant role <role_name>;
        Show grants for tenant user <user_name>;
        Show principals for tenant [role] <role_name>;

      • RAM user

        1. In the MaxCompute console, click Users.

        2. Find the RAM user to which you want to grant permissions, and select the role that you created from the drop-down list in the Roles column to complete the authorization operation.

3. Specify the quota for a job.

   Add the following command before the SQL statement of the job to specify the previously created quota for the job:

   set odps.task.wlm.quota=<quotaname>; 

   quotaname is the name of the quota. This parameter is valid only for quotas of the batch processing type.

   Note

   If you want to specify a quota for your Spark job, you must add the configuration odps.task.wlm.quota = <quotaname>.

   For MaxCompute Query Acceleration (MCQA) jobs, take note of the following points:

   • You cannot use the preceding command to directly specify an interactive quota. An interactive quota can be automatically selected by the system only when MCQA is enabled. For more information, see MaxCompute Query Acceleration.

   • If you run the set odps.task.wlm.quota=<quotaname>; command to specify a quota for an MCQA job, the job is rolled back to run in the quota specified by quotaname when the job fails to run in the interactive quota. In this case, the quota specified by quotaname is not selected as a quota for the MCQA job. Therefore, the MCQA job still consumes the resources in the default computing quota associated with the project in which the MCQA job is run.

   • If you do not explicitly specify the odps.task.wlm.quota parameter, the interactive task attempts to roll back to the quota that is specified by quota rules. If no quota rule is configured, the task rolls back to the default quota of the project.

Quota rules

MaxCompute allows you to configure quota rules. This way, jobs that meet the rules can be scheduled to use the related quota.

Limits

• You can configure rules only for subscription level-2 quotas.

• You can configure a maximum of 10 rules for each level-2 quota.

• You can configure a maximum of 50 items in each of the Project and Job Owner fields for each rule. This indicates that you can configure a rule for a maximum of 50 projects and 50 job owners (UIDs).

• You can enter a maximum of five key-value pairs in the Job Settings field for each rule.

• Valid values of the Job Priority parameter for each rule: [0, 9].

Configure quota rules

1. Log on to the MaxCompute console. In the top navigation bar, select a region.

2. In the left-side navigation pane, choose Workspace > Quotas.

3. On the Quotas page, find the level-2 quota that you want to manage and click Rule Configuration in the Actions column.

4. In the Quota Rule Configuration dialog box, click Add Rule or find an existing rule and click Clone in the Actions column to create a rule.

   The following table describes the parameters.

   Parameter	Description
   Rule Name	The name of the custom quota rule. The name must start with a letter and can contain only letters, digits, and underscores (_).
   Rule Mode	The mode of the quota rule. Valid values: NORMAL: normal mode. In this mode, the following mechanism is provided to schedule a job to use a quota based on the characteristics of the job: If a job matches a rule of a quota, the job is scheduled to use the quota. If a job matches multiple quota rules at the same time, the job is scheduled to use the quota that is created earliest. If a job does not match any quota rule, the job uses the default quota of the project regardless of whether the job matches the rules of the default quota. If the etl_1 quota is specified for a job based on the job level, the job is scheduled to use the etl_1 quota regardless of whether the job matches the rules of the etl_1 quota. EXCLUSIVE: exclusive mode. If a job matches a rule in exclusive mode of a quota based on the characteristics of the job, the job is scheduled to use the quota. Otherwise, the job cannot be scheduled to use the quota. The following examples show the differences between the normal mode and the exclusive mode. For example, the etl_2 quota is the default quota that is associated with the Project_2 project, and rules that run in exclusive mode are configured for the etl_2 quota. If a job that is initiated by the Project_2 project cannot match the rules of the etl_2 quota, the default quota cannot be selected for the job. In this case, an error is reported when the job is submitted. However, if you configure only rules that run in common mode for the etl_2 quota, the job can be scheduled to use the etl_2 quota. For example, the set odps.task.wlm.quota=etl_3 (i.e. use quota etl_3) command is run to specify the etl_3 quota for a job. If a rule that runs in exclusive mode is configured for the etl_3 quota but the job does not match the rule, an error is reported when the job is submitted. However, if you configure only rules that run in common mode for the etl_3 quota, the job can be scheduled to use the etl_3 quota. ANTI: prohibited mode. If a job matches a rule in prohibited mode of a quota based on the characteristics of the job, the job is forbidden to use the quota. The job is preferentially scheduled to use the default quota group of the project to which the job belongs. If the default computing quota of the project is the quota that the job is forbidden to use, the job is scheduled to use the quota that is created earliest in the current region. The pay-as-you-go quota is included. Important The rules that run in exclusive mode and the rules that run in prohibited mode have higher priorities than the rules that run in normal mode. If the job does not match a rule that runs in exclusive mode of Quota A or matches a rule that runs in prohibited mode, the job cannot be scheduled to use Quota A regardless of whether the job matches a rule that runs in common mode of Quota A or is manually specified to use Quota A.
   Project	The projects that you want to configure in the rule. You can configure a maximum of 50 projects.
   Job Type	The types of jobs that you want to configure in the rule. Valid values: SQL: SQL jobs SQLRT: MaxCompute MCQA SQL jobs SQLCost: SQL cost estimation jobs LOT: MapReduce jobs CUPID: Spark or Mars jobs AlgoTask: Platform for AI (PAI) jobs
   Job Priority	The priorities of jobs that you want to configure in the rule. You can enter a value ranging from 0 to 9 to specify the priorities of jobs. For example, if you enter [0,3], the priorities of jobs are 0, 1, 2, and 3.
   Job Owner (UID)	The user IDs of the job owners that you want to configure in the rule. You can enter a maximum of 50 user IDs. Multiple user IDs have the OR relationship. Each user ID occupies a single line. If the p4_ prefix needs to be added to the user ID of a RAM user, the user ID is p4_12344566777. If the v4_ prefix needs to be added to the user ID of a RAM role, the user ID is v4_12344566776.
   Job Settings	You can enter one or more key-value pairs that are in the Key=Value format. For example, you can enter SKYNET_DAGTYPE=3 for a data backfill job that is initiated by an auto triggered node of DataWorks. If you enter multiple key-value pairs, the key-value pairs have the AND relationship. Each key-value pair occupies a single line. You can enter a maximum of five key-value pairs.

5. Click OK.

Examples

• Schedule an MCQA job

  • Acceleration quota and rollback quota

    An MCQA job requires an acceleration quota and a rollback quota. The acceleration quota is of the interactive type and the rollback quota is of the batch type. If a job that uses the acceleration quota times out or a job is rejected to use the acceleration quota due to reasons such as the job type, the system resubmits the job in the rollback quota. You can create a quota rule to add a project or a job owner to both an acceleration quota and a rollback quota to implement the preceding logic.

  • Enable MCQA by project or job owner

    To enable interactive quotas by project, you can configure different ProjectList rules for different quotas when you create interactive quotas.

• Isolate data backfill jobs

  A data backfill job is used to update data. If the data backfill process requires a long period of time, a large number of resources are consumed. To prevent the impact of excessive resource consumption on your daily jobs, you can create a quota and configure rules for the quota to exclusively run data backfill jobs. For example, if the data backfill job that is initiated by an auto triggered node of DataWorks fixedly passes the SKYNET_DAGTYPE:3 parameter to the Job Settings field, you can configure the following quota rules:

  • Create a quota named refill and configure rules for the refill quota. If you want to allow all data backfill jobs that are initiated by an auto triggered node of DataWorks to be automatically scheduled to use the refill quota, you can configure rules that run in normal mode. To configure a rule that runs in normal mode, you need to only enter SKYNET_DAGTYPE=3 in the Job Settings field.

  • You can also configure parameters in a fine-grained manner. For example, if you want only data backfill jobs whose priorities range from 5 to 9 of the P1 project to be scheduled to use the refill quota, perform the following steps to configure a rule that runs in normal mode: Set Project to P1, enter [5,9] in the Job Priority field, and then enter SKYNET_DAGTYPE=3 in the Job Settings field.

    Note

    The priority that is configured in the rule is the priority in MaxCompute but not the baseline priority in DataWorks. Priority in MaxCompute = 9 - Baseline priority in DataWorks.

• Isolate jobs that are initiated by different roles

  Jobs, such as data analytics jobs, scheduled report jobs, and data backfill jobs, may be initiated by different business personnel. The jobs may also come from different departments that have different resource budgets. Therefore, multiple quotas must be created to allow jobs that are initiated by different roles to use different quotas.

  In most cases, you can identify the roles of jobs based on the following parameters: Project, Job Owner (UID), Job Type, Job Priority, and Job Settings.

  • If jobs that are initiated by different roles use different projects, you can set the Project parameter to the names of the required projects in the rule that you want to create for the desired quota.

  • If jobs that are initiated by different roles are submitted by different users, you can enter the user IDs of the job owners in the Job Owner (UID) field in the rule that you want to create for the desired quota.

  • If jobs that are initiated by different roles have different configurations, you can configure Job Settings to distinguish between the roles of the jobs. Job Settings specifies the SET xxx=xxx; statements that are submitted together with the job. The SET statements are classified into custom statements and statements that are automatically added. For example, if a job is initiated by using DataWorks, specific parameters are automatically configured to help identify the job.

  • If different roles submit different job types, you can identify the job type based on the value of the Job Type parameter. For example, the value of the Job Type parameter for a PAI job that is submitted by using PAI is ALGO, and the value of the Job Type parameter for a normal SQL query job that is submitted by using MaxCompute is SQL.

  The preceding rules can be used together to perform finer-grained job scheduling.