Assign and schedule jobs using quota groups - MaxCompute

A quota group (quota) is a pool of computing resources in MaxCompute that provides the CPU and memory for your computing jobs. MaxCompute lets you assign jobs to a suitable quota based on their resource requirements, promoting efficient resource use.

Background

You can use MaxCompute quotas in the following ways:

Associate a project with a default computing quota. Jobs submitted in the project run in this quota.
Specify a computing quota at the job level (use quota). This lets you assign jobs from a single project to different quotas, including both subscription and pay-as-you-go quotas. A quota specified at the job level takes precedence over the project's default quota.
Configure rules for a quota. This schedules jobs that meet the rule's conditions to that quota.

Set a default quota for a project

You can set a default computing quota for a project. Jobs initiated in this project automatically use the default quota for computation. You can associate a project with a default quota when you create the project. You can also change the default quota for an existing project in the console. For more information, see Manage projects.

Specify a quota at the job level

You can specify a quota at the job level by adding a command before the job query. This feature checks whether the job owner has the permission to use the specified quota.

Create a quota
For more information about how to create a quota in the new console, see Configure quotas.
Note
We recommend that you use a combination of letters and numbers for the quota name. Existing Chinese quota names can still be used.
Grant permissions
By default, no accounts (including Alibaba Cloud accounts) or roles have the permission to specify a quota at the job level. You must grant this permission explicitly.
1. Create a role
  1. Log on to the MaxCompute console, and select a region in the upper-left corner.
  2. In the navigation pane on the left, choose Manage Configurations > Tenants.
  3. On the Tenants page, click the Roles tab.
  4. On the Roles tab, click Add Role. In the Add Role dialog box, enter a Role Name and Policy Content, and then click OK to create the role.
```
{
    "Statement": [{
            "Action": [
                "odps:List",
                "odps:Usage"],
            "Effect": "Allow",
            "Resource": ["acs:odps:*:regions/*/quotas/*"]}],
    "Version": "1"
}
```
2. Assign the role to the account that needs to specify a quota at the job level
  Your Alibaba Cloud account or a RAM user with the Super_Administrator permission can grant these permissions.
  The authorization process varies depending on the account type.
  Authorizing Alibaba Cloud accounts
  Run the following commands to authorize an Alibaba Cloud account:
  -- Add the Alibaba Cloud account to the tenant and grant a tenant role to the account. ADD TENANT USER <Aliyun$xxxx>; GRANT TENANT ROLE <role_name> TO USER <Aliyun$xxxx>; -- View the permissions of a tenant role or user. SHOW GRANTS FOR TENANT ROLE <role_name>; SHOW GRANTS FOR TENANT USER <user_name>; SHOW PRINCIPALS FOR TENANT [ROLE] <role_name>;
  Authorizing RAM users
  Follow these steps to authorize a RAM user:
  Log on to the MaxCompute console, and select a region in the upper-left corner.
  In the navigation pane on the left, choose Manage Configurations > Tenants.
  On the Tenants page, click the Users tab.
  On the Users tab, click Modify Role in the Actions column for the target Resource Access Management (RAM) user.
  In the Edit Role dialog box, select the roles to assign to the user from the Available Roles area, move them to the Added Roles area, and then click OK.
Specify the quota for a job
Add the following command before the job query to specify the quota:
```
SET odps.task.wlm.quota = <quota_nick_name>; 
```
The quota_nick_name parameter specifies the nickname of the quota. This setting applies only to quotas for batch processing.
- For a Spark job, you must add the spark.hadoop.odps.task.wlm.quota configuration and set its value to <quota_nick_name>.
- For a MaxCompute Query Acceleration (MCQA) job, note the following:
  - You cannot directly specify an interactive quota by using the preceding command. The system automatically selects an interactive quota only when you enable the query acceleration mode. For more information, see MaxCompute Query Acceleration (MCQA).
  - The quota specified by the set odps.task.wlm.quota=<quota_nick_name>; command is a fallback quota. This means that if a task fails to run in the interactive quota, it falls back to the quota_nick_name quota for execution, rather than the acceleration quota selecting quota_nick_name. Therefore, the resources consumed by successfully accelerated jobs are still charged to the default computing quota bound to the project where the job is executed.
  - If you do not explicitly specify the odps.task.wlm.quota parameter, an interactive job first attempts to fall back to the quota specified by a quota rule. If no quota rule is configured, the job falls back to the project's default quota.

Quota rules

MaxCompute allows you to configure rules for a quota. Jobs that meet the rule conditions can be scheduled to run in the corresponding quota.

Limitations

You can configure rules for both subscription and pay-as-you-go (Standard and Idle) level-2 quotas.
You can configure a maximum of 10 rules for each level-2 quota.
For each rule, you can enter up to 50 objects for the Project and Job Owner parameters. This means one rule can apply to a maximum of 50 projects and 50 job owners (UIDs).
For each rule, you can enter a maximum of five key-value pairs for Job Settings.
For each rule, the value range for job priority is [0, 9].

Configure quota rules

All configured quota rules are active. You can delete unneeded rules. Multiple rules have a logical OR relationship. If a job matches any rule of a quota, it is scheduled to or prohibited from that quota, depending on the rule's mode.

Note

Use them conservatively to avoid creating too many rules, which can complicate future maintenance.

Log on to the MaxCompute console, and select a region in the upper-left corner.
In the navigation pane on the left, choose Manage Configurations > Quotas.
On the Quotas page, click the icon to the left of a level-1 quota to view the list of its level-2 quotas.
Find the level-2 quota you want and click Rule Configuration in the Actions column.

In the Quota Rule Configuration dialog box, click Add Rule, or click Clone in the Actions column of an existing rule to create a new rule. Configure the parameters as described below.

Within a single rule, conditions for multiple projects, job types, and Job Settings have a logical AND relationship.
A rule takes effect approximately 5 minutes after you add it.

Click to expand parameter details

Parameter	Description
Rule Name	The custom name of the quota rule. The name must start with a letter and can contain only letters, digits, and underscores (_).
Rule Mode	The mode in which the quota rule runs. Valid values: NORMAL: normal mode. If a job's characteristics match a quota rule, the job is scheduled to run in that quota. When a job matches a quota rule, it is scheduled to run in that quota. When a job matches multiple quota rules simultaneously, it is scheduled to run in the quota that was created earliest. When a job does not match any quota rule, it uses the project's default quota, even if the job does not match the rules of the default quota. When a quota, such as `etl_1`, is specified at the job level, the job will be scheduled to the `etl_1` quota, even if the job does not match the rules of the `etl_1` quota. EXCLUSIVE: exclusive mode. A job is scheduled to this quota only if its characteristics match this rule. Otherwise, the job is prohibited from using this quota. The following examples illustrate the difference from the normal mode: For example, if a project's default quota, `etl_2`, is associated with `Project_2`, and `etl_2` has an exclusive mode rule, a job from that project must match the rule to use `etl_2`. If it does not match, the submission fails. In normal mode, the job could use `etl_2` and be scheduled to run in `etl_2` even without a match. If you specify a quota named `etl_3` by using the command `set odps.task.wlm.quota=etl_3 (i.e. use quota etl_3)`, and `etl_3` has an exclusive mode rule that the job does not match, the job submission will fail. In contrast, if `etl_3` only had a normal mode rule, the job could be scheduled to run in `etl_3`. ANTI: prohibited mode. If a job matches this rule, it is prohibited from using this quota. The job is then routed to its project's default computing quota. If the default quota is this same prohibited quota, the job is instead scheduled to the oldest available quota in the region, which can include pay-as-you-go quotas. Important Exclusive mode and prohibited mode rules have a higher priority than normal mode rules. If a job does not match an exclusive mode rule of Quota A or matches a prohibited mode rule of Quota A, the job cannot be scheduled to Quota A, regardless of whether it matches a normal mode rule of Quota A or is manually assigned to Quota A.
Project	Select the projects to which the rule applies. You can select up to 50 projects. If you leave this empty, the rule applies to all projects.
Job Type	Select the job types to which the rule applies. Valid values: SQL: SQL jobs SQLRT: MaxCompute Query Acceleration SQL jobs SQLCost: SQL cost estimation jobs LOT: MapReduce jobs CUPID: Spark or Mars jobs AlgoTask: machine learning jobs MaxFrame: MaxFrame distributed computing engine jobs Graph: Graph computing jobs This parameter is empty by default, which means the rule applies to all job types.
Job Priority	Enter the job priority to which the rule applies. Enter a priority range from `0-9`. For example, `[0,3]` matches jobs with priorities 0, 1, 2, and 3. If you leave this empty, the rule applies to all job priorities.
Job Owner (Account ID)	Enter the UIDs of the job owners to which the rule applies. Multiple UIDs have a logical OR relationship. Enter each UID on a new line. You can enter up to 50 UIDs. For a RAM user UID, you must add the `p4_` prefix, for example, `p4_12344566777`. If you leave this empty, the rule applies to all job owners.
Job Settings	Enter key-value pairs from the Job Settings in the format `Key=Value`. Do not add quotes. Multiple parameters have a logical AND relationship. Enter each pair on a new line. You can enter up to five pairs. For example, to run all data backfill jobs initiated by DataWorks scheduling in the current quota, enter `SKYNET_DAGTYPE=3` in the Job Settings. For more information, see Isolate data backfill jobs. If you leave this empty, the rule applies to all settings.

At least one of the following parameters must be specified: Project, Job Type, Job Priority, Job Owner (UID), or Job Settings. Any parameters left empty will match all options.

Use cases

MCQA job scheduling

Schedule MaxCompute Query Acceleration jobs

Acceleration quota and fallback quota.
An MCQA job requires two quotas:
- An acceleration quota (of the interactive type)
- A fallback quota (of the batch processing type).
To implement this, configure a quota rule that associates a project or job owner with both an acceleration and a fallback quota.
Enable MCQA by project, job owner, or other criteria.
When you create a quota of the interactive type, you can configure different ProjectList rule conditions for different quotas. This allows you to enable the interactive quota on a per-project basis.

Data backfill job isolation

Isolate data backfill jobs

A data backfill job, which re-runs jobs to repopulate historical data, can consume a large amount of computing resources, especially over a long time span. To avoid affecting the normal operation of daily jobs, we recommend creating a dedicated quota with specific rules for data backfill jobs.

For example, a DataWorks scheduled task consistently passes the parameter SKYNET_DAGTYPE:3 into the Job Settings when initiating a data backfill job. You can use this behavior to configure the following quota rule:

Create a quota named refill and configure a rule for the refill quota. If you want all data backfill jobs initiated by DataWorks to run in the refill quota by default, configure a normal mode rule. To do this, enter SKYNET_DAGTYPE=3 in the Job Settings.
You can also perform more granular configurations. For example, you may want only data backfill jobs from the P1 project with a job priority between 5 and 9 to run in the refill quota. To do so, configure a normal mode rule, select P1 for the project, enter [5,9] for the priority, and enter SKYNET_DAGTYPE=3 for Job Settings.
Note
The priority here is the MaxCompute job priority, not the DataWorks baseline priority. The conversion formula is: MaxCompute Priority = 9 - DataWorks Baseline Priority.

Job isolation by role

Isolate jobs from different roles

You can create multiple quotas to isolate jobs from different roles.

Typically, you can distinguish job roles using a combination of the following dimensions: Project, Job Owner, Job Type, Priority, and Job Settings.

When jobs from different roles use different projects:
You can add the project name to the Project field in the rule of the desired quota.
When jobs from different roles are submitted by different users:
You can add the job owner's UID to the Job Owner field in the rule of the desired quota.
When jobs from different roles have different configurations:
You can use Job Settings to distinguish roles. Job Settings refers to the set xxx=xxx; statements submitted along with the job. They can be custom-defined or added by the system. For example, jobs from DataWorks automatically include identifying parameters.
When different roles submit different job types:
You can use Job Type to distinguish them. For example, a machine learning job submitted from the PAI platform has the Job Type ALGO. A standard SQL query submitted through MaxCompute has the Job Type SQL.

You can also combine these rule conditions to achieve more fine-grained control over job scheduling.