PAI supports multi-level quotas to divide computing resources hierarchically. You can enable preemption, allowing queued jobs in a specific quota to preempt idle computing resources from sibling or child quotas to improve resource utilization.
Quota hierarchy
PAI lets you organize computing resources into a tree structure called a QuotaTree, dividing them into parent and child quotas to match your organization's structure. This enables fine-grained resource management and allocation.
As shown in the preceding figure, the computing resources are divided into three levels:
Root | Quotais the parent quota.Quota-1,Quota-2, ...,Quota-nare its child quotas and are siblings to each other.Quota-1is the parent quota ofQuota-1.1andQuota-1.2.Quota-1.1andQuota-1.2are sibling quotas and are child quotas ofQuota-1. The same structure applies toQuota-2throughQuota-n.
You can divide and use quotas based on your organizations or projects to manage their respective jobs and resources. When resource contention occurs, you can create multi-level quotas and enable a preemption policy to maximize resource utilization:
Enable sibling-level preemption
Use case: A high-priority job is queued within its quota due to resource scarcity and needs to preempt resources from jobs in sibling quotas.
Mechanism: The system reallocates computing resources from sibling quotas to critical jobs.
Example: If you enable sibling-level preemption for
Quota-1, its queued jobs can preempt resources from running jobs in its sibling quotas, such asQuota-2, when resources are insufficient.
Use case: A job submitted to a parent quota cannot obtain sufficient resources from its own allocation.
Mechanism: The system reallocates computing resources from child quotas to critical jobs in the parent quota.
Example: If you enable child-level preemption for
Quota-1, jobs created in this quota can preempt resources fromQuota-1.1andQuota-1.2when its computing resources are insufficient.
Enable sibling-level preemption
Procedure
When you create or edit a quota, enable sibling-level preemption.
After you turn on the Sibling-level preemption switch, the Preemption Configuration dialog box appears. Configure the preemption policy parameters in this dialog box.
If a job is submitted and its quota has insufficient resources, the system preempts resources from sibling quotas based on the following policy settings:
Parameter | Description |
High priority | The priority of jobs that can preempt resources from sibling quotas. Valid values: [1,9], specified as a single value or a range. Higher numbers indicate higher priority. |
Low preemption priority | The priority of preemptible jobs. Valid values: [1,9], specified as a single value or a range. Higher numbers indicate higher priority. Note Low preemption priority must be lower than High priority. |
Preemptible module | When a quota has insufficient computing resources, the system can preempt resources from jobs in specific modules within sibling quotas, including DLC, DSW, or EAS. |
Example
Preemption configuration:
Configure the preemption policy with the following parameters:
High Priority: Set to a range of 6 to 9.
Low Preemption Priority: Set to a single value of 4.
Preemptible module: Select DLC and DSW.
Result:
If a job with a priority from 6 to 9 is created in this quota and computing resources are insufficient, the system can preempt computing resources from DLC or DSW jobs that have a priority of 4 in sibling quotas.
Enable child-level preemption
Procedure
When you create or edit a quota, enable child-level preemption.
On the Overview tab of the dedicated resource group details page, find the Scheduling Information section, turn on the Child-level preemption switch, and configure Preemptible priority and Preemptible module in the Preemption Configuration dialog box.
If a job is submitted to the parent quota and computing resources are insufficient, the system reclaims computing resources from running jobs in child quotas based on an optimal strategy to allow the job in the parent quota to run.
Parameter | Description |
Preemptible priority | The priority of preemptible jobs in child quotas. Valid values: [1,9], specified as a range. Higher numbers indicate higher priority. |
Preemptible module | The modules in child quotas (such as DLC, DSW, or EAS) from which resources can be preempted. |
Example
Preemption configuration:
Configure the preemption policy with the following parameters:
Preemptible priority: Set to a range of 1 to 3.
Preemptible module: Select DLC and DSW.
Result:
If the parent quota lacks sufficient computing resources for a job, the system can preempt computing resources from DLC or DSW jobs that have a priority from 1 to 3 in child quotas.
Related documentation
Distributed training (DLC) jobs created in a subscription quota can use idle resources. For more information, see Use idle resources.