When multiple teams share a GPU cluster, high-priority jobs can stall waiting for resources held by lower-priority ones. Configure quota preemption policies to let the system automatically reclaim those resources and keep critical workloads running.
PAI supports two independent preemption directions:
| Type | Who preempts whom | Typical trigger |
|---|---|---|
| Self-level preemption | A job in one quota preempts resources from jobs in sibling quotas at the same level | A high-priority job's quota is full, but idle capacity exists in peer quotas |
| Child-level preemption | A parent quota reclaims resources from jobs running in its child quotas | A job submitted to a parent quota cannot get enough resources from the parent's own allocation |
You can enable both types on the same quota to cover resource reclamation in both directions.
Quota hierarchy
PAI organizes computing resources into a tree-like hierarchy called QuotaTree, with parent-level and child-level quotas at each node.
The diagram shows a three-level structure:
Root | Quota is the top-level parent quota. Quota-1, Quota-2, ... Quota-n are its child-level quotas.
Quota-1 is the parent of Quota-1.1 and Quota-1.2. The same structure applies to Quota-2 through Quota-n.
Divide quotas by team or project to isolate resource usage. During resource contention, enable preemption policies to let high-priority jobs claim resources from lower-priority ones.
Enable self-level preemption
Self-level preemption lets jobs in one quota take resources from jobs running in peer quotas at the same level. The preempting job must have a higher priority than the jobs it preempts.
Configure self-level preemption
Enable self-level preemption when creating or editing a quota.
When resources are insufficient, the system preempts resources from same-level quotas based on the following policy settings:
| Preemption policy | Description |
|---|---|
| High priority | Priority of jobs that can preempt resources from same-level quotas. Valid values: 1–9, set as Single Selection or Range. Higher numbers indicate higher priority. |
| Low preemption priority | Priority of jobs in same-level quotas whose resources can be preempted. Valid values: 1–9, set as Single Selection or Range. Higher numbers indicate higher priority. Must be set to a lower number than High priority — otherwise the policy is invalid. |
| Preemptible module | Modules in same-level quotas whose jobs the system can preempt. Supported modules: Deep Learning Containers (DLC), Data Science Workshop (DSW), and Elastic Algorithm Service (EAS). |
Example
Configure the policy as follows:
High priority: Range 6–9
Low preemption priority: Single value 4
Preemptible module: DLC and DSW
Result: When a job with priority 6–9 is submitted to this quota and resources are insufficient, the system preempts resources from DLC or DSW jobs with priority 4 in same-level quotas.
Enable child-level preemption
Child-level preemption lets a parent quota reclaim resources from jobs running under its child quotas. This allows parent-level jobs to get the resources they need without waiting for child-level jobs to finish.
Configure child-level preemption
Enable child-level preemption when creating or editing a quota.
When a job submitted to this parent-level quota cannot get enough resources, the system reclaims resources from jobs in child-level quotas based on the following policy settings:
| Preemption policy | Description |
|---|---|
| Preemptible priority | Priority of jobs in child-level quotas whose resources can be preempted. Valid values: 1–9. Higher numbers indicate higher priority. |
| Preemptible module | Modules in child-level quotas whose jobs the system can preempt when the parent-level quota has insufficient resources. Supported modules: DLC, DSW, and EAS. |
Example
Configure the policy as follows:
Preemptible priority: Range 1–3
Preemptible module: DLC and DSW
Result: When a job submitted to the parent-level quota cannot get enough resources, the system preempts resources from DLC or DSW jobs with priority 1–3 in any child-level quota.
Usage notes
See also
Distributed training (DLC) jobs created in subscription quotas can use idle resources. For more information, see Use idle resources.