Platform for AI (PAI) provides multi-level quotas. This allows you to divide computing resources into different levels for flexible management and allocation. You can enable the child-level or self-level preemption feature to allow jobs that run on the current resource quota to preempt the computing resources of jobs that run on the same-level or child-level quotas. This way, the computing resources can be fully utilized.
Quota levels
PAI allows you to divide computing resources into quotas at the parent-child level based on the enterprise organizational structure. The following figure shows the quotas of computing resources in a tree-like structure. This allows you to manage and allocate computing resources in a flexible and fine-grained manner. The hierarchy makes resource management more aligned with the actual requirements of enterprises and improves the efficiency and flexibility of resource utilization.
The computing resources are divided into three levels, as shown in the preceding figure.
Root Quota is the parent-level quota and Quota-1, Quota-2, ..., Quota-n are the child-level quotas of Root Quota. The child-level quotas are considered as same-level quotas.
Quota-1 is the parent-level quota of Quota-1.1 and Quota-1.2. Quota-1.1 and Quota-1.2 are the child-level quotas of Quota-1. The child-level quotas are considered as same-level quotas. The same rule applies to Quota-2, ..., Quota-n.
You can divide and use quotas based on organizations or projects to manage jobs and resources in a more efficient manner. If resources are insufficient, you can create quotas at multiple levels and enable the preemption policies provided by the product to maximize resource utilization:
Scenario: A job created by using same-level quotas, such as Quota-1, Quota-2, or Quota-n, needs to be preferentially run and resources are insufficient.
Processing mechanism: The system provides a flexible resource scheduling mechanism to schedule same-level computing resources. This ensures that computing resources are available for key jobs.
Example: Enabling intra-quota computing power preemption for Quota-1 allows queued jobs to preempt resources from running jobs within the same resource quota when resources are scarce.
Enable child-level preemption:
Scenario: The same-level quotas are insufficient and cannot meet the job requirements.
Processing mechanism: The system provides a flexible resource scheduling mechanism to schedule child-level computing resources. This ensures that computing resources are available for key jobs.
Example: Enable child-level preemption for Quota-1. When you create a job by using Quota-1 and the computing resources are insufficient, the resources of Quota-1.1 and Quota-1.2 can be preempted.
Enable self-level preemption
Procedure
Enable self-level preemption when you create or modify a quota.
When multiple jobs are run on the quota at the same time, the system allocates the computing resources at the current level based on the optimal preemption policy. The following table describes the preemption configurations.
Preemption policy | Description |
High Priority | The jobs that preempt computing resources at the current level are preferentially run. Valid values: 1 to 9. You can select Single Selection or Range. A larger number indicates a higher priority. A smaller number indicates a lower priority. |
Low Preemption Priority | The jobs whose computing resources are preempted are preferentially run. Valid values: 1 to 9. You can select Single Selection or Range. A larger number indicates a higher priority. A smaller number indicates a lower priority. Note The value of Low Preemption Priority must be less than the value of High Priority. |
Preemptible Module | When the computing resources of a quota are insufficient, the system can preempt the computing resources of modules that run on the same-level quotas, such as Deep Learning Containers (DLC), Data Science Workshop (DSW), and Elastic Algorithm Service (EAS) modules. |
Example
Preemption configurations:
The following figure shows the preemption configurations:
High Priority: Set the value to 6 to 9.
Low Preemption Priority: Select Single Selection and set the value to 4.
Preemptible Module: Select DLC and DSW.
Effect:
If the priority of a job created by using this quota is 6 to 9 and the computing resources are insufficient, the system can preempt the computing resources of the DLC or DSW module with a priority of 4 that uses the same-level quotas.
Enable child-level preemption
Procedure
Enable child-level preemption when you create or modify a quota.
When a job runs on a parent-level quota and the computing resources are insufficient, the system reclaims the computing resources of a child-level quota on which jobs are run based on the optimal preemption policy. This ensures the running of jobs that run on the parent-level quota. The following table describes the preemption configurations.
Preemption policy | Description |
Preemptible Priority | The priority of jobs whose computing resources are preempted and are created by using child-level quotas. Valid values: 1 to 9. You can specify a range. A larger number indicates a higher priority. A smaller number indicates a lower priority. |
Preemptible Module | When the computing resources of a parent-level quota are insufficient, the system can preempt the computing resources of modules that run on the child-level quotas, such as DLC, DSW, and EAS modules. |
Procedure
Preemption configurations:
The following figure shows the preemption configurations:
Preemptible Priority: Set the value to 1 to 3.
Preemptible Module: Select DLC and DSW.
Effect:
If a job is created by using a parent-level quota and the computing resources are insufficient, the system can preempt the computing resources of the DLC or DSW module with a priority of 1 to 3 that uses the child-level quotas.
References
You can use idle resources when you submit training jobs by using subscription quotas in DLC. For more information, see Use idle resources.