Idle-time resources let DLC jobs use spare capacity from other resource quotas without disrupting those workloads, improving overall resource utilization and reducing training costs.
How it works
On the PAI platform, different teams submit training jobs through their own quotas. Some quotas sit idle during certain periods while other teams queue for resources. Idle-time resources solve this problem by letting DLC jobs use the spare capacity from other quotas.
How idle-time resources work:
Idle-time jobs submitted to a general-purpose computing quota can use spare resources within that quota. Idle-time jobs submitted to a Lingjun intelligent computing quota can use spare resources of the same specification across all quotas under the same tenant.
Idle-time jobs aren't limited by the total resource capacity of the associated quota. They can use spare resources from the current quota or other quotas.
When the original quota reclaims its resources for scheduled jobs, idle-time jobs are automatically terminated and the resources are released.
When used with AIMaster and EasyCKPT, idle-time jobs can automatically save progress and resume after preemption, avoiding compute waste.
Prerequisites
A prepaid resource quota (general-purpose computing or Lingjun intelligent computing) bound to a workspace. For more information, see Overview.
Submit a DLC job with idle-time resources
When you submit a DLC training job in the console, configure Idle Resources after selecting a resource quota for either Lingjun Intelligent Computing or General Computing in the Resource Information section. The following table describes the key parameters. For more information, see Create a training job.
Parameter
Description
Resource Type
Select General Computing or Lingjun AI Computing Service.
NoteLingjun AI Computing Service resources are designed for high-performance AI training.
Source
Select Resource Quota.
Idle Resources
Valid values:
Acceptable: The job can use idle-time resources from all quotas under the tenant, or use the selected quota, whichever dequeues first.
Only Idle Resources: The job uses only idle-time resources.
Jobs that run on idle-time resources may be interrupted when resources are reclaimed. Make sure your code includes a checkpoint mechanism so the job can resume after preemption. For more information, see EasyCkpt.
If you select Acceptable and the job starts using the selected quota's own compute resources, it runs as a regular guaranteed job and isn't subject to preemption.
Automatic Fault Tolerance
Idle-time jobs may be preempted. Enable Automatic Fault Tolerance so the system automatically finds new resources and reruns the job when resources are reclaimed. For configuration details, see AIMaster: An elastic and automatic fault tolerance engine.
View the resource usage details of the DLC job.
In the job list or on the job details page, check the resource usage.
In Quota: The job uses resources from its associated quota.
Not in Quota: The job uses idle-time shared resources.
When an idle-time job runs on shared resources, the node name isn't displayed.
When the idle-time resources used by a job are preempted, the instance (pod) status on the job details page is marked as Resource Preempted.
When the lending quota needs to schedule a new job but lacks sufficient resources, the system reclaims the idle-time resources to prioritize that job. The system then marks the instance status as preempted.
References
Idle-time jobs may be preempted. Enable AIMaster: An elastic and automatic fault tolerance engine to automatically resume jobs after preemption. Use EasyCkpt to minimize the loss of training progress when jobs are preempted.