Scheduling of AI workloads - Container Service for Kubernetes

This topic describes the scheduling of elastic resources, AI tasks, heterogeneous resources, and task queues. These features can help maximize resource utilization and improve the efficiency of jobs.

Elastic scheduling

ACK provides hybrid scheduling for different types of elastic resources.

Feature

Description

References

Elastic scheduling

Alibaba Cloud provides different types of elastic resources, such as Elastic Compute Service (ECS) instances and elastic container instances. The following billing methods are supported: subscription, pay-as-you-go, and preemptible instances.

This feature allows you to schedule various types of resources such as ECS instances and elastic container instances. You can also configure priority-based resource scheduling policies for an application. When the system deploys or scales out pods for an application, pods are scheduled to nodes based on the priorities of the nodes that are listed in the scheduling policies that you configured. When the system scales in pods for an application, pods are removed from nodes based on the priorities of the nodes in ascending order.

For example, you can configure the system to prioritize ECS instances during scale-out activities. The system starts to use elastic container instances only after ECS instances are exhausted. The system prioritizes elastic container instances over ECS instances during scale-in activities. This helps you reduce costs and use resources in a more efficient manner.

Task scheduling

ACK provides the gang scheduling and capacity scheduling features for batch processing tasks.

Feature	Description	References
Gang scheduling	For example, if a job requires all-or-nothing scheduling, all tasks of the job must be scheduled at the same time. If only some of the tasks are started, the started jobs must wait until all the remaining tasks are scheduled. If each submitted job contains unscheduled tasks, all submitted jobs remain in the Pending state and the cluster is deadlocked. To resolve this issue, Alibaba Cloud provides the gang scheduling feature. Gang scheduling aims to start all correlated processes at the same time. This prevents the process group from being blocked when the system fails to start some processes.	Work with gang scheduling
Capacity Scheduling	Kubernetes uses resource quotas to allocate resources based on fixed amounts. However, a Kubernetes cluster may be managed by multiple users who use cluster resources in different cycles and ways. As a result, resource utilization is low in the Kubernetes cluster. To improve the resource utilization of a Kubernetes cluster, ACK provides the capacity scheduling feature to optimize resource allocation. This feature is designed based on the Yarn capacity scheduler and the Kubernetes scheduling framework. Capacity scheduling allows you to meet the resource requests in a Kubernetes cluster and improve resource utilization by sharing idle resources.	Use Capacity Scheduling
Kube Queue	Schedulers schedule resources by pod. If you run a large number of tasks in your cluster, the scheduling efficiency of the schedulers is adversely affected. In addition, jobs submitted by different users may interfere with each other during the scheduling process. ack-kube-queue is designed to manage AI, machine learning, and batch workloads in Kubernetes. It allows system administrators to customize job queue management and improve the flexibility of queues. Integrated with a quota system, ack-kube-queue automates and optimizes the management of workloads and resource quotas to maximize resource utilization in Kubernetes clusters.	Use ack-kube-queue to manage job queues

Scheduling of heterogeneous resources

Container Service for Kubernetes (ACK) provides the cGPU, topology-aware CPU scheduling, and topology-aware GPU scheduling features to allow you to schedule heterogeneous resources. For more information about the node labels that ACK uses to schedule GPUs, see Labels used by ACK to control GPUs.

Feature	Description	References
cGPU	cGPU provides GPU sharing to help reduce the costs of GPU resources while ensuring the stability of workloads that require GPU resources. ACK Pro clusters support the following GPU policies: GPU sharing and memory isolation on a one-pod-one-GPU basis. This policy is commonly used in model inference scenarios. GPU sharing and memory isolation on a one-pod-multi-GPU basis. This policy is commonly used to build the code to train distributed models. GPU allocation based on the binpack or spread algorithm. This policy is commonly used to improve GPU utilization and ensure the high availability of GPUs.	cGPU Professional Edition
Topology-aware CPU scheduling and topology-aware GPU scheduling	To ensure the high performance of workloads, schedulers select an optimal scheduling solution based on the topological information about the heterogeneous resources of nodes. The information includes how GPUs communicate with each other by using NVLink and PCIe Switches, and the non-uniform memory access (NUMA) topology of CPUs.	Topology-aware CPU scheduling Overview
Field Programmable Gate Array (FPGA) scheduling	This feature allows you to manage the FPGA resources of a cluster in a unified manner. You can use this feature to schedule workloads that require FPGA resources to FPGA-accelerated nodes.	Use labels to schedule pods to FPGA-accelerated nodes

Task queue scheduling

ACK allows you to customize task queue scheduling for AI workloads, machine learning workloads, and batch jobs.