ack-kube-queue is the job queue component of the cloud-native AI suite. Built around the Kube Queue job queue and combined with the scheduler and quota system, it provides queue management, priority scheduling, and elastic quotas to improve the management efficiency and scheduling performance of AI/ML and batch workloads in Kubernetes.
Overview
Kubernetes clusters often run a large number of job-type workloads and pods, such as AI/ML workloads and batch workloads. This volume places significant load on the scheduler, degrades scheduling performance, and can cause jobs submitted by different users to interfere with each other. Built around the Kube Queue job queue and integrated with the scheduler and the quota system, ack-kube-queue manages these workloads with the following capabilities:
Queue management: Lets system administrators manage job queues by using custom queues, which makes queuing more flexible. For instructions, see Use ack-kube-queue to manage job queues.
Priority scheduling: Dequeues jobs by priority in strict priority scheduling mode and provides multi-queue fair scheduling for jobs under different quotas.
Blocking queues: Supports the blocking queue mode, with a global setting through environment variables and a per-queue switch.
Elastic quotas: Combines with the quota system to automatically optimize workload and ResourceQuota management so that cluster resources are fully utilized.
Re-queuing: Reclaims jobs whose pods remain unschedulable for a long time and returns them to the queue for rescheduling. This prevents jobs from occupying quotas without running and improves quota utilization.
Usage notes
The ack-kube-queue component can be installed only in ACK Pro clusters, ACK Serverless Pro cluster, or ACK Edge Pro cluster, and the Kubernetes version of the cluster must be 1.18 or later.
Install ack-kube-queue when you deploy the cloud-native AI suite, or add it later to a cluster in which the suite is already deployed. After the installation, you can use the features provided by ack-kube-queue, such as blocking queues and strict priority scheduling. For more information about how to install and use ack-kube-queue, see Manage AI and ML workloads with ack-kube-queue.
Release notes
January 2024
Version | Changes | Change date | Change impact |
v0.3.4 | Fixed head-of-line blocking that occasionally occurs in blocking queue mode when the first job in the queue is deleted. | January 4, 2024 | None |
December 2023
Version | Changes | Change date | Change impact |
v0.3.3 | Setting blocking queues globally by using environment variables refreshes the blocking queue mode for all queues. | December 26, 2023 | None |
September 2023
Version | Changes | Change date | Change impact |
v0.3.1 | Fixed queue errors that occasionally occur during QueueUnit deletion. | September 13, 2023 | None |
v0.3.0 | Job sequence information can be retrieved from queues. | September 13, 2023 | None |
August 2023
Version | Changes | Change date | Change impact |
v0.2.1 | Fixed an issue where | August 31, 2023 | None |
v0.2.0 |
| August 29, 2023 | None |
July 2023
Version | Changes | Change date | Change impact |
v0.1.13 | Fixed an issue where the component may stop working due to a missing | July 26, 2023 | None |
v0.1.12 | Added a per-queue switch to enable or disable the blocking queue feature. You can disable the re-queuing feature by setting the timeout parameter in the extension to | July 20, 2023 | None |
June 2023
Version | Changes | Change date | Change impact |
v0.1.11 | QueueUnit status is synchronized when jobs are updated. | June 30, 2023 | None |
v0.1.10 | ARM-based nodes are supported for components such as kube-queue-controller, tf-operator-extension, and pytorch-operator-extension. | June 14, 2023 | None |
May 2023
Version | Changes | Change date | Change impact |
v0.1.9 | Added re-queuing for long-pending jobs and multi-queue fair scheduling. If the pods of a job remain unschedulable for a long time due to topology-aware scheduling constraints, affinity constraints, or resource fragments, ack-kube-queue reclaims the job and returns it to the queue for rescheduling. This prevents the job from occupying quotas without running and improves quota utilization. | May 16, 2023 | None |
April 2023
Version | Changes | Change date | Change impact |
v0.1.8 | Added support for the blocking queue mode and strict priority dequeuing. For more information, see Enable blocking queues and Enable strict priority scheduling. | April 25, 2023 | None |
March 2023
Version | Changes | Change date | Change impact |
v0.1.6 | Fixed an issue where the State of a TFJob is not displayed. | March 15, 2023 | None |
February 2023
Version | Changes | Change date | Change impact |
v0.1.5 | Fixed an issue where ack-kube-queue occasionally fails to delete jobs. | February 28, 2023 | None |
v0.1.4 | Fixed an issue where the Used information of a QueueUnit is occasionally lost after the QueueUnit is dequeued. | February 14, 2023 | None |
January 2023
Version | Changes | Change date | Change impact |
v0.1.3 | Fixed an issue where ack-kube-queue occasionally loses QueueUnits. | January 12, 2023 | None |
v0.1.2 | Fixed an issue where ack-kube-queue occasionally cannot dequeue jobs for a long time. | January 12, 2023 | None |
v0.1.1 | Added support for the multi-queue version. Jobs under different quotas are submitted to separate queues to prevent congestion. | January 10, 2023 | None |
October 2022
Version | Changes | Change date | Change impact |
v0.1.0 | Added the component. This is the initial release. | October 15, 2022 | Initial release. |