Add custom admission checks to ACK Kube Queue with AdmissionCheck - Container Service for Kubernetes

By default, ACK Kube Queue only checks theMax quota defined in an ElasticQuotaTree. After a workload passes the quota check, it is processed by the scheduler. If you need more complex dequeue validation logic, you can use the AdmissionCheck mechanism. AdmissionCheck is an external, pluggable mechanism provided by ACK Kube Queue that provides additional validation before a workload is dequeued from a queue, ensuring that only eligible workloads are scheduled.

How it works

After an AdmissionCheck is added to a cluster, ACK Kube Queue does not immediately set a QueueUnit to the Dequeued phase after it passes the quota check. Instead, the QueueUnit is set to the Reserved phase, and admissionChecks with a Pending state are added to the QueueUnit's status field.

apiVersion: scheduling.x-k8s.io/v1alpha1
kind: QueueUnit
metadata:
  ...
spec:
  ...
status:
  admissionChecks:
  - lastTransitionTime: "2025-01-17T01:46:58Z"
    message: ""
    name: sample-prov
    state: Pending
  lastUpdateTime: "2025-01-17T01:46:58Z"
  message: Reserved resources success, waiting admission checks to be ready
  phase: Reserved
  podState: {}

Each AdmissionCheckController checks its respective admissionChecks, and after a check passes, sets the state in admissionChecks to Ready. When all AdmissionChecks are in the Ready state, the QueueUnit enters the Dequeued state.

apiVersion: scheduling.x-k8s.io/v1alpha1
kind: QueueUnit
metadata:
  ...
spec:
  ...
status:
  admissionChecks:
  - lastTransitionTime: "2025-01-17T01:47:58Z"
    message: Succeeded
    name: sample-prov
    state: Ready
  lastUpdateTime: "2025-01-17T01:50:45Z"
  message: Dequeued
  phase: Dequeued
  podState:
    pending: 1

AdmissionCheck configuration

Configuration details

An administrator must configure the following components in the cluster.

One controller: a custom AdmissionCheckController (ACController).

The AdmissionCheckController is a controller that you maintain to perform custom checks. By listening to the admissionChecks field in the status of a QueueUnit, the controller can identify the QueueUnits that require a custom check. The controller can determine if it needs to check an admissionChecks entry by inspecting the controllerName in the AdmissionCheck that corresponds to the name in the admissionChecks field of the QueueUnit's status. After the check is complete, the controller updates the status of the corresponding admissionChecks entry in the QueueUnit's status to Ready or Rejected based on the result.

Two Custom Resource Definitions (CRDs):

AdmissionCheck

This CRD is installed automatically with ACK Kube Queue. To define the custom check logic, you submit a custom resource (CR) that specifies the controller to handle the check and references its parameters.

The following example shows an AdmissionCheck manifest and its available fields.

apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
  name: sample-prov
spec:
  controllerName: provisioning-request-admission-controller
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: ProvisioningRequestConfig
    name: prov-test-config

Parameter	Type	Description
`controllerName`	string	Specifies the name of the controller that watches and processes this admission check.
`parameters`	struct	A struct type composed of the three string fields `apiGroup`, `kind`, and `name`, which describes the parameters associated with the AdmissionCheck.

Custom check parameters

An AdmissionCheck resource uses the parameters field to reference a custom parameter object. The controller reads these parameters to execute specific check logic. By using distinct parameters for each check, a single controller can perform different types of checks. You must install the CRD and submit the CR for the custom parameters.

A configuration item in a queue:

After you deploy the AdmissionCheckController, the AdmissionCheck, and the custom parameters, your cluster is ready to perform custom checks. You can then enable a custom check for a specific queue by adding it to the queue's configuration.

Configuration example

This example shows how to configure an AdmissionCheck using the ProvisioningRequestAdmissionCheckController.

1. Enable the ProvisioningRequestAdmissionCheck controller

The ProvisioningRequestAdmissionCheckController is a controller provided by ACK Kube Queue that works with ProvisioningRequest. You can enable ProvisioningRequestAdmissionCheck by setting .Values.admissionCheckController.enabled to true when you install the Helm chart for ACK Kube Queue. Once enabled, the ProvisioningRequestAdmissionCheckController is automatically deployed.

# Chart Url: https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/ack-kube-queue-1.22.4.tgz
admissionCheckController:
  enabled: true  # Set to true to enable ProvisioningRequestAdmissionCheck
  image:
    repository: acs/prevreq-controller
    tag: v1.0.0-aliyun-d31082a1
burst: 50
controller:
  defaultPreemptible: true
  enableBlockingMode: false
  enableStrictPriority: false
  image:
    repository: acs/kube-queue
    tag: v0.11.1-aliyun-34b259af
  queueGroupPlugin: elasticquota
  resources:
    # ...

2. Use the controller in an AdmissionCheck

To use the ProvisioningRequestAdmissionCheckController, submit an AdmissionCheck as shown in the following example and set the controllerName of the AdmissionCheck to provisioning-request-admission-controller.

apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
  name: sample-prov
spec:
  controllerName: provisioning-request-admission-controller
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: ProvisioningRequestConfig
    name: prov-test-config

3. Define parameters with ProvisioningRequestConfig

The ProvisioningRequestAdmissionCheck controller uses a ProvisioningRequestConfig to define custom check parameters. The following manifest shows its available fields.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
  name: prov-test-config
spec:
  provisioningClassName: atomic-scale-up.kubernetes.io
  parameters:
  managedResources:
  - nvidia.com/gpu

Parameter	Type	Description
`provisioningClassName`	string	When a `provisioningRequest` is created, this is copied to the `ProvisioningClass` field of the `provisioningRequest`.
`parameters`	map[string]string	is copied to the `Parameters` field of the `provisioningRequest` when the `provisioningRequest` is created.
`managedResources`	[]string	If a QueueUnit contains the field, a `provisioningRequest` is created for it. Otherwise, its `state` is directly set to `Ready`.

4. Add the AdmissionCheck to a queue

You can add checks to the queue configuration in the admissionChecks field. The following is an example.

apiVersion: scheduling.x-k8s.io/v1alpha1
kind: Queue
metadata:
  name: example-queue # Replace with the name of the queue where you want to enable the AdmissionCheck
  namespace: kube-queue
spec:
  ...
  admissionChecks:
  - name: sample-prov # Specify the name of the AdmissionCheck you created earlier
    selector: # A metav1.LabelSelector. Only QueueUnits that match this selector undergo the custom check.
      matchLabels:
        app: gpu