How to customize pod scheduling in ACS - Container Compute Service

Priority-based resource scheduling is provided by Alibaba Cloud to meet elasticity requirements in pod scheduling. During application release or scaling, you can configure a custom resource policy to schedule application pods to Alibaba Cloud Container Compute Service (ACS). This way, Elastic Compute Service (ECS) instances and ACS pods can be deployed in a colocated manner.

Important

In kube-scheduler v1.x.x-aliyun-6.4 and later, the ignorePreviousPod parameter of a ResourcePolicy is set to False and the ignoreTerminatingPod parameter is set to True by default. Existing ResourcePolicies that use the preceding parameters are not affected by this change or further updates.

Limits

If you use this feature together with pod-deletion-cost in scale-in scenarios, you cannot perform scale-ins in reverse order.
The max parameter is available only if your cluster runs Kubernetes 1.26 or later and the version of the scheduler installed in your cluster is 6.7 or later.
If you use this feature together with elastic node pools, invalid nodes may be added to the elastic node pools. Make sure that the elastic node pools are included in units. Do not specify the max parameter for the units.

Usage method

For more information about the priority-based scheduling feature of Container Service for Kubernetes (ACK) Pro clusters, see Configure priority-based resource scheduling.

Prerequisites

An ACK Pro cluster that runs Kubernetes 1.26 or later is created. For more information about how to update the Kubernetes version of an ACK cluster, see Update the Kubernetes version of an ACK cluster.

Specific versions of the scheduler are required based on the Kubernetes version of the cluster. The following table describes the required scheduler versions and virtual node versions for each Kubernetes version.
Kubernetes version
Scheduler version
Virtual node version
≥ V1.26
≥ V1.26.3-aliyun-6.7.0
≥ V2.13.0
For more information about the features of different scheduler versions, see kube-scheduler.
For more information about how to enable the computing power of ACS for an ACK Pro cluster, see Use the computing power of ACS in ACK Pro clusters.

Colocated scheduling of ECS instances and ACS computing power

You want to schedule the pods of a Deployment to multiple types of resources, such as subscription ECS instances, pay-as-you-go ECS instances, and ACS pods. To reduce the resource cost, you want to schedule pods to resources based on the following priorities: subscription ECS instances > pay-as-you-go ECS instances > ACS pods. In scale-in activities, you want to delete pods from these resources based on the following sequence: ACS pods, pay-as-you-go ECS instances, and subscription ECS instances. In this example, each node has 2 vCores and 4 GB of memory. Perform the following steps to configure colocated scheduling of ECS instances and ACS computing power:

Run the following command to add labels that indicate different billing methods to the nodes. You can also use node pools to automatically add labels to the nodes.

kubectl label node cn-beijing.10.0.3.137 paidtype=subscription 
kubectl label node cn-beijing.10.0.3.138 paidtype=subscription
kubectl label node cn-beijing.10.0.6.46 paidtype=pay-as-you-go
kubectl label node cn-beijing.10.0.6.47 paidtype=pay-as-you-go

Node label description

paidtype=subscription: subscription
paidtype=pay-as-you-go: pay-as-you-go

Create a ResourcePolicy based on the following content:

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx               # You must specify the label of the pods to which you want to apply the ResourcePolicy. 
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      paidtype: subscription
  - resource: ecs
    nodeSelector:
      paidtype: pay-as-you-go
  - resource: acs

Use the following template to create a Deployment that provisions two pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx                                           # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy. 
        alibabacloud.com/compute-class: general-purpose      # The compute class of the ACS computing power. Default value: general-purpose.
        alibabacloud.com/compute-qos: default                # The quality of service (QoS) class of the ACS pod. Default value: default.
    spec:
      containers:
      - name: nginx
        image: registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2

Deploy an NGINX application and query the pods.

Run the following command to deploy an NGINX application:
```
kubectl apply -f nginx.yaml 
```
Expected output:
```
deployment.apps/nginx created
```

Run the following command to query the pods:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          66s   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          66s   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

The output indicates that two pods are scheduled to nodes that have the paidtype=subscription label. In this example, cn-beijing.10.0.3.137 and cn-beijing.10.0.3.138 are used.

Scale out pods for the NGINX application.

Run the following command to increase the number of pods to four:
```
kubectl scale deployment nginx --replicas 4
```
Expected output:
```
deployment.apps/nginx scaled
```

Run the following command to query the status of the pod:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running   0          16s     172.29.113.155   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running   0          3m48s   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running   0          16s     172.29.113.88    cn-beijing.10.0.6.46    <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          3m48s   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

The output indicates that new pods are scheduled to nodes that have the paidtype=pay-as-you-go label because nodes that have the paidtype=subscription label are insufficient. In this example, cn-beijing.10.0.6.46 and cn-beijing.10.0.6.47 are used.

Run the following command to increase the number of pods to six:
```
kubectl scale deployment nginx --replicas 6
```
Expected output:
```
deployment.apps/nginx scaled
```

Run the following command to query the status of the pod:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE                           NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running   0          3m10s   172.29.113.155   cn-beijing.10.0.6.47           <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running   0          6m42s   172.29.112.215   cn-beijing.10.0.3.137          <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running   0          3m10s   172.29.113.88    cn-beijing.10.0.6.46           <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          6m42s   172.29.113.23    cn-beijing.10.0.3.138          <none>           <none>
nginx-9cdf7bbf9-s****   1/1     Running   0          36s     10.0.6.68        virtual-kubelet-cn-beijing-j   <none>           <none>
nginx-9cdf7bbf9-v****   1/1     Running   0          36s     10.0.6.67        virtual-kubelet-cn-beijing-j   <none>           <none>

The output indicates that the ECS instance has insufficient resources. The pod is scheduled to ACS. In this example, virtual-kubelet-cn-beijing-j is used.

Scale in pods for the NGINX application.

Run the following command to reduce the number of pods to four:
```
kubectl scale deployment nginx --replicas 4
```
Expected output:
```
deployment.apps/nginx scaled
```

Run the following command to query the status of the pod:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                           NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running       0          4m59s   172.29.113.155   cn-beijing.10.0.6.47           <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running       0          8m31s   172.29.112.215   cn-beijing.10.0.3.137          <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running       0          4m59s   172.29.113.88    cn-beijing.10.0.6.46           <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running       0          8m31s   172.29.113.23    cn-beijing.10.0.3.138          <none>           <none>
nginx-9cdf7bbf9-s****   1/1     Terminating   0          2m25s   10.0.6.68        virtual-kubelet-cn-beijing-j   <none>           <none>
nginx-9cdf7bbf9-v****   1/1     Terminating   0          2m25s   10.0.6.67        virtual-kubelet-cn-beijing-j   <none>           <none>

The output indicates that the pods that run on ACS are deleted.

Run the following command to reduce the number of pods to two:
```
kubectl scale deployment nginx --replicas 2
```
Expected output:
```
deployment.apps/nginx scaled
```

Run the following command to query the status of the pod:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   0/1     Terminating   0          6m43s   172.29.113.155   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running       0          10m     172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-f****   0/1     Terminating   0          6m43s   172.29.113.88    cn-beijing.10.0.6.46    <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running       0          10m     172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

The output indicates that the pods on the nodes that have the paidtype=pay-as-you-go label are deleted.

Wait a few minutes and run the following command to query the pod state again:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          11m   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          11m   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

The output indicates that pods run only on the nodes with the paidtype=subscription label.

References

When you deploy an ACK cluster, you can configure tolerations and node affinity to enable the scheduler to use only ECS instances or the computing power of ACS, or allow the scheduler to automatically apply for ACS computing resources when ECS instances are insufficient. You can configure different scheduling policies to scale resources in different scenarios. For more information, see Use the computing power of ACS in ACK Pro clusters.
High availability and high performance are essential to distributed jobs. In an ACK Pro cluster, you can use Kubernetes-native scheduling semantics to spread distributed jobs across zones for high availability. You can also use Kubernetes-native scheduling semantics to deploy distributed jobs in specific zones based on affinity settings for high performance. For more information, see Node affinity scheduling.

Kubernetes version	Scheduler version	Virtual node version
≥ V1.26	≥ V1.26.3-aliyun-6.7.0	≥ V2.13.0