Mixed ECS and ACS pod scheduling in ACK Pro clusters - Container Compute Service

Define a ResourcePolicy to schedule pods across ECS nodes and ACS in priority order. During scale-out, pods land on higher-priority resources first; during scale-in, lower-priority pods are evicted first.

Important

In kube-scheduler v1.x.x-aliyun-6.4 and later, by default the ignorePreviousPod parameter of a ResourcePolicy is set to False and the ignoreTerminatingPod parameter is set to True. Existing ResourcePolicies that use the preceding parameters are not affected by this change or further updates.

Limits

When used with pod-deletion-cost, reverse-order scale-in is not supported.
The max parameter requires Kubernetes 1.26+ and scheduler version 6.7+.
When used with elastic node pools, include the pools in units and do not specify the max parameter. Otherwise, invalid nodes may be added.

Use priority-based resource scheduling

Configure priority and order for pod placement across resource types. Configure priority-based resource scheduling.

Prerequisites

An ACK Pro cluster running Kubernetes 1.26 or later is created. Update the Kubernetes version of an ACK cluster.

The required scheduler and virtual node versions depend on the cluster Kubernetes version:

Kubernetes version	Scheduler version	Virtual node version
≥ 1.26	≥ 1.26.3-aliyun-6.7.0	≥ 2.13.0

For more information about the features of different scheduler versions, see Container Service for Kubernetes:kube-scheduler.
ACS computing power is enabled for the ACK Pro cluster. Use the computing power of ACS in ACK Pro clusters.

Example: colocated ECS and ACS scheduling

This example schedules pods across subscription ECS, pay-as-you-go ECS, and ACS in that priority order. On scale-in, pods are evicted in reverse. Each node has 2 vCPUs and 4 GB memory.

Add labels to distinguish billing methods on nodes. In production, use node pools to assign labels automatically.

kubectl label node cn-beijing.10.0.3.137 paidtype=subscription 
kubectl label node cn-beijing.10.0.3.138 paidtype=subscription
kubectl label node cn-beijing.10.0.6.46 paidtype=pay-as-you-go
kubectl label node cn-beijing.10.0.6.47 paidtype=pay-as-you-go

Node label description

paidtype=subscription: subscription ECS nodes

paidtype=pay-as-you-go: pay-as-you-go ECS nodes

Create the following ResourcePolicy:

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx               # Must match the 'app' label in the pod template metadata. 
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      paidtype: subscription
  - resource: ecs
    nodeSelector:
      paidtype: pay-as-you-go
  - resource: acs

Create a Deployment with two replicas:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx                                           # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy. 
        alibabacloud.com/compute-class: general-purpose      # The compute class of the ACS computing power. Default value: general-purpose.
        alibabacloud.com/compute-qos: default                # The quality of service (QoS) class of the ACS pod. Default value: default.
    spec:
      containers:
      - name: nginx
        image: registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2

Deploy and verify:

Apply the manifest:

kubectl apply -f nginx.yaml

Expected output:

deployment.apps/nginx created

Verify pod placement:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          66s   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          66s   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

Both pods land on subscription nodes (paidtype=subscription label).

Scale up the nginx application.

Scale to four replicas:

kubectl scale deployment nginx --replicas 4

Expected output:

deployment.apps/nginx scaled

Check pod placement:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running   0          16s     172.29.113.155   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running   0          3m48s   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running   0          16s     172.29.113.88    cn-beijing.10.0.6.46    <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          3m48s   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

New pods land on pay-as-you-go nodes (paidtype=pay-as-you-go label) because paidtype=subscription label nodes are full.

Scale to six replicas:

kubectl scale deployment nginx --replicas 6

Expected output:

deployment.apps/nginx scaled

Check pod placement:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE                           NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running   0          3m10s   172.29.113.155   cn-beijing.10.0.6.47           <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running   0          6m42s   172.29.112.215   cn-beijing.10.0.3.137          <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running   0          3m10s   172.29.113.88    cn-beijing.10.0.6.46           <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          6m42s   172.29.113.23    cn-beijing.10.0.3.138          <none>           <none>
nginx-9cdf7bbf9-s****   1/1     Running   0          36s     10.0.6.68        virtual-kubelet-cn-beijing-j   <none>           <none>
nginx-9cdf7bbf9-v****   1/1     Running   0          36s     10.0.6.67        virtual-kubelet-cn-beijing-j   <none>           <none>

ECS capacity is exhausted. New pods are scheduled to ACS (virtual-kubelet node).

Scale down the nginx application.

Scale to four replicas:

kubectl scale deployment nginx --replicas 4

Expected output:

deployment.apps/nginx scaled

Check pod status:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                           NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running       0          4m59s   172.29.113.155   cn-beijing.10.0.6.47           <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running       0          8m31s   172.29.112.215   cn-beijing.10.0.3.137          <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running       0          4m59s   172.29.113.88    cn-beijing.10.0.6.46           <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running       0          8m31s   172.29.113.23    cn-beijing.10.0.3.138          <none>           <none>
nginx-9cdf7bbf9-s****   1/1     Terminating   0          2m25s   10.0.6.68        virtual-kubelet-cn-beijing-j   <none>           <none>
nginx-9cdf7bbf9-v****   1/1     Terminating   0          2m25s   10.0.6.67        virtual-kubelet-cn-beijing-j   <none>           <none>

The output shows that pods are evicted in reverse order of scheduling priority: first from ACS, then from pay-as-you-go ECS nodes, and last from subscription ECS nodes.

Scale to two replicas:

kubectl scale deployment nginx --replicas 2

Expected output:

deployment.apps/nginx scaled

Check pod status:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   0/1     Terminating   0          6m43s   172.29.113.155   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running       0          10m     172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-f****   0/1     Terminating   0          6m43s   172.29.113.88    cn-beijing.10.0.6.46    <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running       0          10m     172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

Pay-as-you-go node pods (paidtype=pay-as-you-go label) are terminated.

Wait a few minutes, then check again:

kubectl get pods -o wide

Expected output:

NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          11m   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          11m   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

Only pods on paidtype=subscription label nodes remain.

References

When deploying workloads in an ACK cluster, use tolerations and node affinity to restrict scheduling to ECS or ACS resources, or to auto-provision ACS when ECS is insufficient. Different scheduling policies support various scaling scenarios. See Use the computing power of ACS in ACK Pro clusters.
For distributed jobs, leverage Kubernetes-native scheduling to spread workloads across zones for high availability or confine them to specific zones via affinity for high performance. See Node Affinity Scheduling.