Container Service for Kubernetes (ACK) allows you to configure priority-based resource scheduling. A ResourcePolicy specifies the priorities of nodes in descending order for pod scheduling. When the system deploys or scales out pods for an application, pods are scheduled to nodes based on the priorities of the nodes that are listed in the ResourcePolicy. When the system scales in pods for an application, pods are removed from nodes based on the priorities of the nodes in ascending order. This topic describes how to configure priority-based resource scheduling.

Prerequisites

  • An ACK Pro cluster that runs Kubernetes 1.20.11 or later is created. For more information about how to update the Kubernetes version of your cluster, see Update the Kubernetes version of an ACK cluster.
  • Specific versions of the scheduler are required based on the Kubernetes version of the cluster. The following table describes the scheduler versions that are required for each Kubernetes version.
    Kubernetes version Scheduler version
    1.20 1.20.4-ack-7.0 or later
    1.22 1.22.15-ack-2.0 or later
    1.24 v1.24.3-ack-2.0 or later
  • ack-virtual-node is deployed if you want to use elastic container instances. For more information, see Use Elastic Container Instance in ACK clusters.

Limits

  • Priority-based resource scheduling is mutually exclusive with the pod deletion cost feature. For more information about the pod deletion cost feature, see Pod deletion cost.
  • You cannot use priority-based resource scheduling and Elastic Container Instance-based scheduling at the same time. For more information, see Use Elastic Container Instance-based scheduling.
  • This feature uses the BestEffort policy and does not ensure that pods are removed from nodes based on the priorities of the nodes in ascending order when the system scales in pods for an application.
  • Before you create a ResourcePolicy, make sure that no pod is scheduled. Pods that are scheduled before you create a ResourcePolicy are deleted first in scale-in activities.
  • After you modify a ResourcePolicy, only the pods that are scheduled based on the modified ResourcePolicy are deleted based on the modified ResourcePolicy. The pods that are scheduled based on the original ResourcePolicy are deleted based on the original ResourcePolicy.

How to configure priority-based resource scheduling

Create a ResourcePolicy with the following template:

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: xxx
  namespace: xxx
spec:
  selector:
    key1: value1
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      key2: value2
  - resource: ecs
    nodeSelector:
      key3: value3
  - resource: eci
  • selector: the selector that is used to select pods in a namespace. The ResourcePolicy is applied to the selected pods. In this example, pods that have the key1=value1 label are selected.
  • strategy: the scheduling policy. Set the value to prefer.
  • units: the schedulable units. In a scale-out activity, pods are scheduled to nodes based on the priorities of the nodes that are listed under units in descending order. In a scale-in activity, pods are deleted from the nodes based on the priorities of the nodes in ascending order.
    • resource: the type of resource for pod scheduling. Valid values: eci and ecs.
    • nodeSelector: the selector that is used to select nodes with a specified node label. This parameter takes effect only if the resource parameter is set to ecs.

Examples

Example 1: Schedule pods to specified node pools

You want to schedule the pods of a Deployment to specific node pools, for example, Node Pool A and Node Pool B. You want to prioritize the use of Node Pool A and schedule pods to Node Pool B only if the computing resources of Node Pool A are insufficient. In scale-in activities, you want to delete pods from the nodes in Node Pool B first. In this example, Node Pool A contains the following nodes: cn-beijing.10.0.3.137 and cn-beijing.10.0.3.138. Node Pool B contains the following nodes: cn-beijing.10.0.6.47 and cn-beijing.10.0.6.46. Each of these nodes has two vCPUs and 4 GB of memory. Perform the following steps to configure priority-based resource scheduling:

  1. Create a ResourcePolicy with the following template: 
    apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx # You must specify the label of the pods to which you want to apply the ResourcePolicy. 
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      alibabacloud.com/nodepool-id: np7ec79f2235954e879de07b780058****
  - resource: ecs
    nodeSelector:
      alibabacloud.com/nodepool-id: npab2df797738644e3a7b7cbf532bb****
    Note To view the ID of a node pool, choose Nodes > Node Pools on the details page of a cluster in the ACK console. For more information, see Manage node pools.
  2. Use the following template to create a Deployment that provisions two pods: 
    apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy. 
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2
  3. Deploy an NGINX application and query the pods.
    1. Run the following command to deploy an NGINX application: 
      kubectl apply -f nginx.yaml

      Expected output:

      deployment.apps/nginx created
    2. Run the following command to query the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          17s   172.29.112.216   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-k****   1/1     Running   0          17s   172.29.113.24    cn-beijing.10.0.3.138   <none>           <none>
      The output shows that the two pods are scheduled to Node Pool A.
  4. Scale out pods for the NGINX application.
    1. Run the following command to increase the number of pods to four: 
      kubectl scale deployment nginx --replicas 4

      Expected output:

      deployment.apps/nginx scaled
    2. Run the following command to query the status of the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS    RESTARTS   AGE    IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          101s   172.29.112.216   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-k****   1/1     Running   0          101s   172.29.113.24    cn-beijing.10.0.3.138   <none>           <none>
nginx-9cdf7bbf9-m****   1/1     Running   0          18s    172.29.113.156   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-x****   1/1     Running   0          18s    172.29.113.89    cn-beijing.10.0.6.46    <none>           <none>

      The output shows that new pods are scheduled to Node Pool B because the computing resources in Node Pool A are insufficient.

  5. Scale in pods for the NGINX application.
    1. Run the following command to reduce the number of pods to two: 
      kubectl scale deployment nginx --replicas 2

      Expected output:

      deployment.apps/nginx scaled
    2. Run the following command to query the status of the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running       0          2m41s   172.29.112.216   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-k****   1/1     Running       0          2m41s   172.29.113.24    cn-beijing.10.0.3.138   <none>           <none>
nginx-9cdf7bbf9-m****   0/1     Terminating   0          78s     172.29.113.156   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-x****   0/1     Terminating   0          78s     172.29.113.89    cn-beijing.10.0.6.46    <none>           <none>

      The output shows that pods that run on the nodes in Node Pool B are deleted.

Example 2: Schedule pods to ECS instances and elastic container instances

You want to schedule the pods of a Deployment to multiple types of resources, such as subscription Elastic Compute Service (ECS) instances, pay-as-you-go ECS instances, and elastic container instances. To reduce the resource cost, you want to schedule pods to resources based on the following priorities: subscription ECS instances > pay-as-you-go ECS instances > elastic container instances. In scale-in activities, you want to delete pods from these resources based on the following sequence: elastic container instances, pay-as-you-go ECS instances, and subscription ECS instances. In this example, each of the ECS instances has two vCPUs and 4 GB of memory. Perform the following steps to configure priority-based resource scheduling:

  1. Run the following command to add labels that indicate different billing methods to the nodes. You can also use node pools to automatically add labels to the nodes. 
    kubectl label node cn-beijing.10.0.3.137 paidtype=subscription
kubectl label node cn-beijing.10.0.3.138 paidtype=subscription
kubectl label node cn-beijing.10.0.6.46 paidtype=pay-as-you-go
kubectl label node cn-beijing.10.0.6.47 paidtype=pay-as-you-go
  2. Create a ResourcePolicy with the following template: 
    apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx # You must specify the label of the pods to which you want to apply the ResourcePolicy. 
  strategy: prefer
  units:
  - resource: ecs
    nodeSelector:
      paidtype: subscription
  - resource: ecs
    nodeSelector:
      paidtype: pay-as-you-go
  - resource: eci
  3. Use the following template to create a Deployment that provisions two pods: 
    apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy. 
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2
  4. Deploy an NGINX application and query the pods.
    1. Run the following command to deploy an NGINX application: 
      kubectl apply -f nginx.yaml

      Expected output:

      deployment.apps/nginx created
    2. Run the following command to query the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          66s   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          66s   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

      The output shows that two pods are scheduled to nodes that have the paidtype=subscription label.

  5. Scale out pods for the NGINX application.
    1. Run the following command to increase the number of pods to four: 
      kubectl scale deployment nginx --replicas 4

      Expected output:

      deployment.apps/nginx scaled
    2. Run the following command to query the status of the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running   0          16s     172.29.113.155   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running   0          3m48s   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running   0          16s     172.29.113.88    cn-beijing.10.0.6.46    <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          3m48s   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

      The output shows that new pods are scheduled to nodes that have the paidtype=pay-as-you-go label because nodes that have the paidtype=subscription label are insufficient.

    3. Run the following command to increase the number of pods to six: 
      kubectl scale deployment nginx --replicas 6

      Expected output:

      deployment.apps/nginx scaled
    4. Run the following command to query the status of the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE                           NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running   0          3m10s   172.29.113.155   cn-beijing.10.0.6.47           <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running   0          6m42s   172.29.112.215   cn-beijing.10.0.3.137          <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running   0          3m10s   172.29.113.88    cn-beijing.10.0.6.46           <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          6m42s   172.29.113.23    cn-beijing.10.0.3.138          <none>           <none>
nginx-9cdf7bbf9-s****   1/1     Running   0          36s     10.0.6.68        virtual-kubelet-cn-beijing-j   <none>           <none>
nginx-9cdf7bbf9-v****   1/1     Running   0          36s     10.0.6.67        virtual-kubelet-cn-beijing-j   <none>           <none>

      The output shows that new pods are scheduled to elastic container instances because the ECS nodes are insufficient.

  6. Scale in pods for the NGINX application.
    1. Run the following command to reduce the number of pods to four: 
      kubectl scale deployment nginx --replicas 4

      Expected output:

      deployment.apps/nginx scaled
    2. Run the following command to query the status of the pod: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                           NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   1/1     Running       0          4m59s   172.29.113.155   cn-beijing.10.0.6.47           <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running       0          8m31s   172.29.112.215   cn-beijing.10.0.3.137          <none>           <none>
nginx-9cdf7bbf9-f****   1/1     Running       0          4m59s   172.29.113.88    cn-beijing.10.0.6.46           <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running       0          8m31s   172.29.113.23    cn-beijing.10.0.3.138          <none>           <none>
nginx-9cdf7bbf9-s****   1/1     Terminating   0          2m25s   10.0.6.68        virtual-kubelet-cn-beijing-j   <none>           <none>
nginx-9cdf7bbf9-v****   1/1     Terminating   0          2m25s   10.0.6.67        virtual-kubelet-cn-beijing-j   <none>           <none>

      The output shows that the pods that run on elastic containers instances are deleted.

    3. Run the following command to reduce the number of pods to two: 
      kubectl scale deployment nginx --replicas 2

      Expected output:

      deployment.apps/nginx scaled
    4. Run the following command to query the status of the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS        RESTARTS   AGE     IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-4****   0/1     Terminating   0          6m43s   172.29.113.155   cn-beijing.10.0.6.47    <none>           <none>
nginx-9cdf7bbf9-b****   1/1     Running       0          10m     172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-f****   0/1     Terminating   0          6m43s   172.29.113.88    cn-beijing.10.0.6.46    <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running       0          10m     172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

      The output shows that the pods on the nodes that have the paidtype=pay-as-you-go label are deleted.

    5. Run the following command to query the status of the pods: 
      kubectl get pods -o wide

      Expected output:

      NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                    NOMINATED NODE   READINESS GATES
nginx-9cdf7bbf9-b****   1/1     Running   0          11m   172.29.112.215   cn-beijing.10.0.3.137   <none>           <none>
nginx-9cdf7bbf9-r****   1/1     Running   0          11m   172.29.113.23    cn-beijing.10.0.3.138   <none>           <none>

      The output shows that pods run only on the nodes that have the paidtype=subscription label.