Container Service for Kubernetes (ACK) allows you to configure priority-based resource
scheduling. A ResourcePolicy specifies the priorities of nodes in descending order
for pod scheduling. When the system deploys or scales out pods for an application,
pods are scheduled to nodes based on the priorities of the nodes that are listed in
the ResourcePolicy. When the system scales in pods for an application, pods are removed
from nodes based on the priorities of the nodes in ascending order. This topic describes
how to configure priority-based resource scheduling.
Limits
- Priority-based resource scheduling is mutually exclusive with the pod deletion cost
feature. For more information about the pod deletion cost feature, see Pod deletion cost.
- You cannot use priority-based resource scheduling and Elastic Container Instance-based
scheduling at the same time. For more information, see Use ECI elastic scheduling.
- After you modify a ResourcePolicy, only the pods that are scheduled based on the modified
ResourcePolicy are deleted based on the modified ResourcePolicy. The pods that are
scheduled based on the original ResourcePolicy are deleted based on the original ResourcePolicy.
How to configure priority-based resource scheduling
Create a ResourcePolicy with the following template:
apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
name: xxx
namespace: xxx
spec:
selector:
key1: value1
strategy: prefer
units:
- resource: ecs
nodeSelector:
key2: value2
- resource: ecs
nodeSelector:
key3: value3
- resource: eci
selector: the selector that is used to select pods in a namespace. The ResourcePolicy is applied
to the selected pods. In this example, pods that have the key1=value1 label are selected.
strategy: the scheduling policy. Set the value to prefer.
units: the schedulable units. In a scale-out activity, pods are scheduled to nodes based
on the priorities of the nodes that are listed under units in descending order. In a scale-in activity, pods are deleted from the nodes based
on the priorities of the nodes in ascending order.
resource: the type of resource for pod scheduling. Valid values: eci and ecs.
nodeSelector: the selector that is used to select nodes with a specified node label. This parameter takes effect only if the resource parameter is set to ecs.
Examples
Example 1: Schedule pods to specified node poolsYou want to schedule the pods of a Deployment to specific node pools, for example,
Node Pool A and Node Pool B. You want to prioritize the use of Node Pool A and schedule
pods to Node Pool B only if the computing resources of Node Pool A are insufficient.
In scale-in activities, you want to delete pods from the nodes in Node Pool B first.
In this example, Node Pool A contains the following nodes: cn-beijing.10.0.3.137 and cn-beijing.10.0.3.138. Node Pool B contains the following nodes: cn-beijing.10.0.6.47 and cn-beijing.10.0.6.46. Each of these nodes has two vCPUs and 4 GB of memory. Perform the following steps
to configure priority-based resource scheduling:
- Create a ResourcePolicy with the following template:
apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
name: nginx
namespace: default
spec:
selector:
app: nginx # You must specify the label of the pods to which you want to apply the ResourcePolicy.
strategy: prefer
units:
- resource: ecs
nodeSelector:
alibabacloud.com/nodepool-id: np7ec79f2235954e879de07b780058****
- resource: ecs
nodeSelector:
alibabacloud.com/nodepool-id: npab2df797738644e3a7b7cbf532bb****
Note To view the ID of a node pool, choose
Nodes > Node Pools on the details page of a cluster in the ACK console. For more information, see
Manage node pools.
- Use the following template to create a Deployment that provisions two pods:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy.
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
cpu: 2
requests:
cpu: 2
- Deploy an NGINX application and query the pods.
- Run the following command to deploy an NGINX application:
kubectl apply -f nginx.yaml
Expected output:
deployment.apps/nginx created
- Run the following command to query the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-b**** 1/1 Running 0 17s 172.29.112.216 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-k**** 1/1 Running 0 17s 172.29.113.24 cn-beijing.10.0.3.138 <none> <none>
The output shows that the two pods are scheduled to Node Pool A.
- Scale out pods for the NGINX application.
- Run the following command to increase the number of pods to four:
kubectl scale deployment nginx --replicas 4
Expected output:
deployment.apps/nginx scaled
- Run the following command to query the status of the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-b**** 1/1 Running 0 101s 172.29.112.216 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-k**** 1/1 Running 0 101s 172.29.113.24 cn-beijing.10.0.3.138 <none> <none>
nginx-9cdf7bbf9-m**** 1/1 Running 0 18s 172.29.113.156 cn-beijing.10.0.6.47 <none> <none>
nginx-9cdf7bbf9-x**** 1/1 Running 0 18s 172.29.113.89 cn-beijing.10.0.6.46 <none> <none>
The output shows that new pods are scheduled to Node Pool B because the computing
resources in Node Pool A are insufficient.
- Scale in pods for the NGINX application.
- Run the following command to reduce the number of pods to two:
kubectl scale deployment nginx --replicas 2
Expected output:
deployment.apps/nginx scaled
- Run the following command to query the status of the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-b**** 1/1 Running 0 2m41s 172.29.112.216 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-k**** 1/1 Running 0 2m41s 172.29.113.24 cn-beijing.10.0.3.138 <none> <none>
nginx-9cdf7bbf9-m**** 0/1 Terminating 0 78s 172.29.113.156 cn-beijing.10.0.6.47 <none> <none>
nginx-9cdf7bbf9-x**** 0/1 Terminating 0 78s 172.29.113.89 cn-beijing.10.0.6.46 <none> <none>
The output shows that pods that run on the nodes in Node Pool B are deleted.
Example 2: Schedule pods to ECS instances and elastic container instancesYou want to schedule the pods of a Deployment to multiple types of resources, such
as subscription Elastic Compute Service (ECS) instances, pay-as-you-go ECS instances,
and elastic container instances. To reduce the resource cost, you want to schedule
pods to resources based on the following priorities: subscription ECS instances >
pay-as-you-go ECS instances > elastic container instances. In scale-in activities,
you want to delete pods from these resources based on the following sequence: elastic
container instances, pay-as-you-go ECS instances, and subscription ECS instances.
In this example, each of the ECS instances has two vCPUs and 4 GB of memory. Perform
the following steps to configure priority-based resource scheduling:
- Run the following command to add
labels that indicate different billing methods to the nodes. You can also use node pools
to automatically add labels to the nodes. kubectl label node cn-beijing.10.0.3.137 paidtype=subscription
kubectl label node cn-beijing.10.0.3.138 paidtype=subscription
kubectl label node cn-beijing.10.0.6.46 paidtype=pay-as-you-go
kubectl label node cn-beijing.10.0.6.47 paidtype=pay-as-you-go
- Create a ResourcePolicy with the following template:
apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
name: nginx
namespace: default
spec:
selector:
app: nginx # You must specify the label of the pods to which you want to apply the ResourcePolicy.
strategy: prefer
units:
- resource: ecs
nodeSelector:
paidtype: subscription
- resource: ecs
nodeSelector:
paidtype: pay-as-you-go
- resource: eci
- Use the following template to create a Deployment that provisions two pods:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx # The pod label must be the same as the one that you specified for the selector in the ResourcePolicy.
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
cpu: 2
requests:
cpu: 2
- Deploy an NGINX application and query the pods.
- Run the following command to deploy an NGINX application:
kubectl apply -f nginx.yaml
Expected output:
deployment.apps/nginx created
- Run the following command to query the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-b**** 1/1 Running 0 66s 172.29.112.215 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-r**** 1/1 Running 0 66s 172.29.113.23 cn-beijing.10.0.3.138 <none> <none>
The output shows that two pods are scheduled to nodes that have the paidtype=subscription label.
- Scale out pods for the NGINX application.
- Run the following command to increase the number of pods to four:
kubectl scale deployment nginx --replicas 4
Expected output:
deployment.apps/nginx scaled
- Run the following command to query the status of the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-4**** 1/1 Running 0 16s 172.29.113.155 cn-beijing.10.0.6.47 <none> <none>
nginx-9cdf7bbf9-b**** 1/1 Running 0 3m48s 172.29.112.215 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-f**** 1/1 Running 0 16s 172.29.113.88 cn-beijing.10.0.6.46 <none> <none>
nginx-9cdf7bbf9-r**** 1/1 Running 0 3m48s 172.29.113.23 cn-beijing.10.0.3.138 <none> <none>
The output shows that new pods are scheduled to nodes that have the paidtype=pay-as-you-go label because nodes that have the paidtype=subscription label are insufficient.
- Run the following command to increase the number of pods to six:
kubectl scale deployment nginx --replicas 6
Expected output:
deployment.apps/nginx scaled
- Run the following command to query the status of the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-4**** 1/1 Running 0 3m10s 172.29.113.155 cn-beijing.10.0.6.47 <none> <none>
nginx-9cdf7bbf9-b**** 1/1 Running 0 6m42s 172.29.112.215 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-f**** 1/1 Running 0 3m10s 172.29.113.88 cn-beijing.10.0.6.46 <none> <none>
nginx-9cdf7bbf9-r**** 1/1 Running 0 6m42s 172.29.113.23 cn-beijing.10.0.3.138 <none> <none>
nginx-9cdf7bbf9-s**** 1/1 Running 0 36s 10.0.6.68 virtual-kubelet-cn-beijing-j <none> <none>
nginx-9cdf7bbf9-v**** 1/1 Running 0 36s 10.0.6.67 virtual-kubelet-cn-beijing-j <none> <none>
The output shows that new pods are scheduled to elastic container instances because
the ECS nodes are insufficient.
- Scale in pods for the NGINX application.
- Run the following command to reduce the number of pods to four:
kubectl scale deployment nginx --replicas 4
Expected output:
deployment.apps/nginx scaled
- Run the following command to query the status of the pod:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-4**** 1/1 Running 0 4m59s 172.29.113.155 cn-beijing.10.0.6.47 <none> <none>
nginx-9cdf7bbf9-b**** 1/1 Running 0 8m31s 172.29.112.215 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-f**** 1/1 Running 0 4m59s 172.29.113.88 cn-beijing.10.0.6.46 <none> <none>
nginx-9cdf7bbf9-r**** 1/1 Running 0 8m31s 172.29.113.23 cn-beijing.10.0.3.138 <none> <none>
nginx-9cdf7bbf9-s**** 1/1 Terminating 0 2m25s 10.0.6.68 virtual-kubelet-cn-beijing-j <none> <none>
nginx-9cdf7bbf9-v**** 1/1 Terminating 0 2m25s 10.0.6.67 virtual-kubelet-cn-beijing-j <none> <none>
The output shows that the pods that run on elastic containers instances are deleted.
- Run the following command to reduce the number of pods to two:
kubectl scale deployment nginx --replicas 2
Expected output:
deployment.apps/nginx scaled
- Run the following command to query the status of the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-4**** 0/1 Terminating 0 6m43s 172.29.113.155 cn-beijing.10.0.6.47 <none> <none>
nginx-9cdf7bbf9-b**** 1/1 Running 0 10m 172.29.112.215 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-f**** 0/1 Terminating 0 6m43s 172.29.113.88 cn-beijing.10.0.6.46 <none> <none>
nginx-9cdf7bbf9-r**** 1/1 Running 0 10m 172.29.113.23 cn-beijing.10.0.3.138 <none> <none>
The output shows that the pods on the nodes that have the paidtype=pay-as-you-go label are deleted.
- Run the following command to query the status of the pods:
kubectl get pods -o wide
Expected output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-9cdf7bbf9-b**** 1/1 Running 0 11m 172.29.112.215 cn-beijing.10.0.3.137 <none> <none>
nginx-9cdf7bbf9-r**** 1/1 Running 0 11m 172.29.113.23 cn-beijing.10.0.3.138 <none> <none>
The output shows that pods run only on the nodes that have the paidtype=subscription label.