You can use the UnitedDeployment controller to customize the workload template and distribution policy in order to dynamically adjust the number of replicas on each subset. This topic describes how to install and use the UnitedDeployment controller in Container Service Kubernetes (ACK) clusters to implement auto scaling based on fine-grained update and deployment policies.
Introduction to UnitedDeployment
UnitedDeployment is a controller provided by Kruise. The UnitedDeployment controller can be used to create and manage multiple workloads through custom templates and subsets. Each subset is associated with a workload.
Limits
Subsets must be of the StatefulSet, Advanced StatefulSet, CloneSet, or Deployment type. For more information, see Use OpenKruise to deploy cloud-native applications.
Prerequisites
An ACK cluster that runs Kubernetes 1.16 or later is created. For more information, see Create an ACK managed cluster and Update an ACK cluster.
The ack-kruise component is installed. For more information, see Manage components.
A kubectl client is connected to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
The ACK Virtual Node component is installed if you use elastic container instances. For more information, see Manage components.
Scenario 1: Fine-grained deployment across zones
To improve the availability of applications, you usually need to deploy computing or storage resources across multiple zones. The following example shows how to use the UnitedDeployment controller to conduct cross-zone fine-grained scheduling.
Three nodes reside in different zones. A label that indicates the zone whether the node resides is added to each node. For example, the node=zone-a label is added to the node in Zone A, the node=zone-b label is added to the node in Zone B, and the node=zone-c label is added to the node in Zone C.
kubectl label node cn-beijing.10.80.20.131 node=zone-a node/cn-beijing.10.80.20.131 labeled #Add the node=zone-a label to node 10.80.20.131. kubectl label node cn-beijing.10.80.20.132 node=zone-b node/cn-beijing.10.80.20.132 labeled #Add the node=zone-a label to node 10.80.20.132. kubectl label node cn-beijing.10.80.20.133 node=zone-c node/cn-beijing.10.80.20.133 labeled #Add the node=zone-a label to node 10.80.20.133.Prepare a custom UnitedDeployment YAML file.
apiVersion: apps.kruise.io/v1alpha1 kind: UnitedDeployment metadata: name: sample-ud spec: replicas: 6 revisionHistoryLimit: 10 selector: matchLabels: app: sample template: # statefulSetTemplate or advancedStatefulSetTemplate or cloneSetTemplate or deploymentTemplate statefulSetTemplate: metadata: labels: app: sample spec: selector: matchLabels: app: sample template: metadata: labels: app: sample spec: containers: - image: nginx:alpine name: nginx topology: subsets: - name: subset-a nodeSelectorTerm: matchExpressions: - key: node operator: In values: - zone-a replicas: 1 - name: subset-b nodeSelectorTerm: matchExpressions: - key: node operator: In values: - zone-b replicas: 50% - name: subset-c nodeSelectorTerm: matchExpressions: - key: node operator: In values: - zone-c updateStrategy: manualUpdate: partitions: subset-a: 0 subset-b: 0 subset-c: 0 type: ManualYAML file description:
Define a custom workload template in the UnitedDeployment controller and define a sebset for each zone.
The workload is a StatefulSet. The workload template is defined below statefulSetTemplate. The UnitedDeployment controller creates a StatefulSet on each subset based on the template.
The subsets section defines a sebset for each zone. Pods of subset-a are deployed on the node whose label is node=zone-a. Pods of subset-b are deployed on the node whose label is node=zone-b. Pods of subset-c are deployed on the node whose label is node=zone-c.
Run the following command to deploy the UnitedDeployment controller:
kubectl apply -f test.yamlExpected output:
uniteddeployment.apps.kruise.io/sample-ud createdThe output indicates that the UnitedDeployment controller is deployed.
Run the following command to check whether the pods and StatefulSets are created:
kubectl get pod NAME READY STATUS RESTARTS AGE sample-ud-subset-a-cplwg-0 1/1 Running 0 6m5s sample-ud-subset-b-rj7kt-0 1/1 Running 0 6m4s sample-ud-subset-b-rj7kt-1 1/1 Running 0 5m49s sample-ud-subset-b-rj7kt-2 1/1 Running 0 5m43s sample-ud-subset-c-g6jvx-0 1/1 Running 0 6m5s sample-ud-subset-c-g6jvx-1 1/1 Running 0 5m51s kubectl get statefulset NAME READY AGE sample-ud-subset-a-cplwg 1/1 7m34s sample-ud-subset-b-rj7kt 3/3 7m34s sample-ud-subset-c-g6jvx 2/2 7m34sThe output indicates that the pods and StatefulSets are created.
Scenario 2: Use the UnitedDeployment controller to colocate applications on ECS instances and elastic container instances
To handle business spikes, you may need to ensure the resource supply of cluster nodes and control the costs at the same time. To address this issue, you can configure ACK to automatically deploy applications on elastic container instances when Elastic Compute Service (ECS) instances are out of stock.
Prepare a custom UnitedDeployment YAML file.
apiVersion: apps.kruise.io/v1alpha1 kind: UnitedDeployment metadata: name: ud-nginx spec: replicas: 6 revisionHistoryLimit: 10 selector: matchLabels: app: ud-nginx template: # statefulSetTemplate or advancedStatefulSetTemplate or cloneSetTemplate or deploymentTemplate deploymentTemplate: metadata: labels: app: ud-nginx spec: selector: matchLabels: app: ud-nginx template: metadata: labels: app: ud-nginx spec: containers: - image: nginx:alpine name: nginx topology: subsets: - name: subset-a nodeSelectorTerm: matchExpressions: - key: alibabacloud.com/nodepool-id operator: In values: - np92019eec42004d878fcdc990fcb9**** #Replace with the ID of Node Pool A. replicas: 1 - name: subset-b nodeSelectorTerm: matchExpressions: - key: alibabacloud.com/nodepool-id operator: In values: - np011de1f2de3d48bd8a92a015fc5c**** #Replace with the ID of Node Pool B. replicas: 1 - name: subset-c nodeSelectorTerm: matchExpressions: - key: node operator: In values: - zone-c replicas: 1 - name: subset-eci nodeSelectorTerm: matchExpressions: - key: type operator: In values: - virtual-kubelet tolerations: - key: virtual-kubelet.io/provider operator: Exists replicas: 3 updateStrategy: manualUpdate: partitions: subset-a: 0 subset-b: 0 subset-c: 0 type: ManualYAML file description:
subset-a, subset-b, and subset-c all use ECS instances. The number of replicas on each subset is specified and the replicas are deployed on ECS instances.
A nodeSelectorTerm and toleration are added to subset-d to deploy replicas on elastic container instances.
Run the following command to deploy the UnitedDeployment controller:
kubectl apply -f test.yamlExpected output:
uniteddeployment.apps.kruise.io/ud-nginx createdRun the following command to check whether pods are created:
kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ud-nginx-subset-a-7lbtd-5b5bd77549-5bw6l 1/1 Running 0 73s 192.168.0.126 cn-hangzhou.10.80.20.133 <none> <none> ud-nginx-subset-b-nvvfw-5c9bcd6766-lv6sp 1/1 Running 0 73s 192.168.10.239 cn-hangzhou.10.80.20.134 <none> <none> ud-nginx-subset-c-r2mkr-7675856649-vh7zr 1/1 Running 0 73s 192.168.0.127 cn-hangzhou.10.80.20.133 <none> <none> ud-nginx-subset-d-m78fd-7796b66fd8-7p52j 1/1 Running 0 73s 192.168.0.130 virtual-kubelet-cn-hangzhou-h <none> <none> ud-nginx-subset-d-m78fd-7796b66fd8-fd7f7 1/1 Running 0 73s 192.168.0.129 virtual-kubelet-cn-hangzhou-h <none> <none> ud-nginx-subset-d-m78fd-7796b66fd8-mn4qb 1/1 Running 0 73s 192.168.0.131 virtual-kubelet-cn-hangzhou-h <none> <none>The output indicates that pods are deployed by the UnitedDeployment controller as expected.
Scenario 3: Use the UnitedDeployment controller with the HPA
To use the UnitedDeployment controller with the Horizontal Pod Autoscaler (HPA), make sure that your Kruise version is 1.5.0 or later.
To control the scheduling of replicas that exceed the specified number, use the maxReplicas field in the UnitedDeployment controller. You can schedule the excessive replicas to low-priority resources. For example, you can configure HPA to use subscription ECS instances when the number of replicas is lower than 10, use preemptible instances when the number of replicas exceeds 10 but does not exceed 20, and use elastic container instances when the number of replicas exceeds 20. You can configure the HPA to control the horizontal scaling of the UnitedDeployment. Pods are scaled based on the specified priorities of resources.
Prepare a file named test.yaml.
apiVersion: apps.kruise.io/v1alpha1 kind: UnitedDeployment metadata: name: ud-nginx spec: replicas: 6 selector: matchLabels: app: sample template: deploymentTemplate: metadata: labels: app: sample spec: selector: matchLabels: app: sample template: metadata: labels: app: sample spec: containers: - image: nginx:alpine name: nginx resources: requests: cpu: "500m" topology: subsets: - name: ecs maxReplicas: 4 - name: eci maxReplicas: null --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: united-deployment-hpa spec: scaleTargetRef: apiVersion: apps.kruise.io/v1alpha1 kind: UnitedDeployment name: ud-nginx #Replace with the name of the UnitedDeployment controller. minReplicas: 4 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50YAML file description:
Schedule the fourth pod to an ECS instance, which means that the first four replicas are scheduled to ECS instances.
Schedule the fifth pod to an elastic container instance.
Scale the number of replicas from five to four. The pod on the elastic container instance is preferably deleted.
Scale the number of replicas to less than four pods. Pods on ECS instances are deleted.
When you use the HPA with the UnitedDeployment controller, set the scaleTargetRef field of the HPA to UnitedDeployment and its name.
Run the following command to deploy the UnitedDeployment controller:
kubectl apply -f test.yamlExpected output:
horizontalpodautoscaler.autoscaling/united-deployment-hpa createdThe output indicates that the UnitedDeployment controller is deployed.
Run the following command to query the status of the pods:
kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ud-nginx-eci-dxfbz-864bdb77b-2d4t9 1/1 Running 0 3m9s 192.168.5.129 cn-hangzhou.192.168.5.120 <none> <none> ud-nginx-eci-dxfbz-864bdb77b-zppfh 1/1 Running 0 3m9s 192.168.8.11 cn-hangzhou.192.168.8.251 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-5mlgh 1/1 Running 0 3m9s 192.168.8.4 cn-hangzhou.192.168.8.251 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-6bdkz 1/1 Running 0 3m9s 192.168.6.145 cn-hangzhou.192.168.6.32 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-dnsfl 1/1 Running 0 3m9s 192.168.6.150 cn-hangzhou.192.168.6.20 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-mrzwc 1/1 Running 0 3m9s 192.168.5.128 cn-hangzhou.192.168.5.120 <none> <none>The output indicates that the number of replicas of the Deployment is dynamically adjusted based on the scheduling policy.
Trigger the HPA to perform a scale-in activity and run the following command to query the status of pods:
kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ud-nginx-ecs-5lm7r-868c4ccd5d-5mlgh 1/1 Running 0 8m14s 192.168.8.4 cn-hangzhou.192.168.8.251 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-6bdkz 1/1 Running 0 8m14s 192.168.6.145 cn-hangzhou.192.168.6.32 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-dnsfl 1/1 Running 0 8m14s 192.168.6.150 cn-hangzhou.192.168.6.20 <none> <none> ud-nginx-ecs-5lm7r-868c4ccd5d-mrzwc 1/1 Running 0 8m14s 192.168.5.128 cn-hangzhou.192.168.5.120 <none> <none>The output indicates that the number of replicas is scaled from six to four. Pods on elastic container instances are preferably deleted.