Container Compute Service:In-place vertical scaling of containers

Comparison between AVPA and open source VPA

Use scenarios

• Gaming businesses: hot upgrade and downgrade for periodically fluctuating CPU workloads.

• Online businesses: vertically scale hotspot pods.

Limits

• AVPA cannot be used together with open source VPA or HPA.

• Vertical scaling based on CPU metrics is supported. If the CPU specifications no longer meet the ACS requirements after scaling, the scaling request is rejected. For example, the system cannot upgrade from 2vCpu2Gi to 3vCpu2Gi. In addition, you must configure resource requests for the containers to be scaled.

• Vertical scaling automatically adjusts the CPU requests and limits of pods. In ACK and ACK serverless clusters, vertical scaling may be constrained due to the limited node resources.

• In ACK and ACK serverless clusters, the version of ack-virtual-node must be at least v2.14.0.

• AVPA does not create profiles and takes effect only on existing pods. Therefore, new pods are created based on the original resource specifications defined in the workload.

Step 1: Enable the in-place scaling feature gate

1. Log on to the ACS console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left navigation pane, click Add-ons.

3. In the Core Components section, find the Kube API Server card and click Configuration. In the featureGates field, enter InPlacePodVerticalScaling=true to enable the in-place scaling feature gate.

   Note

   During the configuration process, the status on the Kube API Server card changes to Executing. When the status returns to Installed, the in-place scaling feature gate is enabled.

Step 2: Install AVPA

In the left-side navigation pane, choose Applications > Helm. Search for and install the ack-advanced-vertical-pod-autoscaler component. For more information, see Use Helm to manage applications in ACK.

Step 3: Deploy the workload and create a local connection

1. Create a YAML file. You can also create a YAML file named shadow-hello-avpa.yaml for stress testing.

   Create a YAML file named hello-avpa.yaml. apiVersion: apps/v1 kind: Deployment metadata: name: hello-avpa namespace: default spec: replicas: 1 selector: matchLabels: name: hello-avpa template: metadata: annotations: scaling.alibabacloud.com/enable-inplace-resource-resize: 'true' labels: name: hello-avpa vpa: enabled spec: containers: - image: 'registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/simulation-resource-consumer:1.13' name: hello-avpa resources: limits: cpu: '2' memory: '4Gi' requests: cpu: '2' memory: '4Gi' --- apiVersion: v1 kind: Service metadata: name: hello-avpa-svc namespace: default spec: ports: - port: 80 protocol: TCP targetPort: 8080 selector: name: hello-avpa type: ClusterIP

   (Optional) Create a YAML file named shadow-hello-avpa.yaml. apiVersion: apps/v1 kind: Deployment metadata: name: shadow-hello-avpa namespace: default spec: replicas: 1 selector: matchLabels: name: shadow-hello-avpa template: metadata: annotations: scaling.alibabacloud.com/enable-inplace-resource-resize: 'true' labels: vpa: enabled name: shadow-hello-avpa spec: containers: - image: 'registry.cn-hangzhou.aliyuncs.com/acs-demo-ns/simulation-resource-consumer:1.13' name: shadow-hello-avpa resources: limits: cpu: '2' memory: '4Gi' requests: cpu: '2' memory: '4Gi' --- apiVersion: v1 kind: Service metadata: name: shadow-avpa-svc namespace: default spec: ports: - port: 80 protocol: TCP targetPort: 8080 selector: name: shadow-hello-avpa type: ClusterIP

2. Deploy the workload.

   kubectl apply -f hello-avpa.yaml
   kubectl apply -f shadow-hello-avpa.yaml

3. Run kubectl port-forward to create a local connection.

   Important

   Port forwarding set up by using kubectl port-forward is not reliable, secure, or extensible in production environments. It is only for development and debugging. Do not use this command to set up port forwarding in production environments. For more information about networking solutions used for production in ACK clusters, see Ingress management.

   kubectl port-forward svc/hello-avpa-svc -n default 28080:80
   kubectl port-forward svc/shadow-avpa-svc -n default 28081:80

Step 4: Configure AVPA

You can create an AdvancedVerticalPodAutoscaler resource to configure elastic scaling.

1. Create a YAML file named avpa.yaml.

   TargetRef mode

   The following scaling configuration is based on the TargetRef mode of AVPA to adjust the resources of the pods created by the Deployment named hello-avpa in the default namespace based on the CPU utilization.

   apiVersion: autoscaling.alibabacloud.com/v1beta1
   kind: AdvancedVerticalPodAutoscaler
   metadata:
     name: hello-avpa
     namespace: default
   spec:
     metrics:
       - containerResource:
           container: hello-avpa
           name: cpu
           target:
             averageUtilization: 30
             type: Utilization
         type: ContainerResource
         watermark: low
       - containerResource:
           container: hello-avpa
           name: cpu
           target:
             averageUtilization: 50
             type: Utilization
         type: ContainerResource
         watermark: high
     scaleResourceLimit:
       maximum:
         cpu: '4'
       minimum:
         cpu: '1'
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: hello-avpa

   The following table describes some of the parameters.

   Parameter	Required	Description
   scaleTargetRef	Yes	The target workload. Currently, AdvancedStatefulSet and CloneSet for Kubernetes-native Deployments, StatefulSets, Jobs, and OpenKruise are supported.
   metrics.containerResource	Yes	Specify the type and utilization threshold of the resource whose metrics are collected: container: the name of the container from which metrics are collected. name: the name of the metric to collect. Only CPU metrics are supported. target: the threshold information. type: set to Utilization. averageUtilization: the average utilization threshold. Note For an ACK serverless pod, metric calculation prioritizes the container's resource limits. If resource limits are not set, the calculation is based on the resource requests. Metric calculation is not supported for pods that have neither resource limits nor requests defined.
   metrics.watermark	Yes	Specify the type of threshold. Valid values: low: the low threshold. When the metric value drops below the threshold, pods are scaled in. high: the high threshold. When the metric value exceeds the threshold, pods are scaled out.
   metrics.type	Yes	Specify the metric collection granularity. You can aggregate metrics by container or pod. The default is ContainerResource. Note Currently, you can collect and aggregate metrics by container.
   scaleResourceLimit.minimum	No	The lower limit of vertical scaling (for cpu only). cpu: The default is 250m (millicores).
   scaleResourceLimit.maximum	No	The upper limit of vertical scaling (for cpu only). cpu: The default is 64 (cores).

   Selector mode

   The following Selector mode is introduced in AVPA 0.2.0, which provides a more flexible selector to simplify the pod configuration.

   apiVersion: autoscaling.alibabacloud.com/v1beta1
   kind: AdvancedVerticalPodAutoscaler
   metadata:
     name: hello-avpa
     namespace: default
   spec:
     metrics:
       - containerResource:
           container: "*"
           name: cpu
           target:
             averageUtilization: 30
             type: Utilization
         type: ContainerResource
         watermark: low
       - containerResource:
           container: "*"
           name: cpu
           target:
             averageUtilization: 50
             type: Utilization
         type: ContainerResource
         watermark: high
     scaleResourceLimit:
       maximum:
         cpu: '4'
       minimum:
         cpu: '1'
     # You can modify the following sample selector configuration on demand.
     selector:
       matchLabels:
         vpa: enabled
       matchExpressions:
       # A label that applies to all ACS pods.
       - key: alibabacloud.com/compute-class 
         operator: Exists
       - key: name 
         operator: In
         values: 
         - hello-avpa
       # A reserved switch to disable AVPA for certain pods.
       - key: alibabacloud.com/disable-avpa 
         operator: DoesNotExist

   Unlike scaleTargetRef that takes effect only on one workload, the selector can select any pods to meet scaling requirements in different scenarios.

   Parameter	Required	Description
   selector	No	The selector and scaleTargetRef are mutually exclusive. The selector can simplify the AVPA configuration to centrally manage pods created by a type of workload. This saves you the need to configure AVPA for each workload.
   metrics.containerResource	Yes	Specify the type and utilization threshold of the resource whose metrics are collected: container: the name of the container from which metrics are collected. Note You can enter a wildcard character (*) to match all containers. If a container matches both the wildcard expression and a specific container match rule, the container match rule prevails. name: the name of the metric to collect. Only CPU metrics are supported. target: the threshold information. type: set to Utilization. averageUtilization: the average utilization threshold.
   metrics.watermark	Yes	Specify the type of threshold. Valid values: low: the low threshold. When the metric value drops below the threshold, pods are scaled in. high: the high threshold. When the metric value exceeds the threshold, pods are scaled out.
   metrics.type	Yes	Specify the metric collection granularity. You can aggregate metrics by container or pod. The default is ContainerResource. Note Currently, you can collect and aggregate metrics by container.
   scaleResourceLimit.minimum	No	The lower limit of vertical scaling (for cpu only). cpu: The default is 250m (millicores).
   scaleResourceLimit.maximum	No	The upper limit of vertical scaling (for cpu only). cpu: The default is 64 (cores).

   Container resource utilization calculation rules:

   For serverless pods, such as ACK serverless pods, resource specifications are provisioned based on resource limits. Therefore, utilization is first calculated based on the limit: $$cpuUtilization=\frac{cpuMetricValue}{cpuLimitValue}$$. If a limit is not set, the calculation is based on the requested resources: $$cpuUtilization=\frac{cpuMetricValue}{cpuRequestValue}$$. Calculating container metrics is not supported if neither the request nor the limit is set.

   For example, if a container declaration causes ACS to create a pod with 4 vCPUs and 4 GiB, and PodMetrics reports a CPU usage of 1000m, the CPU utilization is $$1/4=25\%$$

   ...
   spec:
     containers:
       - resources:
           limits:
             cpu: '4'
             memory: '4Gi'
           requests:
             cpu: '1'
             memory: '4Gi'
   

2. You can create an AVPA configuration with more fine-grained control over the scaling behavior, such as the scaling step.

   apiVersion: autoscaling.alibabacloud.com/v1beta1
   kind: AdvancedVerticalPodAutoscaler
   metadata:
     name: hello-avpa
     namespace: default
   spec:
     behavior:
       parallelism: 1
       stabilizationWindowSeconds: 600
       scaleDown: #Defines the resource step and policy for scaling down. A gradual decrease is generally recommended, but each scale-down must be large enough to exceed the normalization granularity.
         policies:
         - type: CpuPercent
           value: 10%
           periodSeconds: 60
         - type: Cpus
           value: 500m
           periodSeconds: 60
         selectPolicy: Max
       scaleUp: #Defines the resource step and policy for scaling up. A relatively steep increase is generally recommended to meet load growth with fewer scaling events.
         policies:
         - type: CpuPercent
           value: 60%
           periodSeconds: 60
         - type: Cpus
           value: 500m
           periodSeconds: 60
         selectPolicy: Max
     metricObserveWindowSeconds: 600
     metrics:
       - containerResource:
           container: hello-avpa
           name: cpu
           target:
             averageUtilization: 30
             type: Utilization
         type: ContainerResource
         watermark: low
       - containerResource:
           container: hello-avpa
           name: cpu
           target:
             averageUtilization: 50
             type: Utilization
         type: ContainerResource
         watermark: high
     scaleResourceLimit:
       maximum:
         cpu: '4'
       minimum:
         cpu: '1'
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: hello-avpa

   In the spec.behavior parameter, scaleDown and scaleUp describe the resource policies for scaling down and scaling up, respectively. The sub-parameters that they contain have the same semantics.

   Parameter	Required	Description
   policies[].type	No	Cpus: The absolute value of the CPU scaling step. CpuPercent: Specifies the CPU scaling step as a percentage of the current resource specifications.
   policies[].value	No	Use this configuration when type is set to Cpus. The value can be set to 250m, 1, etc. Use this configuration when type is set to CpuPercent. The value can be a percentage, such as 5%.
   selectPolicy	No	The calculation policy. Valid values are: Max: Uses the maximum value as the target step size when multiple policies are set. Min: When multiple policies are set, the minimum value is used as the target step size.
   metricObserveWindowSeconds	No	The time window for data collection. When the collected metrics span this time window, the system calculates the load and determines whether to perform scaling. The unit is seconds. The default value is 600 seconds, and the minimum value is 300 seconds.
   behavior.parallelism	No	The parallelism for scaling, which is the number of Pods scaled simultaneously, with a default value of 1.
   behavior.stabilizationWindowSeconds	No	The cooldown period for scaling is specified in seconds, with a default value of 600 and a minimum value of 300. You must also ensure that the cooldown period is not shorter than the data collection window time metricObserveWindowSeconds.

   When scaling is required, AVPA uses a default policy to calculate the target resource specifications. It calculates the scaling step for scaling up or down by using the average of highWaterMark and lowWaterMark as the target load level. For example, for a 1 vCPU container, if the high and low thresholds are set to 60 and 40 respectively and the current utilization is 100%, the target utilization is 50%. The expected scale-up step is 1 vCPU, scaling the container up to 2 vCPU.

   Based on the configuration in the preceding YAML file, the system scales up in increments of 600m, which is the maximum value of $$1000m\ast 60\%$$ and 500m.

   Note

   Recommended configuration: Scale up rapidly and scale down gradually. This allows you to quickly meet increasing load demands while maintaining stability by slowly releasing resources.

   scaleUp: Scales up in relatively large increments to meet workload requirements with fewer scaling activities. For example, use type: Cpus and value: 1.

   scaleDown: Gradually scales down resources in small increments as the load decreases to improve stability. For example, type: Cpus, value: 250m.

3. Deploy the YAML file.

   kubectl apply -f avpa.yaml

Step 5: Perform stress testing

Send requests to the backend pod through the traffic Ingress to generate CPU loads. Observe the workload monitoring data before and after AVPA is enabled based on the same resource specifications and loads.

1. Send requests to generate loads.

   The current test pod has a CPU resource specification of 2 vCPU. Initially, the load is increased to about 50%, consuming 1000 millicores, and is maintained for 2000 s. After that, the load is increased by an additional 100 millicores every 60 s, eventually reaching 4000 millicores after 30 minutes.

   # Increase the loads. The upper limit of each command is 1,000 millicores.
   curl --data "millicores=1000&durationSec=2000" http://localhost:28080/ConsumeCPU
   curl --data "millicores=1000&durationSec=2000" http://localhost:28081/ConsumeCPU
   # Continuously increase the loads.
   for i in {1..30}
   do
     sleep 60
     curl --data "millicores=100&durationSec=2000" http://localhost:28080/ConsumeCPU
     curl --data "millicores=100&durationSec=2000" http://localhost:28081/ConsumeCPU
   done

2. Monitor the metric data.

   In the left-side navigation pane, click Operations > Prometheus Monitoring > Prometheus Monitoring. On the Application Monitoring > Deployment Details > Deployment Details tab, view the monitoring data.

   1. The CPU loads continuously increase and then reach the threshold. After 10 minutes, CPU resources are scaled out to 2 vCPUs. After 30 minutes, CPU resources are scaled out to 4 vCPUs. After the scale-out, the CPU loads significantly drop.

      

   2. Monitor the loads of the shadow workload. The CPU resources are exhausted within a short period of time.

      

3. View pod events.

   The pod events show configuration changes in the YAML template during scaling.

   On the Events tab, select the Pod sub-tab. You can see events related to vertical scaling, including StartingVerticalScalingPod, InplaceResizing, InplaceResizedFinished, and FinishedVerticalScalingPod. The events show the CPU request scaling up from 2 to 3,200m, 3,500m, and finally 4. The entire in-place adjustment process takes about 34 to 36 seconds.

4. Simulate scale-in activities due to low loads.

   After the CPU loads drop below the threshold, CPU resources are scaled in.

   • After the loads reach the scale-in threshold, CPU resources are scaled in at the specified step to ensure stability.

   • The container resources will be scaled to the minimum specification 1 vCPU claimed in the AVPA configuration if the loads consistently remain at a low level.

   

(Optional) Step 6: Delete resources

1. Delete the workloads, Service, and AVPA resource.

   kubectl delete -f hello-avpa.yaml
   kubectl delete -f shadow-hello-avpa.yaml

2. In the left-side navigation pane, choose Applications > Helm. Find the row containing ack-advanced-vertical-pod-autoscaler and click Delete in the Actions column.

How do I view the status of AVPA?

Use kubectl to query the real-time status of AVPA.

# In this example, the cluster connection information is stored in the ~/.kube/acs-test file.
# export KUBECONFIG=~/.kube/acs-test 
kubectl get avpa -n [namespace] [-oyaml]

Expected output:

$ kubectl get avpa
NAME         TARGETTYPE   TARGETNAME   REPLICAS   UPDATING   WAITING   LASTSCALED   AGE
hello-avpa                             1          0          0         11d          11d

How do I query pods are being scaled?

In AVPA versions later than 0.3.0, pods that are being scaled have the avpa.alibabacloud.com/resizing-lock label.

kubectl get po -n  [ns] -lavpa.alibabacloud.com/resizing-lock

How do I query pods that encounter scaling failures?

In AVPA versions later than 0.3.0, scaling failures are recorded in Kubernetes events. You can query scaling failures in the Logstore by specifying the InplaceResizedTimeoutFailed keyword.