Container Service for Kubernetes:Implement horizontal pod autoscaling based on Alibaba Cloud metrics

In this example, a Deployment, a Service, and an Ingress named NGINX are created to configure Horizontal Pod Autoscaler (HPA). This allows you to implement horizontal auto scaling based on the QPS of the Ingress in Simple Log Service (SLS).

Step 1: Install the ack-alibaba-cloud-metrics-adapter component

The ack-alibaba-cloud-metrics-adapter component allows Kubernetes to obtain the monitoring data of Alibaba Cloud services, such as Elastic Compute Service (ECS), Server Load Balancer (SLB), and ApsaraDB RDS, by using the External Metrics API, which can enhance the monitoring and auto-scaling capabilities of clusters.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the name of the cluster that you want to manage and choose Applications > Helm in the left-side navigation pane.

3. On the Helm page, click Deploy. In the Basic Information step, configure the parameters and select ack-alibaba-cloud-metrics-adapter. Then, click Next. The following table describes the parameters.

4. In the Parameters step, configure the Chart Version parameter and click OK.

Step 2: Create an application and a Service

1. Create a file named nginx-test.yaml.

   View YAML examples

   apiVersion: apps/v1 
   kind: Deployment
   metadata:
     name: nginx-deployment-basic
     labels:
       app: nginx
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx:1.7.9 
           ports:
           - containerPort: 80
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx
     namespace: default
   spec:
     ports:
       - port: 80
         protocol: TCP
         targetPort: 80
     selector:
       app: nginx
     type: ClusterIP

2. Run the following command to create a Deployment and a Service:

   kubectl apply -f nginx-test.yaml

Step 3: Create an Ingress

1. In the left-side navigation pane, choose Network > Ingresses. In the upper-left corner of the Ingresses page, click Create Ingress.

2. In the Create Ingress dialog box, configure the parameters and click OK. After you create an Ingress, the Ingresses page appears.

3. In the Name column, find and click the name of newly created Ingress to view information about the Ingress. For more information about Ingresses, see Ingress management.

Step 4: Configure HPA

You can configure two metrics to scale SLS projects in HPA, such as the sls_ingress_qps and sls_ingress_latency_p9999 metrics.

• Set the sls_ingress_qps metric to AverageValue. This specifies that the metric value is the result of dividing the total QPS by the number of pods.

• Set the sls_ingress_latency_p9999 metric to Value. This specifies that the latency is not divided by the number of pods.

1. Create a file named ingress-hpa.yaml and add the following content to the file:

   View YAML examples

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: ingress-hpa
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: nginx-deployment-basic
     minReplicas: 2
     maxReplicas: 10
     metrics:
       - type: External
         external:
           metric:
             name: sls_ingress_qps
             selector:
               matchLabels:
                 sls.project: "***" # Specify the SLS project that you want to use. 
                 sls.logstore: "nginx-ingress"
                 sls.ingress.route: "default-nginx-80"
           target:
             type: AverageValue
             averageValue: 10
       - type: External
         external:
           metric:
             name: sls_ingress_latency_p9999
             selector:
               matchLabels:
                 # default ingress log project is k8s-log-clusterId
                 sls.project: "***" 
                 # default ingress logstre is nginx-ingress
                 sls.logstore: "nginx-ingress"
                 # namespace-svc-port
                 sls.ingress.route: "default-nginx-80"
                 # sls vpc endpoint, default true
                 # sls.internal.endpoint:true
           target:
             type: Value
             # sls_ingress_latency_p9999>10ms specifies that HPA automatically increases the number of nginx-deployment-basic pods if the value exceeds 10 milliseconds. 
             value: 10

   The following table describes the parameters that are used to configure HPA.

   Parameter	Required	Description
   sls.ingress.route	Yes	Parameter format: <namespace>-<svc>-<port>. <namespace> specifies the namespace to which the Ingress belongs. <svc> specifies the name of the Service that you selected when you created the Ingress. <port> specifies the port of the Service. Example: default-nginx-80
   sls.logstore	Yes	The name of the Logstore in SLS. The default value of sls.logstore is nginx-ingress.
   sls.project	Yes	The name of the project in SLS. The default value of sls.project is k8s-log-cluster ID.
   sls.internal.endpoint	No	Specifies whether SLS is accessed over an internal network. Default value: true. true: Access SLS over an internal network. false: Access SLS over the Internet.

2. Run the following command to configure HPA:

   kubectl apply -f ingress-hpa.yml

Step 5: Verify the configuration

1. After you configure HPA, run the following command to perform a stress test:

   ab -t 300 -c 10 <Domain name of the Ingress> # Use Apache Benchmark to send requests to the Service exposed by the Ingress. The test requires 300 seconds to complete and 10 concurrent requests are sent per second.

2. Check whether HPA works as expected.

   1. In the left-side navigation pane of the ACK console, click Clusters. On the Clusters page, find the cluster that you want to manage and choose More > Open Cloud Shell in the Actions column.

   2. Run the following command to check the status of HPA:

      kubectl get hpa ingress-hpa

      Expected output:

      NAME            REFERENCE                              TARGETS           MINPODS    MAXPODS    REPLICAPS   AGE
      ingress-hpa     Depolyment/nginx-deployment-basic      21/10 (avg)       2          10         10          7m49s

      If the value of the REPLICAS parameter is the same as the value of the MAXPODS parameter, HPA scaled out the application as expected.

FAQ

How do I use a CLI to obtain the data of the sls_ingress_qps QPS metrics?

kubectl get --raw  /apis/external.metrics.k8s.io/v1beta1/namespaces/*/sls_ingress_qps?labelSelector=sls.project={{SLS_Project}},sls.logstore=nginx-ingress

Error from server: {
    "httpCode": 400,
    "errorCode": "ParameterInvalid",
    "errorMessage": "key (slb_pool_name) is not config as key value config,if symbol : is  in your log,please wrap : with quotation mark \"",
    "requestID": "xxxxxxx"
}

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "sls_ingress_qps",
      "timestamp": "2025-02-26T16:45:00Z", 
      "value": "50", # The value of QPS.
      "metricLabels": {
        "sls.project": "your-sls-project-name",
        "sls.logstore": "nginx-ingress"
      }
    }
  ]
}

What do I do if the target column is <unknown> after I run the kubectl get hpa command?

How do I find the metrics that are supported by HPA?

How do I configure horizontal autoscaling after I change the format of NGINX Ingress logs?

References

• Horizontal pod scaling based on Managed Service for Prometheus metrics

• Configure horizontal pod autoscaling for multiple applications based on the metrics of the NGINX Ingress controller