Container Service for Kubernetes:Horizontally scale pods with Alibaba Cloud metrics

This guide will walk you through the following steps, using an nginx application as an example. You will deploy an nginx Deployment, Service, and Ingress, then configure an HPA to automatically scale the Deployment based on the Ingress QPS metric collected from Simple Log Service (SLS).

Step 1: Deploy ack-alibaba-cloud-metrics-adapter

ack-alibaba-cloud-metrics-adapter enables the Kubernetes HPA to scale workloads based on metrics from Alibaba Cloud services, such as Elastic Compute Service (ECS), Server Load Balancer (SLB), and ApsaraDB RDS (RDS), via the External Metrics API.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the navigation pane on the left, choose Applications > Helm.

3. On the Helm page, click Deploy. Configure the Basic Information parameters, select ack-alibaba-cloud-metrics-adapter, then click Next.

4. In the Parameters step, configure the Chart Version parameter and click OK.

Step 2: Create an application and a service

1. Create a file named nginx-test.yaml.

   YAML manifest

   apiVersion: apps/v1 
   kind: Deployment
   metadata:
     name: nginx-deployment-basic
     labels:
       app: nginx
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx:1.7.9 
           ports:
           - containerPort: 80
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx
     namespace: default
   spec:
     ports:
       - port: 80
         protocol: TCP
         targetPort: 80
     selector:
       app: nginx
     type: ClusterIP

2. Run the following command to create the Deployment and Service.

   kubectl apply -f nginx-test.yaml

Step 3: Create an Ingress

1. In the left navigation pane of the cluster management page, choose Network > Ingresses. In the upper-left corner of the Ingresses page, click Create Ingress.

2. In the Create Ingress panel, complete the required fields and click OK. After the Ingress is created, you are redirected to the Ingresses page.

3. Under the Name column, click the Ingress name to view its routing rules. For Ingress details, see Ingress management.

Step 4: Configure the HPA

In this step, you will configure the HPA to use two metrics from your SLS project for scaling decisions: sls_ingress_qps and sls_ingress_latency_p9999.

The target type for each metric is configured differently:

• sls_ingress_qps:

  Target type: AverageValue

  Description: This target type tells the HPA to divide the total QPS metric by the current number of pod replicas. The scaling decision is then based on this per-Pod average.

• sls_ingress_latency_p9999:

  Target type: Value

  Description: This target type tells the HPA to use the metric's raw value directly, without dividing it by the number of pods.

1. Create a file named ingress-hpa.yaml and copy the following content into it.

   View YAML example

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: ingress-hpa
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: nginx-deployment-basic
     minReplicas: 2
     maxReplicas: 10
     metrics:
       - type: External
         external:
           metric:
             name: sls_ingress_qps
             selector:
               matchLabels:
                 sls.project: "***" # Replace with your actual SLS project name.
                 sls.logstore: "nginx-ingress"
                 sls.ingress.route: "default-nginx-80"
           target:
             type: AverageValue
             averageValue: 10
       - type: External
         external:
           metric:
             name: sls_ingress_latency_p9999
             selector:
               matchLabels:
                 # The default Ingress log project is k8s-log-<clusterId>.
                 sls.project: "***" 
                 # The default Ingress Logstore is nginx-ingress.
                 sls.logstore: "nginx-ingress"
                 # Format: <namespace>-<svc>-<port>.
                 sls.ingress.route: "default-nginx-80"
                 # Specifies whether to use the SLS VPC endpoint. Default is true.
                 # sls.internal.endpoint: true
           target:
             type: Value
             # If sls_ingress_latency_p9999 exceeds 10ms, HPA automatically increases the number of pods for nginx-deployment-basic.
             value: 10

   The following table describes the parameters in the HPA configuration.

   Parameter	Required	Description
   sls.ingress.route	Yes	Specifies the target Ingress route in the format: <namespace>-<svc>-<port>. <namespace>: The namespace where the Ingress is located. <svc>: The name of the Service targeted by the Ingress. <port>: The name of the port on the Service . Example: default-nginx-80.
   sls.logstore	Yes	The name of the SLS Logstore where the metrics are stored. Default value in a cluster is nginx-ingress.
   sls.project	Yes	The name of the SLS Project. Default value in a cluster is k8s-log-Cluster ID.
   sls.internal.endpoint	No	Specifies whether to access SLS via a private (internal) or public endpoint. true (default): Access SLS via the internal VPC endpoint. false: Access SLS via the public endpoint.

2. Run the following command to create the HPA.

   kubectl apply -f ingress-hpa.yaml

Step 5: Verify the results

After configuring the HPA, you can verify its behavior by generating load and observing the scaling status.

1. Run the following command to start a load test against the service exposed by your Ingress.

   # This command uses Apache Benchmark (ab) to simulate 10 concurrent users (-c 10) for a duration of 300 seconds (-t 300).
   # Replace <your-ingress-domain> with the hostname you configured.
   ab -t 300 -c 10 <your-ingress-domain> 

2. Verify the scaling status.

   Run the following command to check the HPA status:

   kubectl get hpa ingress-hpa

   Expected output:

   NAME            REFERENCE                              TARGETS           MINPODS    MAXPODS    REPLICAS   AGE
   ingress-hpa     Depolyment/nginx-deployment-basic      21/10 (avg)       2          10         10          7m49s

   The scale-out is considered successful if the value in the REPLICAS column has reached the value in the MAXPODS column. This confirms that the HPA has responded to the increased load by scaling up the application to its maximum configured limit.

FAQ

kubectl get --raw  /apis/external.metrics.k8s.io/v1beta1/namespaces/*/sls_ingress_qps?labelSelector=sls.project={{SLS_Project}},sls.logstore=nginx-ingress

References

• Horizontal pod autoscaling based on Prometheus Service metrics

• Horizontally scale multiple applications based on Nginx Ingress traffic metrics