Container Service for Kubernetes:Scale pods based on Nginx Ingress metrics

Prerequisites

Before you begin, ensure that you have:

• An ACK cluster with kubectl configured

• An Ingress controller with SLS log collection enabled (enabled by default when you create a cluster)

• Permissions to install Helm charts and create HPA resources

How it works

The ack-alibaba-cloud-metrics-adapter component bridges Alibaba Cloud monitoring services and the Kubernetes External Metrics API. Once installed, HPA can query SLS Ingress metrics—such as sls_ingress_qps and sls_ingress_latency_p9999—and adjust the replica count of your Deployment accordingly.

Step 1: Install ack-alibaba-cloud-metrics-adapter

The ack-alibaba-cloud-metrics-adapter component enables Kubernetes to retrieve monitoring data from Alibaba Cloud services—including ECS, SLB, and RDS—through the External Metrics API.

1. Log on to the Container Service Management Console. In the navigation pane, click Clusters.

2. Click the name of your cluster. In the left navigation pane, choose Applications > Helm.

3. On the Helm page, click Create. Complete the Basic Information configuration, select ack-alibaba-cloud-metrics-adapter, and click Next.

4. On the Parameter Settings page, select a Chart Version and click OK.

Step 3: Create an Ingress

1. In the left navigation pane of the cluster management page, choose Network > Ingress. On the Ingress page, click Create Ingress.

2. Fill in the required fields and click OK. The console redirects you to the Ingress page.

3. In the Name column, click the Ingress name to view its routing rules. For details, see Manage Ingress.

Step 4: Configure HPA

HPA supports two SLS Ingress metrics with different scaling semantics:

1. Create ingress-hpa.yaml with the following content:

   Parameter	Required	Description
   sls.ingress.route	Yes	Format: <namespace>-<svc>-<port>. Example: default-nginx-80
   sls.logstore	Yes	The Logstore in Simple Log Service. Default: nginx-ingress
   sls.project	Yes	The Project in Simple Log Service. Default: k8s-log-<cluster-id>
   sls.internal.endpoint	No	Whether to access SLS over the internal network. Default: true. Set to false to use the public network.

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: ingress-hpa
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: nginx-deployment-basic
     minReplicas: 2
     maxReplicas: 10
     metrics:
       - type: External
         external:
           metric:
             name: sls_ingress_qps
             selector:
               matchLabels:
                 sls.project: "<your-sls-project>"      # Default: k8s-log-<cluster-id>
                 sls.logstore: "nginx-ingress"           # Default SLS Logstore for Nginx Ingress logs
                 sls.ingress.route: "default-nginx-80"  # Format: <namespace>-<svc>-<port>
           target:
             type: AverageValue
             averageValue: 10   # Scale up when average QPS per pod exceeds 10
       - type: External
         external:
           metric:
             name: sls_ingress_latency_p9999
             selector:
               matchLabels:
                 sls.project: "<your-sls-project>"
                 sls.logstore: "nginx-ingress"
                 sls.ingress.route: "default-nginx-80"
                 # sls.internal.endpoint: "true"         # true = internal network (default); false = public network
           target:
             type: Value
             value: 10   # Scale up when 99.99th-percentile latency exceeds 10 ms

   The following table describes the parameters used in the HPA configuration.

2. Apply the configuration:

   kubectl apply -f ingress-hpa.yaml

Verify scaling

Run a 5-minute load test against your Ingress domain to trigger scale-up:

# Send 10 concurrent requests for 300 seconds
ab -t 300 -c 10 <domain-name-configured-for-the-ingress>

Watch the HPA status in real time:

kubectl get hpa ingress-hpa --watch

Expected output when scale-up succeeds (REPLICAS reaches MAXPODS):

NAME            REFERENCE                              TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
ingress-hpa     Deployment/nginx-deployment-basic      21/10 (avg)       2         10        10         7m49s

When traffic drops below the threshold, REPLICAS returns to the minimum value.

Troubleshooting

Start troubleshooting by running kubectl describe hpa ingress-hpa and checking the Conditions field. Use the Type and Status columns to identify the issue, then follow the steps below.

The `target` column shows `unknown`

1. Run kubectl describe hpa <hpa-name> and check the Conditions field:

   • If AbleToScale is False: confirm the Deployment is healthy.

   • If ScalingActive is False: proceed to the next step.

2. Run the following command to check whether the External Metrics API is available:

   kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/"

   If the response is Error from server (NotFound), check the startup status of alibaba-cloud-metrics-adapter. If the adapter is running normally and the HPA metrics are Ingress-related, deploy the SLS component first. For details, see Collect and analyze Nginx Ingress access logs.

3. Verify the format of sls.ingress.route. The value must be <namespace>-<svc>-<port>, where:

   • namespace: the namespace where the Ingress resides

   • svc: the Service name associated with the Ingress

   • port: the port name of that Service

Failed to pull ack-alibaba-cloud-metrics-adapter image (version 1.3.7)

When upgrading to version 1.3.7, you may see:

Failed to pull image "registry-<region-id>-vpc.ack.aliyuncs.com/acs/alibaba-cloud-metrics-adapter-amd64:v0.2.9-ba634de-aliyun".

This happens because the component does not support direct updates. To resolve this:

1. Back up the current component configuration.

2. Uninstall the old version.

3. Install the latest version using the backup configuration.

FAQ

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/sls_ingress_qps?labelSelector=sls.project={{SLS_Project}},sls.logstore=nginx-ingress"

What's next

• Horizontal pod autoscaling based on Alibaba Cloud Prometheus metrics

• Multi-application horizontal pod autoscaling based on Nginx Ingress traffic metrics