Alibaba Cloud Service Mesh:Use ASM metrics to automatically scale workloads

Step 1: Enable Prometheus metric collection

Enable ASM to export metrics to your Prometheus instance. For detailed instructions, see Collect monitoring metrics in Managed Service for Prometheus.

Step 2: Deploy the custom metrics adapter

Install kube-metrics-adapter, which registers itself as a provider for the External Metrics API and translates HPA metric requests into Prometheus queries.

1. Install the adapter with Helm. Replace http://prometheus.istio-system.svc:9090 with the URL of your Prometheus server if it runs in a different namespace or uses a different service name.

      helm -n kube-system install asm-custom-metrics ./kube-metrics-adapter \
        --set prometheus.url=http://prometheus.istio-system.svc:9090

2. Verify the installation.

   1. Confirm that the autoscaling/v2 API is available.

         kubectl api-versions | grep "autoscaling/v2"

      Expected output:

         autoscaling/v2

   2. Check that the adapter pod is running.

         kubectl get po -n kube-system | grep metrics-adapter

      Expected output:

         asm-custom-metrics-kube-metrics-adapter-85c6d5d865-2****   1/1     Running   0   19s

   3. Confirm that the External Metrics API endpoint is registered.

         kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

      Expected output:

         {
           "kind": "APIResourceList",
           "apiVersion": "v1",
           "groupVersion": "external.metrics.k8s.io/v1beta1",
           "resources": []
         }

      The resources array is empty because no HPA has registered a metric query yet. It populates after you deploy an HPA in Step 4.

Step 3: Deploy the sample application

Step 4: Configure the HPA with ASM metrics

Define an HPA that scales the podinfo Deployment based on the request rate. The configuration maps three pieces together:

This example uses HPA API version autoscaling/v2, which requires Kubernetes 1.23 or later. The v2beta2 version was removed in Kubernetes 1.26.

1. Create a file named hpa.yaml with the following content.

   Key fields explained:

   Field	Description
   metric-config.external.prometheus-query.prometheus/processed-requests-per-second	Annotation that embeds the PromQL query. The suffix (processed-requests-per-second) is a user-defined name that links the annotation to the HPA metric selector.
   metrics[].external.metric.name	Must be prometheus-query -- the fixed identifier that kube-metrics-adapter recognizes.
   metrics[].external.metric.selector.matchLabels.query-name	Must match the suffix of the annotation key (processed-requests-per-second).
   target.averageValue	The threshold per pod. The HPA divides the total query result by the current replica count and compares it to this value.

      apiVersion: autoscaling/v2
      kind: HorizontalPodAutoscaler
      metadata:
        name: podinfo
        namespace: test
        annotations:
          # Annotation format: metric-config.external.prometheus-query.prometheus/<query-name>
          # <query-name> must match the query-name label in spec.metrics below.
          # The value is the PromQL query that kube-metrics-adapter sends to Prometheus.
          metric-config.external.prometheus-query.prometheus/processed-requests-per-second: |
            sum(
                rate(
                    istio_requests_total{
                      destination_workload="podinfo",
                      destination_workload_namespace="test",
                      reporter="destination"
                    }[1m]
                )
            )
      spec:
        maxReplicas: 10
        minReplicas: 1
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: podinfo
        metrics:
          - type: External          # Use External because Prometheus metrics are not Kubernetes objects
            external:
              metric:
                name: prometheus-query                  # Fixed metric name for kube-metrics-adapter
                selector:
                  matchLabels:
                    query-name: processed-requests-per-second  # Must match the annotation key above
              target:
                type: AverageValue
                averageValue: "10"   # Scale out when the average request rate exceeds 10 req/s per pod

2. Apply the HPA.

      kubectl apply -f hpa.yaml

3. Verify that the External Metrics API now lists the registered metric. Expected output: The prometheus-query resource confirms that the HPA registered its metric with the adapter.

      kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

      {
        "kind": "APIResourceList",
        "apiVersion": "v1",
        "groupVersion": "external.metrics.k8s.io/v1beta1",
        "resources": [
          {
            "name": "prometheus-query",
            "singularName": "",
            "namespaced": true,
            "kind": "ExternalMetricValueList",
            "verbs": [
              "get"
            ]
          }
        ]
      }

Verify automatic scaling

1. Generate sustained load from the load tester pod. This sends 50 requests per second (10 concurrent workers at 5 QPS each) for 5 minutes.

      # Get the load tester pod name (skip this if $loadtester is already set from Step 3)
      export loadtester=$(kubectl -n test get pod -l "app=loadtester" \
        -o jsonpath='{.items[0].metadata.name}')
   
      kubectl -n test exec -it ${loadtester} -c loadtester -- \
        hey -z 5m -c 10 -q 5 http://podinfo.test:9898

2. Monitor the HPA in a separate terminal. Expected output (after approximately one minute):

   Note

   The HPA syncs metrics every 30 seconds. A scaling decision takes effect only if no scaling event occurred in the previous 3--5 minutes. This cooldown prevents rapid, conflicting scaling decisions and gives the Cluster Autoscaler time to provision new nodes if needed.

      watch kubectl -n test get hpa/podinfo

      NAME      REFERENCE            TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
      podinfo   Deployment/podinfo   8308m/10 (avg)   1         10        6          124m

3. Observe scale-down after the load test ends. When the load test completes, the request rate drops to zero. The HPA gradually scales down the Deployment until the replica count returns to 1.