Alibaba Cloud Service Mesh:Autoscale pods based on Mixerless Telemetry metrics

Deploy the metrics adapter

Run the following Helm command to install the metrics adapter in the kube-system namespace:

helm --kubeconfig <kubeconfig-path> -n kube-system install asm-custom-metrics \
  $KUBE_METRICS_ADAPTER_SRC/deploy/charts/kube-metrics-adapter \
  --set prometheus.url=http://prometheus.istio-system.svc:9090

Replace <kubeconfig-path> with the path to your kubeconfig file.

Verify the metrics adapter

1. Check that the metrics adapter pod is running: Expected output:

      kubectl --kubeconfig <kubeconfig-path> get po -n kube-system | grep metrics-adapter

      asm-custom-metrics-kube-metrics-adapter-6fb4949988-ht8pv   1/1     Running     0          30s

2. Confirm that the autoscaling/v2beta2 API is available: Expected output:

      kubectl --kubeconfig <kubeconfig-path> api-versions | grep "autoscaling/v2beta"

      autoscaling/v2beta1
      autoscaling/v2beta2

3. Verify that the External Metrics API endpoint is accessible: Expected output: The resources array is empty because no HPA has registered external metrics yet.

      kubectl --kubeconfig <kubeconfig-path> get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

      {
        "kind": "APIResourceList",
        "apiVersion": "v1",
        "groupVersion": "external.metrics.k8s.io/v1beta1",
        "resources": []
      }

Deploy the Flagger load tester

1. Download the Flagger YAML files from the Flagger GitHub repository.

2. Deploy the load tester in the test namespace: Replace <flagger-path> with the local path to the downloaded Flagger repository.

      kubectl --kubeconfig <kubeconfig-path> apply -f <flagger-path>/kustomize/tester/deployment.yaml -n test
      kubectl --kubeconfig <kubeconfig-path> apply -f <flagger-path>/kustomize/tester/service.yaml -n test

Metrics reference

The following table summarizes the Istio metrics available for HPA scaling:

Common PromQL labels:

Option 1: Scale based on request count

This HPA scales the podinfo deployment when the per-second request rate exceeds a threshold.

1. Create a file named requests_total_hpa.yaml:

   Key fields:

   Field	Description
   annotations	Defines the PromQL query. This query computes the per-second request rate for all services in the test namespace, using server-side (reporter="destination") metrics over a 1-minute window.
   averageValue: "10"	Triggers a scale-out when the average request rate reaches or exceeds 10 requests per second.
   maxReplicas / minReplicas	Pod count stays between 1 and 5.

      apiVersion: autoscaling/v2beta2
      kind: HorizontalPodAutoscaler
      metadata:
        name: podinfo-total
        namespace: test
        annotations:
          metric-config.external.prometheus-query.prometheus/processed-requests-per-second: |
            sum(rate(istio_requests_total{destination_workload_namespace="test",reporter="destination"}[1m]))
      spec:
        maxReplicas: 5
        minReplicas: 1
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: podinfo
        metrics:
          - type: External
            external:
              metric:
                name: prometheus-query
                selector:
                  matchLabels:
                    query-name: processed-requests-per-second
              target:
                type: AverageValue
                averageValue: "10"

2. Deploy the HPA:

      kubectl --kubeconfig <kubeconfig-path> apply -f resources_hpa/requests_total_hpa.yaml

3. Verify that the HPA registered its external metric: Expected output: The prometheus-query resource now appears in the list, confirming the HPA is connected to the metrics adapter.

      kubectl --kubeconfig <kubeconfig-path> get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

      {
        "kind": "APIResourceList",
        "apiVersion": "v1",
        "groupVersion": "external.metrics.k8s.io/v1beta1",
        "resources": [
          {
            "name": "prometheus-query",
            "singularName": "",
            "namespaced": true,
            "kind": "ExternalMetricValueList",
            "verbs": [
              "get"
            ]
          }
        ]
      }

Option 2: Scale based on average latency

This HPA scales the podinfo deployment when the average request latency exceeds a threshold.

Create a file named podinfo-latency-avg.yaml, then deploy it with kubectl apply:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: podinfo-latency-avg
  namespace: test
  annotations:
    metric-config.external.prometheus-query.prometheus/latency-average: |
      sum(rate(istio_request_duration_milliseconds_sum{destination_workload_namespace="test",reporter="destination"}[1m]))
      /sum(rate(istio_request_duration_milliseconds_count{destination_workload_namespace="test",reporter="destination"}[1m]))
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  metrics:
    - type: External
      external:
        metric:
          name: prometheus-query
          selector:
            matchLabels:
              query-name: latency-average
        target:
          type: AverageValue
          averageValue: "0.005"

Key fields:

Option 3: Scale based on P95 latency

This HPA scales the podinfo deployment when the 95th percentile (P95) request latency exceeds a threshold.

Create a file named podinfo-p95.yaml, then deploy it with kubectl apply:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: podinfo-p95
  namespace: test
  annotations:
    metric-config.external.prometheus-query.prometheus/p95-latency: |
      histogram_quantile(0.95,sum(irate(istio_request_duration_milliseconds_bucket{destination_workload_namespace="test",destination_canonical_service="podinfo"}[5m]))by (le))
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  metrics:
    - type: External
      external:
        metric:
          name: prometheus-query
          selector:
            matchLabels:
              query-name: p95-latency
        target:
          type: AverageValue
          averageValue: "4"

Key fields: