Container Service for Kubernetes:Configure horizontal pod autoscaling with Prometheus metrics

Option A: Use built-in ACK metrics

Prometheus Service is installed by default in ACK and collects metrics including:

• cAdvisor container metrics

• Node Exporter infrastructure metrics

• GPU Exporter metrics

• Any additional metrics you have connected to Prometheus Service

To see all connected metrics:

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, choose Operations > Prometheus Monitoring.

3. In the upper-right corner, click Alert Settings . If the link does not appear, update Managed Service for Prometheus to the latest version. See Prerequisites.

4. In the Prometheus Service console, click Settings in the left navigation pane to view all connected metrics.

Option B: Expose custom metrics from a pod

If your application exposes Prometheus-format metrics, scrape them with a ServiceMonitor. This example deploys a sample app that exposes http_requests_total and configures scraping.

Update the adapter configuration

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. Click the name of your cluster. In the left navigation pane, click Applications > Helm.

3. Find ack-alibaba-cloud-metrics-adapter and click Update in the Actions column.

4. In the Update Release panel, update the YAML with your configuration and click OK.

   YAML details

   AlibabaCloudMetricsAdapter:
   ......
     prometheus:
       enabled: true    # Enable the Prometheus adapter.
       # Prometheus Service data request URL. See "Get the Prometheus data request URL" below.
       url: https://cn-beijing.arms.aliyuncs.com:9443/api/v1/prometheus/xxxx/xxxx/xxxx/cn-beijing
       # Prometheus V1 with token authentication enabled:
       prometheusHeader:
       - Authorization: {Token}
       # Prometheus V2 without password-free access (uncomment and replace the V1 block above):
       # prometheusHeader:
       # - Authorization: Basic <base64Encode(<accessKey:secretKey>)>
       metricsRelistInterval: 1m  # Interval for refreshing the metric list. Keep the default.
       logLevel: 5                # Debug log level. Keep the default.
       adapter:
         rules:
           default: false  # Do not expose predefined metrics by default.
           custom:
           # Example: convert container_memory_working_set_bytes to container_memory_working_set_bytes_per_second
           - seriesQuery: 'container_memory_working_set_bytes{namespace!="",pod!=""}'
             resources:
               overrides:
                 namespace: { resource: "namespace" }
                 pod: { resource: "pod" }
             name:
               matches: "^(.*)_bytes"
               as: "${1}_bytes_per_second"
             metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
           # Example: convert container_cpu_usage_seconds_total to container_cpu_usage_core_per_second
           - seriesQuery: 'container_cpu_usage_seconds_total{namespace!="",pod!=""}'
             resources:
               overrides:
                 namespace: { resource: "namespace" }
                 pod: { resource: "pod" }
             name:
               matches: "^(.*)_seconds_total"
               as: "${1}_core_per_second"
             metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'
           # Example: convert http_requests_total to http_requests_per_second
           - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
             resources:
               overrides:
                 namespace: {resource: "namespace"}
                 pod: {resource: "pod"}
             name:
               matches: "^(.*)_total"
               as: "${1}_per_second"
             metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
   ......

The following table lists key adapter fields. For the full reference, see Adapter configuration reference.

Verify the adapter is working

Run these commands to confirm the adapter exposes metrics through the Kubernetes aggregation API.

Check Custom Metrics:

# List all available Custom Metrics
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq .

# Query container_memory_working_set_bytes_per_second for pods in kube-system
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kube-system/pods/*/container_memory_working_set_bytes_per_second" | jq .

# Query container_cpu_usage_core_per_second for pods in kube-system
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kube-system/pods/*/container_cpu_usage_core_per_second" | jq .

A successful response looks similar to:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kube-system/pods/%2A/container_cpu_usage_core_per_second"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "kube-system",
        "name": "ack-cost-exporter-7f44d55c66-cgtz7",
        "apiVersion": "/v1"
      },
      "metricName": "container_cpu_usage_core_per_second",
      "timestamp": "2025-12-30T03:30:21Z",
      "value": "4m",
      "selector": null
    }
  ]
}

The response confirms the adapter discovered and associated the metric with the correct pod. The items array lists one entry per pod. The value field uses the m suffix (milli-units), so 4m means 0.004 CPU cores per second.

Check External Metrics:

# List all available External Metrics
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" | jq .

# Query http_requests_per_second in the default namespace
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/http_requests_per_second" | jq .

A successful response looks similar to:

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "http_requests_per_second",
      "metricLabels": {},
      "timestamp": "2025-12-30T03:29:40Z",
      "value": "328m"
    }
  ]
}

Unlike Custom Metrics, the External Metrics response contains a single aggregated value (not per pod) because the metric is not scoped to a Kubernetes object.

Template variables in metricsQuery

The metricsQuery field is a Go template, not plain PromQL. The adapter populates it with values from the HPA request before querying Prometheus. The delimiters are << and >> (not {{ and }}) to avoid conflicts with PromQL syntax.

Discovery

seriesQuery specifies which Prometheus metric to convert and accepts any valid PromQL selector, including label filters.

- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'

The label filters namespace!="" and pod!="" restrict the metric to pods with both labels set, a requirement for resource association in the next step.

To narrow matching series, add a seriesFilters block:

- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  seriesFilters:
    - isNot: "^container_.*_seconds_total"

seriesFilters accepts two operators:

• is:<regex>: keep only series whose name matches the regex.

• isNot:<regex>: exclude series whose name matches the regex.

Association

resources.overrides maps Prometheus label names to Kubernetes API resources. This tells the adapter which label to match when HPA requests a metric for a specific pod or namespace.

- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}

The keys (namespace, pod) are the Prometheus label names. The values ("namespace", "pod") are Kubernetes API resource types, as listed by kubectl api-resources -o wide. Each key must exist as a label in your Prometheus data.

Naming

name converts the Prometheus metric name into the HPA metric name using a regular expression. The original Prometheus metric name is unchanged.

- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"

matches is a regex that captures part of the Prometheus metric name. as defines the HPA metric name, using ${1} to reference the first capture group. Here, http_requests_total becomes http_requests_per_second.

For External Metrics, convert uppercase letters in the Prometheus metric name to lowercase in the HPA metric name.

To list all available HPA metric names:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

Querying

metricsQuery defines the PromQL expression the adapter sends to Prometheus after substituting template variables.

- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

The label selectors in metricsQuery are injected via <<.LabelMatchers>> at query time and do not inherit filters from seriesQuery.

Alibaba Cloud Prometheus Service

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. Click the name of your cluster. In the left navigation pane, choose Operations > Prometheus Monitoring.

3. In the upper-right corner, click Alert Settings . If the link does not appear, update ack-arms-prometheus to the latest version. See Prerequisites.

4. In the Prometheus Service console, click Settings > Settings to find the HTTP API Address (Grafana Read URL) . Use the Internal Network address when available; otherwise use the Internet address.

   

5. Configure authentication for your Prometheus version.

   • Prometheus V1 (token authentication disabled by default): If enabled, copy the token from the Prometheus console and configure the adapter.

     

     prometheus:
       prometheusHeader:
       - Authorization: {Token}

   • Prometheus V2 (AccessKey authentication enabled by default): If password-free access is not enabled, Base64-encode your AccessKey ID and AccessKey secret.

     1. Generate a Base64-encoded string.

        Concatenate your AccessKey ID and AccessKey secret as AccessKey:AccessSecret, then Base64-encode:

        echo -n 'accessKey:secretKey' | base64

     2. Configure the component.

        Enter the generated string in the Basic <encoded string> format into the Authorization field of prometheusHeader.

        ...
            prometheus:
              prometheusHeader:
              - Authorization: Basic YWxxxxeQ==
        ...

Open source Prometheus

For self-managed Prometheus, expose its API through a Kubernetes Service and set the Service URL in the adapter configuration.

This example uses the ack-prometheus-operator Helm chart from ACK Marketplace. See Open source Prometheus monitoring.

1. Deploy ack-prometheus-operator.

   1. Log on to the ACK console. In the left navigation pane, choose Marketplace > Marketplace.

   2. Search for ack-prometheus-operator, click its card, then click Deploy.

   3. Select the Cluster and Namespace, set the Release Name, and click Next. Adjust Parameters as needed and click OK.

2. Verify the deployment.

   1. Expose the Prometheus API through a Service. This example uses the ack-prometheus-operator-prometheus Service.

   2. In a browser, go to ServiceIP:9090. For public access, expose the Service through an SLB instance.

   3. In the Prometheus console, click Status > Targets to view all scrape jobs. If all jobs show a State of UP, scraping is working correctly.

   4. Note the Service name and namespace from the Labels column. Here, the Service is ack-prometheus-operator-prometheus in the monitoring namespace.

3. Set the Prometheus URL in the adapter. For internal access:

   AlibabaCloudMetricsAdapter:
     prometheus:
       enabled: true
       url: http://ack-prometheus-operator-prometheus.monitoring.svc.cluster.local:9090

   For public access:

   AlibabaCloudMetricsAdapter:
     prometheus:
       enabled: true
       url: http://your_domain.com:9090   # Replace with your public IP address or domain.

To add a Prometheus data source, see Add a Prometheus data source in Grafana.