Container Service for Kubernetes:Horizontal pod autoscaling with Alibaba Cloud Prometheus

Prerequisites

• Alibaba Cloud Managed Service for Prometheus is deployed. For more information, see Connect to and configure Alibaba Cloud Managed Service for Prometheus.

• The ack-alibaba-cloud-metrics-adapter component is deployed. For more information, see Deploy the ack-alibaba-cloud-metrics-adapter component.

  Note

  You can log on to the ACK console and deploy the ack-alibaba-cloud-metrics-adapter component from the Marketplace > App Marketplace page.

How it works

By default, HPA only supports autoscaling based on CPU and memory, which can be insufficient for many operational needs. Alibaba Cloud Managed Service for Prometheus is fully compatible with the open source Prometheus ecosystem. It supports monitoring for a wide range of components, provides various out-of-the-box dashboards, and offers a fully managed Prometheus service. The process involves three main steps:

1. Expose monitoring metrics in an ACK cluster by using Alibaba Cloud Managed Service for Prometheus.

2. The ack-alibaba-cloud-metrics-adapter component is responsible for converting Prometheus monitoring metrics into Kubernetes aggregated metrics that are consumable by HPA. For more information, see Autoscaling on multiple metrics and custom metrics.

3. Configure and deploy an HPA to scale based on the metrics from the previous step.

   Metrics are categorized into two types:

   • Custom Metric: Scales based on metrics related to a target Kubernetes object, such as a Pod. An example is a metric specific to the Pod itself. For more information, see autoscaling-on-multiple-metrics-and-custom-metrics.

   • External Metric: Scales based on metrics that are not related to the target Kubernetes object, such as a Pod. For example, you can scale the Pods of a workload based on the overall business QPS metric. For more information, see autoscaling-on-metrics-not-related-to-kubernetes-objects.

This topic describes how to configure ack-alibaba-cloud-metrics-adapter to convert metrics from Alibaba Cloud Managed Service for Prometheus into HPA-compatible metrics and implement autoscaling based on these metrics.

Step 1: Get Prometheus monitoring data

Example 1: Use default ACK metrics

You can use the default metrics from Alibaba Cloud Managed Service for Prometheus, which is installed by default in ACK, for horizontal pod autoscaling. Supported metrics include cAdvisor container metrics, Node-Exporter basic node monitoring metrics, GPU-Exporter metrics, and any other metrics you have integrated with Alibaba Cloud Managed Service for Prometheus. To view the integrated metrics:

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Prometheus Monitoring.

3. In the upper-right corner, click Go to ARMS Prometheus.

4. In the left-side navigation pane of the Application Real-Time Monitoring Service (ARMS) console, click Settings to view all metrics integrated with Alibaba Cloud Managed Service for Prometheus.

Example 2: Use pod-reported metrics

This example deploys a sample application that exposes metrics in the standard Prometheus format, such as the http_requests_total metric, which indicates the number of visits. For more information, see metric_type.

1. Deploy the application workload.

   1. Log on to the ACK console. In the left navigation pane, click Clusters.

   2. On the Clusters page, click the name of the target cluster. In the left-side navigation pane, click Workloads > Deployments.

   3. On the Deployments page, click Create from YAML in the upper-right corner. On the Create page, select Custom from the Sample Template drop-down list, enter the following YAML, and click Create.

      YAML details

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: sample-app
        labels:
          app: sample-app
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: sample-app
        template:
          metadata:
            labels:
              app: sample-app
          spec:
            containers:
            - image: registry-cn-hangzhou.ack.aliyuncs.com/acs/autoscale-demo:v0.1.2-dfbc5fd-aliyun
              name: metrics-provider
              ports:
              - name: http
                containerPort: 8080
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: sample-app
        namespace: default
        labels:
          app: sample-app
      spec:
        ports:
          - port: 8080
            name: http
            protocol: TCP
            targetPort: 8080
        selector:
          app: sample-app
        type: ClusterIP

2. Add a ServiceMonitor.

   1. Log on to the Application Real-Time Monitoring Service (ARMS) console.

   2. In the left-side navigation pane, click Integration Management. On the Integrated Environments page, on the Container Environment tab, select the region where your cluster is located, and then click the environment name that matches your cluster name.

   3. On the container environment page, click the Metric Scraping tab. In the left-side navigation pane, click ServiceMonitor. On the ServiceMonitor page, click Create. In the Add ServiceMonitor Configuration panel, click YAML Edit, add the following ServiceMonitor configuration, and then create the ServiceMonitor.

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        annotations:
          arms.prometheus.io/discovery: 'true'
        name: sample-app
        namespace: default
      spec:
        endpoints:
        - interval: 30s
          port: http
          path: /metrics
        namespaceSelector:
          any: true
        selector:
          matchLabels:
            app: sample-app

3. Verify the monitoring status.

   Click the Self-Monitoring tab. In the Targets section, if default/sample-app/0(1/1 up) is displayed, the application is successfully monitored by Alibaba Cloud Managed Service for Prometheus.

4. On the Prometheus dashboard, query the value of http_requests_total for a recent time range to confirm that monitoring data is being collected correctly.

Step 2: Configure the ack-alibaba-cloud-metrics-adapter component

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left-side navigation pane, click Applications > Helm.

3. On the Helm page, find ack-alibaba-cloud-metrics-adapter and click Update in the Actions column.

4. In the Update Release panel, configure the following YAML, and then click OK.

   YAML details

     AlibabaCloudMetricsAdapter:
     ......
       prometheus:
         enabled: true    # Set to true to enable the Prometheus adapter feature.
         # The URL of your Alibaba Cloud Managed Service for Prometheus instance.
         url: https://cn-beijing.arms.aliyuncs.com:9443/api/v1/prometheus/xxxx/xxxx/xxxx/cn-beijing
       	# If you use Alibaba Cloud Managed Service for Prometheus V1 with token authentication enabled, configure prometheusHeader Authorization.
         prometheusHeader:
         - Authorization: {Token}
         	# If you use Alibaba Cloud Managed Service for Prometheus V2 without password-free access enabled, configure prometheusHeader Authorization.
         # prometheusHeader:
         # - Authorization: Basic <base64Encode(<accessKey:secretKey>)>
         metricsRelistInterval: 1m # The interval at which the metric list is re-fetched. We recommend that you keep the default value of 1m.
       	logLevel: 5								# The debug log level for the component. We recommend that you keep the default value.
         adapter:
           rules:
             default: false  			# The default metric discovery configuration. We recommend that you set this to false.
             custom:
             # Example 1: This is an example of a custom metric configuration.
             # This config converts the Prometheus metric container_memory_working_set_bytes to a custom metric container_memory_working_set_bytes_per_second
             # and the CPU metric container_cpu_usage_seconds_total to container_cpu_usage_core_per_second.
             # You can run the following command to check the memory/CPU value:
             # kubectl get --raw  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kube-system/pods/*/container_memory_working_set_bytes_per_second"
             # kubectl get --raw  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kube-system/pods/*/container_cpu_usage_core_per_second"
             # For more information, see https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/horizontal-pod-autoscaling-based-on-alibaba-cloud-managed-service-for-prometheus-metrics
             - seriesQuery: 'container_memory_working_set_bytes{namespace!="",pod!=""}'
               resources:
                 overrides:
                   namespace: { resource: "namespace" }
                   pod: { resource: "pod" }
               name:
                 matches: "^(.*)_bytes"
                 as: "${1}_bytes_per_second"
               metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)' # The labelSelector in the metricsQuery configuration does not inherit the filter labels from seriesQuery.
             - seriesQuery: 'container_cpu_usage_seconds_total{namespace!="",pod!=""}'
               resources:
                 overrides:
                   namespace: { resource: "namespace" }
                   pod: { resource: "pod" }
               name:
                 matches: "^(.*)_seconds_total"
                 as: "${1}_core_per_second"
               metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)'  # The labelSelector in the metricsQuery configuration does not inherit the filter labels from seriesQuery.
             - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
               resources:
                 overrides:
                   namespace: {resource: "namespace"}
                   pod: {resource: "pod"}
               name:
                 matches: "^(.*)_total"
                 as: "${1}_per_second"
               metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
             # Example 2: This is an example of an external metric configuration.
             # For more information, see https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/configure-hpa-based-on-external-metrics
             # When adding a new conversion rule, make sure that the metric labels in Alibaba Cloud Managed Service for Prometheus are consistent with the labels here. If not, modify the metric labels in ARMS Prometheus.
             #- seriesQuery: http_requests_total{namespace!="",pod!=""}
             #  resources:
             #    overrides:
             #      # Here, resource refers to a Kubernetes API resource. You can view them by running kubectl api-resources -o wide.
             #      # Here, the key corresponds to the LabelName in the Prometheus data. Make sure this LabelName exists in your Prometheus metric data.
             #      namespace: {resource: "namespace"}
             #      pod: {resource: "pod"}
             #  name:
             #    matches: ^(.*)_total
             #   as: ${1}_per_second
             #  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)
             # This is an example of an external metric configuration.
             # For more information, see https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/configure-hpa-based-on-external-metrics
             #- seriesQuery: arms_app_requests_count
             #  metricsQuery: sum by (rpc) (sum_over_time(<<.Series>>{rpc="/demo/queryUser/{id}",service="arms-demo:arms-k8s-demo",prpc="__all__",ppid="__all__",endpoint="__all__",destId="__all__",<<.LabelMatchers>>}[1m]))
             #  name:
             #    as: ${1}_per_second_queryuser
             #    matches: ^(.*)_count
             #  resources:
             #    namespaced: false
             # This is an example of a custom metric from a user-defined Prometheus metric: http_requests_total
             # For more information, see https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/horizontal-pod-autoscaling-based-on-alibaba-cloud-managed-service-for-prometheus-metrics
             # - seriesQuery: '{__name__=~"^some_metric_count$"}'
             #   resources:
             #     template: <<.Resource>>
             #   name:
             #     matches: ""
             #     as: "my_custom_metric"
             #   metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
       ......

   The following table describes some of the fields. For a detailed description of the ack-alibaba-cloud-adapter configuration file, see ack-alibaba-cloud-metrics-adapter component configuration reference.

   Parameter	Description
   AlibabaCloudMetricsAdapter. prometheus.adapter.rules.custom	Modify the value of this field to match the content in the example YAML.
   alibabaCloudMetricsAdapter. prometheus.url	The URL of your Alibaba Cloud Managed Service for Prometheus instance. For information about how to obtain the Prometheus data request URL, see How to obtain the Prometheus data request URL.
   AlibabaCloudMetricsAdapter. prometheus.prometheusHeader[].Authorization	The authentication information. For more information, see How to obtain the Prometheus data request URL. Prometheus V1: Authentication is not required by default. If token authentication is enabled, you need to configure this field. Prometheus V2: Authentication is enabled by default. If password-free access is not enabled, you need to configure this field.
   AlibabaCloudMetricsAdapter. prometheus.adapter.rules.default	Specifies whether to create predefined metrics. We recommend that you disable this by setting the value to false.

After configuring and deploying the component, run the following commands to verify that the Kubernetes aggregation API is receiving data.

Step 3: Configure and deploy the HPA

You can expose Prometheus metrics as either custom metrics or external metrics. The following sections describe how to use both methods for HPA scaling.

ack-alibaba-cloud-metrics-adapter configuration

The ack-alibaba-cloud-metrics-adapter component converts Prometheus metrics into HPA-compatible metrics in four steps:

1. Discovery: The ack-alibaba-cloud-metric-adapter discovers available metrics from Prometheus.

2. Association: Associates the metrics with Kubernetes resources such as Pods, Nodes, and namespaces.

3. Naming: Defines the names of the converted metrics for HPA to reference.

4. Querying: Defines how to query metric data from Prometheus.

For example, the following configuration converts the http_requests_total metric from the sample-app container into http_requests_per_second for HPA:

- seriesQuery: http_requests_total{namespace!="",pod!=""}
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: ^(.*)_total
    as: ${1}_per_second
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

1. Discovery

   Specify the Prometheus metric to be converted. You can use seriesFilters to precisely filter metrics. seriesQuery can search by labels. Sample code:

   seriesQuery: http_requests_total{namespace!="",pod!=""}
   seriesFilters:
       - isNot: "^container_.*_seconds_total"

   seriesFilters is optional and is used to filter metrics:

   • is:<regex>: Matches metrics that contain the regular expression.

   • isNot:<regex>: Matches metrics that do not contain the regular expression.

2. Association

   Set the mapping between Prometheus metric labels and Kubernetes resources. The labels for the http_requests_total metric include namespace!="" and pod!="".

   - seriesQuery: http_requests_total{namespace!="",pod!=""}
     resources:
       overrides:
         namespace: {resource: "namespace"}
         pod: {resource: "pod"}

3. Naming

   Converts the Prometheus metric name to an HPA metric name, but does not change the Prometheus metric name itself. If you use the original Prometheus metric, you do not need to configure this.

   You can run the command kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" to view all available HPA metrics.

   - seriesQuery: http_requests_total{namespace!="",pod!=""}
     resources:
       overrides:
         namespace: {resource: "namespace"}
         pod: {resource: "pod"}
     name:
       matches: "^(.*)_total"
       as: "${1}_per_second"

4. Querying

   A template for querying the Prometheus API. The ack-alibaba-cloud-adapter component populates this template with parameters from the HPA, sends a request to the Prometheus API, and provides the returned value to the HPA for scaling.

   - seriesQuery: http_requests_total{namespace!="",pod!=""}
     resources:
       overrides:
         namespace: {resource: "namespace"}
         pod: {resource: "pod"}
     name:
       matches: ^(.*)_total
       as: "${1}_per_second"
     metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

Prometheus data request URL

For more information about how to obtain Prometheus data, see Add Prometheus as a data source in Grafana.

Related topics

• To implement HPA by using external metrics such as HTTP request rate or Ingress QPS, see Horizontal pod autoscaling based on Nginx Ingress component metrics.

• To use Nginx Ingress to autoscale multiple applications and dynamically adjust Pod replica counts based on application load, see Horizontally scale multiple applications based on Nginx Ingress traffic metrics.