Horizontal pod scaling based on ARMS Prometheus metrics - Best Practices| Alibaba Cloud Documentation Center

By default, Horizontal Pod Autoscaler (HPA) supports only auto scaling based on the CPU and memory usage. This cannot meet the O&M requirements. Application Real-Time Monitoring Service (ARMS) Prometheus is a fully managed monitoring service that is interfaced with the open source Prometheus ecosystem. ARMS Prometheus monitors a wide array of components and provides multiple ready-to-use dashboards. This topic describes how to convert ARMS Prometheus metrics to metrics that are supported by HPA. This way, HPA can perform auto scaling based on ARMS Prometheus metrics.

Prerequisites

Before you convert ARMS Prometheus metrics to metrics that are supported by HPA, you must install the following components:

Install the ARMS Prometheus component. For more information, see Enable ARMS Prometheus.
Install the alibaba-cloud-metrics-adapter component. For more information, see Deploy alibaba-cloud-metrics-adapter.

Example

This example shows how to configure alibaba-cloud-metrics-adapter to convert ARMS Prometheus metrics to metrics that are supported by HPA. This example also shows how to configure HPA to perform auto scaling based on ARMS Prometheus metrics.

Deploy a workload.
1. Log on to the ACK console.
2. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
3. In the left-side navigation pane of the details page, choose Workloads > Deployments.
4. On the Deployments page, click Create from YAML in the upper-right corner.
5. On the Create page, deploy an application named sample-app and a Service by using a YAML template and click Create.
  Note The application pod is used to expose the http_requests_total metric, which indicates the number of requests.
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: luxas/autoscale-demo:v0.1.2
        name: metrics-provider
        ports:
        - name: http
          containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app
  namespace: default
  labels:
    app: sample-app
spec:
  ports:
    - port: 8080
      name: http
      protocol: TCP
      targetPort: 8080
  selector:
    app: sample-app
  type: ClusterIP
```
Create a ServiceMonitor.
1. Log on to the ARMS console.
2. In the left-side navigation pane, choose Prometheus Monitoring > Prometheus Instances.
3. In the upper-left corner of the Prometheus Monitoring page, select the region in which your Container Service for Kubernetes (ACK) cluster is deployed and click the Prometheus instance that you want to manage. Then, you are redirected to the instance details page.
4. In the left-side navigation pane, click Service Discovery. Then, click the Configure tab.
5. On the Configure tab, click the ServiceMonitor tab.
6. On the ServiceMonitor tab, click Add ServiceMonitor.
  In this example, the following template is used to create a ServiceMonitor.
```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sample-app
  namespace: default
spec:
  endpoints:
  - interval: 30s
    port: http
    path: /metrics
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: sample-app
```
Confirm the status of ARMS Prometheus.
Go to the Service Discovery page and click the Targets tab. If the tab displays default/sample-app/0(1/1 up), ARMS Prometheus is monitoring the application that you deployed.

Modify the configurations of alibaba-cloud-metrics-adapter.

Log on to the ACK console.
On the Clusters page, find the cluster that you want to manage and click its name or click Details in the Actions column.
In the left-side navigation pane of the cluster details page, choose Applications > Helm.
On the Helm page, find alibaba-cloud-metrics-adapter in the Release Name column and click Update in the Actions column.

Copy the YAML content in alibaba-cloud-metrics-adapter to the editor and click OK.

The following code block shows some of the YAML content:

AlibabaCloudMetricsAdapter:
  affinity: {}
  commonLabels: ""
  env:
  - AccessKeyId: ""
  - AccessKeySecret: ""
  - Region: ""
  fullnameOverride: ""
  image:
    pullPolicy: IfNotPresent
    # The public image address. If you want to specify a private image address, replace the prefix with registry-vpc.{{RegionId}}.aliyuncs.com. 
    repository: registry.aliyuncs.com/acs/alibaba-cloud-metrics-adapter-amd64
    tag: v0.2.0-alpha-2f697ee
  listenPort: 443
  nameOverride: ""
  nodeSelector: {}
  podAnnotations: {}
  prometheus:
    adapter:
      rules:
        custom:
        # Add a conversion rule. Make sure that the labels of the ARMS Prometheus metrics are the same as the labels that are specified in the conversion rule. If the labels are not the same, modify the labels of the ARMS Prometheus metrics. 
        - seriesQuery: http_requests_total{namespace!="",pod!=""}
          resources:
            overrides:
              # The resource field specifies an API resource of Kubernetes. You can run the kubectl api-resources -o wide command to view the Kubernetes API resources. 
              # The key field specifies the LabelName of the ARMS Prometheus metric. Make sure that the ARMS Prometheus metric uses the specified LabelName. 
              namespace: {resource: "namespace"}
              pod: {resource: "pod"}
          name:
            matches: ^(.*)_total
            as: ${1}_per_second
          metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)
        default: false
    #1. Set the enabled field to true to enable the ARMS Prometheus adapter. 
    enabled: true
    logLevel: 2
    metricsRelistInterval: 1m
    tls:
      ca: ""
      certificate: ""
      enable: false
      key: ""
    #2. Specify the endpoint of the ARMS Prometheus API.  For more information, see the Obtain the Prometheus API endpoint section. 
    url: http://arms-prometheus-proxy.aliyun.com:9090/api/v1/prometheus/8cba801fff65546a3012e9a6843afd/1240538168824185/ce63742a509f948dda8ef18e75e703356/cn-shenzhen
  ramRoleType: restricted
  replicas: 1
  service:
    type: ClusterIP
  serviceAccountName: admin
  tolerations: []
ConfigReloader:
  image:
    # The public image address. If you want to specify a private image address, replace the prefix with registry-vpc.{{RegionId}}.aliyuncs.com. 
    repository: registry.aliyuncs.com/acs/configmap-reload
    tag: v0.0.1

Note

url: Specify the endpoint of the ARMS Prometheus API. For more information, see Obtain the Prometheus API endpoint.
For more information about the configuration file of ack-alibaba-cloud-adapter, see ack-alibaba-cloud-adapter configurations.

Deploy HPA.

HPA allows you to use ARMS Prometheus metrics to expose customer metrics and external metrics. You can configure HPA to perform auto scaling based on customer metrics or external metrics.

Method 1: Configure HPA to perform auto scaling based on custom metrics

Run the following command to query detailed information about the custom metrics that are supported by HPA:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second" | jq .

Returned result:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests_per_second"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "sample-app-579bc6774c-rmjfd",
        "apiVersion": "/v1"
      },
      "metricName": "http_requests_per_second",
      "timestamp": "2022-01-28T08:42:58Z",
      "value": "33m",
      "selector": null
    }
  ]
}

Use the following YAML content to deploy HPA. Then, run the kubectl apply -f hpa.yaml command to create an HPA application.

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
#Describe the object that you want HPA to scale. HPA can dynamically change the number of pods that are deployed for the object. 
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
#Specify the upper limit and lower limit of pods. 
  minReplicas: 1
  maxReplicas: 10
#Specify the metrics based on which HPA performs auto scaling. You can specify different types of metrics at the same time. 
  metrics:
  - type: Pods
    pods:
      #Specify the metric pods/http_requests. 
      metricName: http_requests_per_second
 # Specify an AverageValue type threshold. You can specify only AverageValue type thresholds for Pods metrics. 
      targetAverageValue: 500m       #If the value contains decimal places and ACK requires high precision, you can use the m or k unit. For example, 1001m is equal to 1.001 and 1k is equal to 1000.

Perform a stress test after you enable load balancing for the Service.

Run the following command to perform a stress test:
```
ab -c 50 -n 2000 LoadBalancer(sample-app):8080/
```

Run the following command to query information about HPA:

kubectl get hpa sample-app

Expected output:

NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   33m/500m   1         10        1          7m

Method 2: Configure HPA to perform auto scaling based on external metrics

Run the following command to query detailed information about the external metrics that are supported by HPA:

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/http_requests_per_second" | jq .

Returned result:

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "http_requests_per_second",
      "metricLabels": {},
      "timestamp": "2022-01-28T08:40:20Z",
      "value": "33m"
    }
  ]
}

Use the following YAML content to deploy HPA. Then, run the kubectl apply -f hpa.yaml command to deploy an HPA application.

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: http_requests_per_second
          selector:
            matchLabels:
              job: "sample-app"
#You can specify only Value and AverageValue type thresholds for external metrics. 
        target:
          type: AverageValue
          averageValue: 500m

Perform a stress test after you enable load balancing for the Service.

Run the following command to perform a stress test:
```
ab -c 50 -n 2000 LoadBalancer(sample-app):8080/
```

Run the following command to query information about HPA:

kubectl get hpa sample-app

Expected output:

NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
sample-app   Deployment/sample-app   33m/500m   1         10        1          7m

ack-alibaba-cloud-adapter configurations

ack-alibaba-cloud-adapter performs the following steps to convert ARMS Prometheus metrics to metrics that are supported by HPA:

Discovery: discovers ARMS Prometheus metrics that can be used by HPA.
Association: associates the metrics with Kubernetes resources, such as pods, nodes, and namespaces.
Naming: defines the names of the converted metrics for HPA.
Querying: defines the template of the requests that are sent to the ARMS Prometheus API.

In the preceding example, the http_requests_total metric that is exposed by the sample-app pod is converted to the http_requests_per_second metric for HPA. The following code block shows the configurations of ack-alibaba-cloud-adapter that is used in the example:

- seriesQuery: http_requests_total{namespace!="",pod!=""}
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: ^(.*)_total
    as: ${1}_per_second
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

Note

seriesQuery: the Prometheus Query Language (PromQL) query data.
metricsQuery: aggregates the PromQL query data in seriesQuery.
resources: maps the labels in the PromQL query data to resources. The resources field specifies Kubernetes API resources such as pods, namespaces, and nodes. You can run the kubectl api-resources -o wide command to query Kubernetes API resources. The key field specifies the LabelName of the ARMS Prometheus metric. Make sure that the ARMS Prometheus metric uses the specified LabelName.
name: uses a regular expression to convert the name of the ARMS Prometheus metric to a name that is easy to identify. In this example, the metric name is converted from http_request_total to http_request_per_second.

Discovery
Specify the ARMS Prometheus metric that you want to convert. You can use seriesFilters to filter metrics. You can use seriesQuery to search metrics by label, as shown in the following code block:
```
seriesQuery: http_requests_total{namespace!="",pod!=""}
seriesFilters:
    - isNot: "^container_.*_seconds_total"
```
Note seriesFilters is optional. You can use seriesFilters to filter metrics:
- is: matches metrics that contain <regex>.
- isNot: matches metrics that do not contain <regex>.
Association
Map the labels of ARMS Prometheus metrics to Kubernetes resources The labels of the http_requests_total metric are kubernetes_namespace!="" and kubernetes_pod_name!="".
```
- seriesQuery: http_requests_total{namespace!="",pod!=""}
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
```
Naming
Name the HPA metrics that are converted from the ARMS Prometheus metrics. The names of the ARMS Prometheus metrics remain unchanged. You do not need to configure the naming settings if you directly use the ARMS Prometheus metrics.

Note You can run the kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" command to query the metrics that are supported by HPA.
```
- seriesQuery: http_requests_total{namespace!="",pod!=""}
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"
```
Querying
The template of requests that are sent to the ARMS Prometheus API. ack-alibaba-cloud-adapter passes parameters in HPA to the request template, sends a request to the ARMS Prometheus API based on the template, and then sends the returned parameter values to HPA for auto scaling.
```
- seriesQuery: http_requests_total{namespace!="",pod!=""}
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: ^(.*)_total
    as: ${1}_per_second
  metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)
```

Obtain the Prometheus API endpoint

For ARMS Prometheus

Log on to the ARMS console.
In the left-side navigation pane, choose Prometheus Monitoring > Prometheus Instances.
In the upper-left corner of the Prometheus Monitoring page, select the region in which your ACK cluster is deployed and click the Prometheus instance that you want to manage. Then, you are redirected to the instance details page.
In the left-side navigation pane, click Settings. On the page that appears, click the Settings tab.
On the Settings tab, record the HTTP API endpoint. This endpoint is used to import monitoring data to Grafana.
We recommend that you call the ARMS Prometheus API over an internal network. You can call the API over the Internet if no internal network is available.

For open source Prometheus

Deploy Prometheus.
1. Log on to the ACK console.
2. In the left-side navigation pane of the ACK console, choose Marketplace > App Catalog.
3. On the Marketplace page, click the App Catalog tab. Then, find and click ack-prometheus-operator.
4. On the ack-prometheus-operator page, click Deploy.
5. In the Deploy wizard, select a cluster and a namespace, and then click Next.
6. On the Parameters wizard page, set the parameters and click Next.
  Check the deployment result.
  1. Run the following command to map Prometheus in the cluster to local port 9090:
```
kubectl port-forward svc/ack-prometheus-operator-prometheus 9090:9090 -n monitoring
```
  2. Enter localhost:9090 in the address bar of a browser to visit the Prometheus page.
  3. In the top navigation bar, choose Status > Targets to view all data collection tasks. Tasks in the UP state are running as normal.
Check service and namespace in the Labels column.
If ServiceName is ack-prometheus-operator-prometheus and ServiceNamespace is monitoring, the endpoint of the open source Prometheus API is:
```
http://ack-prometheus-operator-prometheus.monitoring.svc.cluster.local:9090
```