Container Service for Kubernetes:Implement auto scaling based on GPU metrics

Enable Prometheus monitoring

Enable Prometheus monitoring for your cluster.

Skip this step if you selected to install Prometheus when you created the cluster.

Install ack-alibaba-cloud-metrics-adapter

Get the Grafana read address

1. Log on to the ACK consoleACK consoleACK consoleARMS console.

2. In the left navigation pane, choose Managed Service for Prometheus > Instances.

3. At the top of the page, select the region where your Container Service for Kubernetes (ACK) cluster is located, then click the name of the target instance.

4. On the Settings page, click the Settings tab. In the HTTP API URL (Grafana Read Address) section, copy the internal network address.

Configure the Prometheus URL

1. Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.

2. On the App Marketplace page, click the App Catalog tab. Search for and click ack-alibaba-cloud-metrics-adapter.

3. On the ack-alibaba-cloud-metrics-adapter page, click Deploy.

4. In the Basic Information wizard, select a cluster and namespace, then click Next.

5. In the Parameters wizard, select a Chart Version. In the Parameters section, set the Prometheus url parameter to the HTTP API address you copied, then click OK.

Available GPU metrics

DCGM_CUSTOM_PROCESS_GPU_MEM_USED_RATIO is calculated as:

GPU memory utilization = Actual GPU memory usage of the current pod container (Used) / GPU memory allocated to the current pod container (Allocated)

For the full list of available metrics, see Monitoring metric descriptions.

Add adapter rules

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.

3. In the Actions column for the ack-alibaba-cloud-metrics-adapter Helm release, click Update. Add the following rules to the custom field.

   - metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
     resources:
       overrides:
         NodeName:
           resource: node
     seriesQuery: DCGM_FI_DEV_GPU_UTIL{} # GPU utilization
   - metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
     resources:
       overrides:
         NamespaceName:
           resource: namespace
         NodeName:
           resource: node
         PodName:
           resource: pod
     seriesQuery: DCGM_CUSTOM_PROCESS_SM_UTIL{} # Container GPU utilization.
   - metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
     resources:
       overrides:
         NodeName:
           resource: node
     seriesQuery: DCGM_FI_DEV_FB_USED{} # GPU memory usage.
   - metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
     resources:
       overrides:
         NamespaceName:
           resource: namespace
         NodeName:
           resource: node
         PodName:
           resource: pod
     seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{} # Container GPU memory usage.
   - metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>) / sum(DCGM_CUSTOM_CONTAINER_MEM_ALLOCATED{}) by (<<.GroupBy>>)
     name:
       as: ${1}_GPU_MEM_USED_RATIO
       matches: ^(.*)_MEM_USED
     resources:
       overrides:
         NamespaceName:
           resource: namespace
         PodName:
           resource: pod
     seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{NamespaceName!="",PodName!=""}  # Container GPU memory utilization.

   After saving, the configuration looks like the following:

   

Verify that custom metrics are registered

Run the following command to confirm HPA can see the GPU metrics:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

The output lists all registered custom metrics. A successful configuration includes entries for DCGM_FI_DEV_GPU_UTIL, DCGM_CUSTOM_PROCESS_SM_UTIL, DCGM_FI_DEV_FB_USED, and DCGM_CUSTOM_PROCESS_MEM_USED. The following shows an example for DCGM_CUSTOM_PROCESS_SM_UTIL:

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nodes/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "DCGM_CUSTOM_PROCESS_GPU_MEM_USED_RATIO",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Configure HPA

HPA calculates the desired replica count using the following formula:

Desired Replicas = ceil[Current Replicas × (Current Metric / Desired Metric)]

For example, with 1 current replica, a current utilization of 23, and a target of 20: ceil(1 x 23/20) = 2.

This example triggers scale out when container GPU utilization (DCGM_CUSTOM_PROCESS_SM_UTIL) exceeds 20%. Create the HPA resource. The API version depends on your Kubernetes version.

cat <<EOF | kubectl create -f -
apiVersion: autoscaling/v2  # Use the autoscaling/v2 version for HPA configuration.
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bert-intent-detection
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: DCGM_CUSTOM_PROCESS_SM_UTIL
      target:
        type: Utilization
        averageValue: 20 # A scale-out is triggered when the container's GPU utilization exceeds 20%.
EOF

cat <<EOF | kubectl create -f -
apiVersion: autoscaling/v2beta1  # Use the autoscaling/v2beta1 version for HPA configuration.
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bert-intent-detection
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: DCGM_CUSTOM_PROCESS_SM_UTIL # GPU utilization of the pod.
      targetAverageValue: 20 # A scale-out is triggered when the container's GPU utilization exceeds 20%.
EOF

Confirm the HPA is active and reading the metric:

kubectl get hpa

Expected output:

NAME      REFERENCE                          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
gpu-hpa   Deployment/bert-intent-detection   0/20      1         10        1          74s

TARGETS shows 0/20, meaning current GPU utilization is 0% against a 20% threshold.

How can I tell if a GPU card is being used?

Check the GPU Monitoring tab in the ACK console. A GPU card under load shows increasing utilization; an idle card stays flat.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Prometheus Monitoring.

3. On the Prometheus Monitoring page, click the GPU Monitoring tab. Observe the GPU card utilization graph.