Auto scaling based on GPU metrics - Container Service for Kubernetes

Use Kubernetes Custom Metrics and Managed Service for Prometheus to collect GPU metrics and auto scale containers with HPA.

Prerequisites

A GPU node is added to your cluster, or a dedicated GPU cluster is created.

How it works

GPUs accelerate computation in scenarios such as deep learning model training and inference. Auto scaling based on GPU metrics such as utilization and memory helps optimize costs.

Kubernetes Horizontal Pod Autoscaler (HPA) uses CPU and memory metrics by default. For GPU metrics, the Prometheus Adapter fetches data from Managed Service for Prometheus and exposes it through the Custom Metrics API. HPA then scales workloads based on GPU utilization, memory usage, and other indicators. The following diagram illustrates the process.

Step 1: Deploy Prometheus and the metrics adapter

Enable Prometheus monitoring for your cluster.
Note
Skip this step if Prometheus was installed during cluster creation.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Prometheus Monitoring.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.
In the left navigation pane, choose Managed Service for Prometheus > Instances.
Install and configure ack-alibaba-cloud-metrics-adapter.
1. Get the HTTP API URL
1. Log on to the ARMS console.
2. Select the region of your ACK cluster. Click the target instance.
3. On the Settings page, on the Settings tab, copy the internal network address from the HTTP API Address (Grafana Read Address).
2. Configure the Prometheus URL
1. Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.
2. On the Marketplace page, click the App Catalog tab. Search for and click ack-alibaba-cloud-metrics-adapter.
3. On the ack-alibaba-cloud-metrics-adapter page, click Deploy.
4. In the Basic Information wizard, select your cluster and namespace. Click Next.
5. In the Parameters wizard, select a Chart Version. In the Parameters section, set the url parameter to the copied HTTP API URL. Click OK.

Step 2: Configure adapter rules

1. Query GPU metrics

For available GPU metrics, see Monitoring metric descriptions.

2. Configure adapter rules

Log on to the ACK console. In the left navigation pane, click Clusters.

In the Helm list, find ack-alibaba-cloud-metrics-adapter. In the Actions column, click Update. Add the following rules to the custom field.

Click to expand the rules

- metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NodeName:
        resource: node
  seriesQuery: DCGM_FI_DEV_GPU_UTIL{} # GPU utilization.
- metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NamespaceName:
        resource: namespace
      NodeName:
        resource: node
      PodName:
        resource: pod
  seriesQuery: DCGM_CUSTOM_PROCESS_SM_UTIL{} # Container GPU utilization.
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NodeName:
        resource: node
  seriesQuery: DCGM_FI_DEV_FB_USED{} # GPU memory usage.
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NamespaceName:
        resource: namespace
      NodeName:
        resource: node
      PodName:
        resource: pod
  seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{} # Container GPU memory usage.
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>) / sum(DCGM_CUSTOM_CONTAINER_MEM_ALLOCATED{}) by (<<.GroupBy>>)
  name:
    as: ${1}_GPU_MEM_USED_RATIO
    matches: ^(.*)_MEM_USED
  resources:
    overrides:
      NamespaceName:
        resource: namespace
      PodName:
        resource: pod
  seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{NamespaceName!="",PodName!=""}  # Container GPU memory utilization.

Full configuration after adding the rules:

Chart Url  https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/ack-alibaba-cloud-metrics-adapter-1.3.2.tgz
  adapter:
    rules:
      custom:
      - metricsQuery: sum(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}) by (&lt;&lt;.GroupBy&gt;&gt;)
        name:
          as: ${1}_bytes_per_second
          matches: ^(.*)_bytes
        resources:
          overrides:
            namespace:
              resource: namespace
            pod:
              resource: pod
        seriesQuery: container_memory_working_set_bytes{namespace!="",pod!=""}
      - metricsQuery: sum(rate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[1m])) by (&lt;&lt;.GroupBy&gt;&gt;)
        name:
          as: ${1}_core_per_second
          matches: ^(.*)_seconds_total
        resources:
          overrides:
            namespace:
              resource: namespace
            pod:
              resource: pod
        seriesQuery: container_cpu_usage_seconds_total{namespace!="",pod!=""}
      - metricsQuery: &lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}
        resources:
          overrides:
            NodeName:
              resource: node
        seriesQuery: DCGM_FI_DEV_GPU_UTIL{} # GPU utilization.
      - metricsQuery: &lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}
        resources:
          overrides:
            NamespaceName:
              resource: namespace
            NodeName:
              resource: node
            PodName:
              resource: pod
        seriesQuery: DCGM_CUSTOM_PROCESS_SM_UTIL{} # Container GPU utilization.
      - metricsQuery: &lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}
        resources:
          overrides:
            NodeName:
              resource: node
        seriesQuery: DCGM_FI_DEV_FB_USED{} # GPU memory usage.
      - metricsQuery: &lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}
        resources:
          overrides:
            NamespaceName:
              resource: namespace
            NodeName:
              resource: node
            PodName:
              resource: pod
        seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{} # Container GPU memory usage.
      - metricsQuery: sum(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;})by(&lt;&lt;.GroupBy&gt;&gt;) / sum(DCGM_CUSTOM_CONTAINER_MEM_ALLOCATED{})by(&lt;&lt;.GroupBy&gt;&gt;)
        name:
          as: ${1}_GPU_MEM_USED_RATIO
          matches: ^(.*)_MEM_USED
        resources:
          overrides:
            NamespaceName:
              resource: namespace
            PodName:
              resource: pod
        seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{NamespaceName!="",PodName!=""}  # Container GPU memory utilization.
      default: false
    enabled: true
    logLevel: 5

Verify the configuration. The output should contain metrics such as DCGM_FI_DEV_GPU_UTIL, DCGM_CUSTOM_PROCESS_SM_UTIL, DCGM_FI_DEV_FB_USED, and DCGM_CUSTOM_PROCESS_MEM_USED. The following example highlights DCGM_CUSTOM_PROCESS_SM_UTIL ; your actual output may vary.

Click to expand the output example

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

Expected output: The resources array contains DCGM_CUSTOM_PROCESS_SM_UTIL metrics.

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nodes/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "namespaces/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "DCGM_CUSTOM_PROCESS_GPU_MEM_USED_RATIO",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Step 3: Implement auto scaling

Test auto scaling by deploying a GPU inference service, running a stress test, and observing GPU-based scaling behavior.

1. Deploy the inference service

Deploy the inference service.

Click to expand the command details

cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bert-intent-detection
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bert-intent-detection
  template:
    metadata:
      labels:
        app: bert-intent-detection
    spec:
      containers:
      - name: bert-container
        image: registry.cn-hangzhou.aliyuncs.com/ai-samples/bert-intent-detection:1.0.1
        ports:
        - containerPort: 80
        resources:
          limits:
            nvidia.com/gpu: 1
---
apiVersion: v1
kind: Service
metadata:
  name: bert-intent-detection-svc
  labels:
    app: bert-intent-detection
spec:
  selector:
    app: bert-intent-detection
  ports:
  - protocol: TCP
    name: http
    port: 80
    targetPort: 80
  type: LoadBalancer
EOF

Check Pod and Service status.

Check the Pod status.

kubectl get pods -o wide

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE     IP           NODE                        NOMINATED NODE   READINESS GATES
bert-intent-detection-7b486f6bf-f****   1/1     Running   0          3m24s   10.15.1.17   cn-beijing.192.168.94.107   <none>           <none>

One Pod is deployed on GPU node 192.168.94.107.

Check the Service status.

kubectl get svc bert-intent-detection-svc

Expected output:

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
bert-intent-detection-svc   LoadBalancer   172.16.186.159   47.95.XX.XX   80:30118/TCP   5m1s

The output shows the service name, confirming that the Service deployed successfully.

Log on to GPU node 192.168.94.107 via SSH and check GPU usage.

nvidia-smi

Expected output:

Wed Feb 16 11:48:07 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:07.0 Off |                    0 |
| N/A   32C    P0    55W / 300W |  15345MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   2305118      C   python                          15343MiB |
+-----------------------------------------------------------------------------+

The output shows the inference service running on the GPU. GPU utilization is 0% because no requests have been received yet.

Verify the inference service deployment.

curl -v  "http://47.95.XX.XX/predict?query=Music"

Expected output:

*   Trying 47.95.XX.XX...
* TCP_NODELAY set
* Connected to 47.95.XX.XX (47.95.XX.XX) port 80 (#0)
> GET /predict?query=Music HTTP/1.1
> Host: 47.95.XX.XX
> User-Agent: curl/7.64.1
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 9
< Server: Werkzeug/1.0.1 Python/3.6.9
< Date: Wed, 16 Feb 2022 03:52:11 GMT
<
* Closing connection 0
PlayMusic # Intent recognition result.

A 200 status code with an intent recognition result confirms successful deployment.

2. Configure the HPA

This example triggers scale-out when Pod GPU utilization exceeds 20%. The following table lists HPA-supported metrics.

Metric	Description	Unit
DCGM_FI_DEV_GPU_UTIL	GPU card utilization. This metric applies only to dedicated GPU scheduling. Important In shared GPU scenarios, NVIDIA provides only card-level utilization, not per-application metrics. Running `nvidia-smi` within a Pod shows utilization of the entire card.	%
DCGM_FI_DEV_FB_USED	GPU card memory usage. This metric applies only to dedicated GPU scheduling.	MiB
DCGM_CUSTOM_PROCESS_SM_UTIL	Container GPU utilization.	%
DCGM_CUSTOM_PROCESS_MEM_USED	Container GPU memory usage.	MiB
DCGM_CUSTOM_PROCESS_GPU_MEM_USED_RATIO	Container GPU memory utilization. `Container GPU memory utilization = Actual GPU memory used by the container / GPU memory allocated to the container`	%

Deploy the HPA.

Cluster version ≥ 1.23

cat <<EOF | kubectl create -f -
apiVersion: autoscaling/v2  # Use the autoscaling/v2 HPA configuration.
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bert-intent-detection
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: DCGM_CUSTOM_PROCESS_SM_UTIL
      target:
        type: Utilization
        averageValue: 20 # Triggers a scale-out when the container's GPU utilization exceeds 20%.
EOF

Cluster version < 1.23

cat <<EOF | kubectl create -f -
apiVersion: autoscaling/v2beta1  # Use the autoscaling/v2beta1 HPA configuration.
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bert-intent-detection
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: DCGM_CUSTOM_PROCESS_SM_UTIL # GPU utilization of the Pod.
      targetAverageValue: 20 # Triggers a scale-out when the container's GPU utilization exceeds 20%.
EOF

Check the HPA status.

kubectl get hpa

Expected output:

NAME      REFERENCE                          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
gpu-hpa   Deployment/bert-intent-detection   0/20      1         10        1          74s

TARGETS shows 0/20, meaning current GPU utilization is 0. Auto scaling triggers when utilization exceeds 20%.

3. Test auto scaling

Test scale-out

Run a stress test.
```
hey -n 10000 -c 200 "http://47.95.XX.XX/predict?query=music"
```
Note
HPA calculates desired replicas with: Desired Replicas = ceil[Current Replicas * (Current Metric / Desired Metric)]. For example, with 1 replica at metric value 23 and target 20, the result is 2 replicas.

During the stress test, observe HPA and Pod status.

Check the HPA status.

kubectl get hpa

Expected output:

NAME      REFERENCE                          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
gpu-hpa   Deployment/bert-intent-detection   23/20     1         10        2          7m56s

TARGETS shows 23/20. GPU utilization exceeds the 20% target, triggering scale-out.

Check the Pod status.

kubectl get pods

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
bert-intent-detection-7b486f6bf-f****   1/1     Running   0          44m
bert-intent-detection-7b486f6bf-m****   1/1     Running   0          14s

Two Pods are running, matching the target of 2.

Scale-out confirmed.

Test scale-in

After the stress test stops, GPU utilization drops below 20% and scale-in begins.

Check the HPA status.

kubectl get hpa

Expected output:

NAME      REFERENCE                          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
gpu-hpa   Deployment/bert-intent-detection   0/20      1         10        1          15m

TARGETS shows 0/20 , meaning GPU utilization is 0. After about 5 minutes, scale-in begins.

Check the Pod status.

kubectl get pods

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
bert-intent-detection-7b486f6bf-f****   1/1     Running   0          52m

One Pod remains, confirming scale-in.

FAQ

How to check GPU usage?

On the GPU Monitoring tab, observe GPU utilization trends: increasing utilization indicates an active card, while a flat trend means idle. To view the trend:

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Prometheus Monitoring page, click the GPU Monitoring tab.