Container Service for Kubernetes:Implement auto scaling based on GPU metrics

Enable Prometheus monitoring

Enable Alibaba Cloud Prometheus monitoring for your ACK cluster.

When creating a cluster, select Enable Managed Service for Prometheus to skip this step.

Install ack-alibaba-cloud-metrics-adapter

A. Get the HTTP API endpoint

1. Log on to the ARMS console.

2. In the left-side navigation pane, choose Managed Service for Prometheus > Instances.

3. In the top navigation bar, select the region where your ACK cluster is deployed, and click the name of the Prometheus instance used by your cluster.

4. On the Settings page, click the Settings tab and copy the internal endpoint from the HTTP API URL section.

B. Configure the Prometheus URL

1. Log on to the ACK console. In the left-side navigation pane, choose Marketplace > Marketplace.

2. On the Marketplace page, click the App Catalog tab, then find and click ack-alibaba-cloud-metrics-adapter.

3. On the ack-alibaba-cloud-metrics-adapter page, click Deploy.

4. On the Basic Information wizard page, select a cluster and a namespace, then click Next.

5. On the Parameters wizard page, select a chart version from the Chart Version drop-down list. In the Parameters section, set the Prometheus URL to the HTTP API endpoint you copied, then click OK.

Available GPU metrics

For the full list of GPU metrics collected by Managed Service for Prometheus, see Introduction to metrics.

Add rules to the adapter

1. Log on to the ACK console. In the left-side navigation pane, click ACK consoleACK consoleACK consoleClusters.

2. On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.

3. On the Helm page, click Update in the Actions column of ack-alibaba-cloud-metrics-adapter. Under custom, add the following rules:

- metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NodeName:
        resource: node
  seriesQuery: DCGM_FI_DEV_GPU_UTIL{} # GPU utilization (node-level, exclusive mode only)
- metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NamespaceName:
        resource: namespace
      NodeName:
        resource: node
      PodName:
        resource: pod
  seriesQuery: DCGM_CUSTOM_PROCESS_SM_UTIL{} # GPU utilization per pod
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NodeName:
        resource: node
  seriesQuery: DCGM_FI_DEV_FB_USED{} # GPU memory used (node-level, exclusive mode only)
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
  resources:
    overrides:
      NamespaceName:
        resource: namespace
      NodeName:
        resource: node
      PodName:
        resource: pod
  seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{} # GPU memory used per pod
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>) / sum(DCGM_CUSTOM_CONTAINER_MEM_ALLOCATED{}) by (<<.GroupBy>>)
  name:
    as: ${1}_GPU_MEM_USED_RATIO
    matches: ^(.*)_MEM_USED
  resources:
    overrides:
      NamespaceName:
        resource: namespace
      PodName:
        resource: pod
  seriesQuery: DCGM_CUSTOM_PROCESS_MEM_USED{NamespaceName!="",PodName!=""} # GPU memory utilization per pod (used/allocated)

The following figure shows an example configuration.

The table below describes each rule field.

Verify the rules

Run the following command. If the output includes DCGM_FI_DEV_GPU_UTIL, DCGM_CUSTOM_PROCESS_SM_UTIL, DCGM_FI_DEV_FB_USED, and DCGM_CUSTOM_PROCESS_MEM_USED, the rules are active.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

The following shows an example output where DCGM_CUSTOM_PROCESS_SM_UTIL appears scoped to nodes, pods, and namespaces:

{
  [
    ...
    {
      "name": "nodes/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    ...
    {
      "name": "pods/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    ...
    {
      "name": "namespaces/DCGM_CUSTOM_PROCESS_SM_UTIL",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
    ...
    {
      "name": "DCGM_CUSTOM_PROCESS_GPU_MEM_USED_RATIO",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
    ...
  ]
}

Configure the HPA

The table below describes the GPU metrics available to the HPA.

The following example triggers scale-out when pod GPU utilization (DCGM_CUSTOM_PROCESS_SM_UTIL) exceeds 20%.

cat <<EOF | kubectl create -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bert-intent-detection
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: DCGM_CUSTOM_PROCESS_SM_UTIL
      target:
        type: Utilization
        averageValue: 20 # Scale out when average GPU utilization across pods exceeds 20%
EOF

cat <<EOF | kubectl create -f -
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bert-intent-detection
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: DCGM_CUSTOM_PROCESS_SM_UTIL
      targetAverageValue: 20 # Scale out when average GPU utilization across pods exceeds 20%
EOF

Verify the HPA is active:

kubectl get hpa

Expected output:

NAME      REFERENCE                          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
gpu-hpa   Deployment/bert-intent-detection   0/20      1         10        1          74s

TARGETS shows 0/20—current GPU utilization is 0, and the HPA will scale out once it exceeds 20%.

Test auto scaling

How do I confirm whether a GPU is in use?

Check the GPU Monitoring tab in Prometheus Monitoring. An increase in GPU utilization indicates the GPU is active; a flat line indicates no workload is running.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Operations > Prometheus Monitoring.

3. On the Prometheus Monitoring page, click the GPU Monitoring tab and observe the GPU utilization trend.