Container Service for Kubernetes:Multi-Cluster HPA Practice Using vLLM Custom Metrics

How it works

In large-scale inference scenarios, LLM online services face sharp and unpredictable traffic spikes. A multi-cluster architecture is a common solution for handling peak traffic:

• Users with on-premises data centers typically use a hybrid cloud architecture. During traffic peaks, they scale out workloads to cloud clusters.

• Cloud users often deploy multiple clusters across different regions to avoid resource shortages in a single region.

The ACK One fleet manages resources across multiple clusters and supports cross-cluster scheduling and Auto Scaling. It supports both deployment models through three mechanisms:

• Multi-cluster priority-based scheduling: Set scheduling priorities for clusters. Replicas go to high-priority clusters first. When resources run low, the fleet scales out to low-priority clusters and scales in by removing replicas from low-priority clusters first.

• Inventory-aware intelligent scheduling: The fleet cluster works with GoatScaler in child clusters and uses ECS inventory to schedule replicas intelligently.

• Unified centralized Auto Scaling: Create a FederatedHPA to scale workloads across clusters from a central point. The fleet cluster's Metrics Adapter collects and aggregates metrics from child clusters (which also support Prometheus Adapter), then scales workloads based on the aggregated metrics.

This tutorial demonstrates the following scenario:

• cluster1 is the high-priority cluster; cluster2 is the low-priority cluster.

• When traffic increases, FederatedHPA triggers a scale-out event.

• New replicas are scheduled to cluster2 based on available capacity.

• After traffic subsides, FederatedHPA scales the deployment back to the minimum replica count.

Prerequisites

Before you begin, make sure you have:

• Two ACK clusters, each with a GPU node pool and at least one initialized GPU instance

• Both clusters added to an ACK One fleet, with the ack-alibaba-cloud-metrics-adapter component installed (for registered clusters, install the open-source Prometheus Adapter instead)

• The kubectl amc plugin installed

Step 1: Configure metric collection in child clusters

Configure the ack-alibaba-cloud-metrics-adapter component in both ACK clusters to expose vLLM custom metrics.

1. Log on to the Container Service Management Console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.

3. Locate ack-alibaba-cloud-metrics-adapter and click Update in the Actions column. In the prometheus.adapter.rules field, add the following metric collection rules:

   • vllm:num_requests_waiting: Number of requests waiting to be processed, exposed as num_requests_waiting

   • vllm:num_requests_running: Number of requests currently being processed, exposed as num_requests_running

   Variable	Description
   <<.Series>>	The metric series name matched by seriesQuery
   <<.LabelMatchers>>	Label filters generated from the resource overrides (namespace, Pod name)
   <<.GroupBy>>	The grouping labels for aggregation

   rules:
     - seriesQuery: 'vllm:num_requests_waiting'
       resources:
         overrides:
           kubernetes_namespace: {resource: "namespace"}
           kubernetes_pod_name: {resource: "pod"}
       name:
         matches: 'vllm:num_requests_waiting'
         as: 'num_requests_waiting'
       metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'
     - seriesQuery: 'vllm:num_requests_running'
       resources:
         overrides:
           kubernetes_namespace: {resource: "namespace"}
           kubernetes_pod_name: {resource: "pod"}
       name:
         matches: 'vllm:num_requests_running'
         as: 'num_requests_running'
       metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

   The template variables in metricsQuery work as follows:

4. Click OK.

5. Verify that the custom metrics are registered correctly:

   kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/qwen3-xxxxx/num_requests_waiting"
   kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/qwen3-xxxxx/num_requests_running"

   If the output includes vLLM metric values, the configuration succeeded.

Step 2: Deploy the vLLM inference service in the fleet

Save the following YAML as deployment.yaml and run kubectl apply -f deployment.yaml.

The Deployment uses the Qwen/Qwen3-0.6B model downloaded from ModelScope, served on port 8000. Prometheus scrape annotations on the Pod enable automatic metric collection without a separate PodMonitor.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3
  namespace: test
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: qwen3
  template:
    metadata:
      annotations:
        # Enables Prometheus scraping — equivalent to a PodMonitor
        prometheus.io/path: /metrics
        prometheus.io/port: "8000"
        prometheus.io/scrape: "true"
      labels:
        app: qwen3
    spec:
      containers:
      # Serves Qwen3-0.6B downloaded from ModelScope
      - command:
        - sh
        - -c
        - export VLLM_USE_MODELSCOPE=True; vllm serve Qwen/Qwen3-0.6B --served-model-name
          qwen3-0.6b --port 8000 --trust-remote-code --tensor_parallel_size=1 --max-model-len
          2048 --gpu-memory-utilization 0.8
        image: kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm-openai:v0.9.1
        imagePullPolicy: IfNotPresent
        name: vllm
        ports:
        - containerPort: 8000
          name: restful
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8000
          timeoutSeconds: 1
        resources:
          limits:
            nvidia.com/gpu: "1"
          requests:
            nvidia.com/gpu: "1"
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

Step 3: Configure multi-cluster distribution policy in the fleet

Save the following YAML as propagationpolicy.yaml and run kubectl apply -f propagationpolicy.yaml.

Key fields:

• autoScaling.ecsProvision: Enables inventory-aware intelligent scheduling. The fleet provisions ECS instances automatically when existing GPU instances are insufficient.

• clusterAffinities: Defines priority-based cluster groups. The scheduler fills groups in order—high-priority first, then low-priority when the high-priority group runs out of capacity.

apiVersion: policy.one.alibabacloud.com/v1alpha1
kind: PropagationPolicy
metadata:
  name: vllm-deploy-pp
  namespace: test
spec:
  autoScaling:
    ecsProvision: true
  placement:
    clusterAffinities:
    - affinityName: high-priority
      clusterNames:
      - ${cluster1_id}
    - affinityName: low-priority
      clusterNames:
      - ${cluster2_id}
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Weighted
      weightPreference:
        dynamicWeight: AvailableReplicas
  preserveResourcesOnDeletion: false
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    namespace: test
  - apiVersion: v1
    kind: Service
    namespace: test
  schedulerName: default-scheduler

Step 4: Create a FederatedHPA in the fleet

FederatedHPA monitors CPU, memory, custom metrics, and external metrics for Pods. This example uses the vLLM custom metrics num_requests_waiting and num_requests_running to trigger scaling.

Save the following YAML as federatedhpa.yaml and run kubectl apply -f federatedhpa.yaml.

apiVersion: autoscaling.one.alibabacloud.com/v1alpha1
kind: FederatedHPA
metadata:
  name: vllm-fhpa
  namespace: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qwen3
  minReplicas: 1
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 30
    scaleUp:
      stabilizationWindowSeconds: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: num_requests_waiting
        target:
          type: AverageValue
          averageValue: ${waiting_size} # Optimal value varies by GPU and model
    - type: Pods
      pods:
        metric:
          name: num_requests_running
        target:
          type: AverageValue
          averageValue: ${running_size} # Optimal value varies by GPU and model

Key fields in the metrics block:

Verify results

kubectl scale deployment -n test qwen3 --replicas=2

FAQ

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

What's next

• Manage north-south traffic

• FederatedHPA