Configure an elastic scaling policy for a PD-decoupled inference service - Container Service for Kubernetes

In a large language model (LLM) inference architecture with Prefill-Decode (PD) separation, the Prefill and Decode stages have fundamentally different resource profiles. Prefill is a compute-intensive phase that processes input tokens and builds the key-value (KV) cache — its duration scales with input length. Decode is a memory-bandwidth-bound phase that generates one token per step. Because the two stages respond to load differently, traditional metrics such as CPU or GPU utilization cannot reliably signal when to scale either stage. This topic describes how to use KEDA and the NATS queue backlog to autoscale the Prefill role to allocate resources on demand and optimize service cost and performance.

The NATS message queue backlog is a more accurate signal for the Prefill stage: each pending message in the dynamo_prefill_queue stream represents one inference request waiting for a Prefill worker. When the backlog grows, Prefill capacity is saturated and additional workers are needed.

This topic shows how to use KEDA (Kubernetes Event-driven Autoscaling) to autoscale the Prefill role in a Dynamo PD-decoupled inference service, triggered by the NATS JetStream message backlog. The Dynamo framework is used as an example. For other frameworks, adjust the NATS stream name and consumer name accordingly.

Prerequisites

Before you begin, ensure that you have:

Limitations

This scaling solution applies only to the Prefill role. The Decode role requires a separate scaling policy. For the Decode role, it is recommended to use GPU memory utilization as the scaling metric.
This example is based on the Dynamo inference framework. For other frameworks, adjust the NATS stream name and consumer name in the ScaledObject configuration.

How it works

For PD-decoupled inference services deployed with a RoleBasedGroup (RBG), each role (Prefill, Decode, Processor) can be scaled independently. The scaling path works as follows:

Pending inference requests are pushed as messages to the dynamo_prefill_queue stream in NATS JetStream.
Prefill instances act as consumers and pull messages from the queue.
KEDA's NATS JetStream Scaler monitors the queue backlog (the number of unprocessed messages).
When the backlog exceeds the lagThreshold, KEDA signals the Horizontal Pod Autoscaler (HPA) to scale out the Prefill role.

A RoleBasedGroupScalingAdapter bridges KEDA and the RBG. Because KEDA requires a standard scale subresource to drive the HPA, the ScalingAdapter implements this interface for the bound RBG role.

Scaling behavior when `minReplicaCount` is 1 or higher

KEDA has two scaling phases: an activation phase (0 to 1 replica) and a scaling phase (1 to N replicas, driven by HPA). When minReplicaCount is set to 1 or higher, the activation phase is bypassed entirely — KEDA never scales the role to zero, and the activationLagThreshold parameter has no effect. All scaling decisions are handled by the HPA.

To enable scale-to-zero, set minReplicaCount: 0 and tune activationLagThreshold separately.

Step 1: Create a ScalingAdapter for the RBG role

Enable the ScalingAdapter for the Prefill role when creating the RBG. The system automatically creates a RoleBasedGroupScalingAdapter resource and binds it to the role.

Create an rbg.yaml file. In the prefill role spec, set scalingAdapter.enable: true.

Sample YAML code

apiVersion: workloads.x-k8s.io/v1alpha1
kind: RoleBasedGroup
metadata:
  name: dynamo-pd
  namespace: default
spec:
  roles:
    - name: processor
      replicas: 1
      template:
        spec:
          containers:
            - command:
                - sh
                - -c
                - cd /workspace/examples/llm; dynamo serve graphs.pd_disagg:Frontend -f ./configs/qwen3.yaml
              env:
                - name: DYNAMO_NAME
                  value: dynamo
                - name: DYNAMO_NAMESPACE
                  value: default
                - name: ETCD_ENDPOINTS
                  value: http://etcd:2379
                - name: NATS_SERVER
                  value: nats://nats:4222
                - name: DYNAMO_RP_TIMEOUT
                  value: "60"
              image: # The address of the Dynamo Runtime container image you built
              name: processor
              ports:
                - containerPort: 8000
                  name: health
                  protocol: TCP
                - containerPort: 9345
                  name: request
                  protocol: TCP
                - containerPort: 443
                  name: api
                  protocol: TCP
                - containerPort: 9347
                  name: metrics
                  protocol: TCP
              readinessProbe:
                initialDelaySeconds: 30
                periodSeconds: 30
                tcpSocket:
                  port: 8000
              resources:
                limits:
                  cpu: "8"
                  memory: 12Gi
                requests:
                  cpu: "8"
                  memory: 12Gi
              volumeMounts:
                - mountPath: /models/Qwen3-32B/
                  name: model
                - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                  name: dynamo-configs
                  subPath: qwen3.yaml
                - mountPath: /workspace/examples/llm/graphs/pd_disagg.py
                  name: dynamo-configs
                  subPath: pd_disagg.py
          volumes:
            - name: model
              persistentVolumeClaim:
                claimName: llm-model
            - configMap:
                name: dynamo-configs
              name: dynamo-configs
    - name: prefill
      replicas: 2
      scalingAdapter:
        enable: true        # Enables the ScalingAdapter for this role
      template:
        spec:
          containers:
            - command:
                - sh
                - -c
                - cd /workspace/examples/llm; dynamo serve components.prefill_worker:PrefillWorker -f ./configs/qwen3.yaml
              env:
                - name: DYNAMO_NAME
                  value: dynamo
                - name: DYNAMO_NAMESPACE
                  value: default
                - name: ETCD_ENDPOINTS
                  value: http://etcd:2379
                - name: NATS_SERVER
                  value: nats://nats:4222
                - name: DYNAMO_RP_TIMEOUT
                  value: "60"
              image: # The address of the Dynamo Runtime container image you built
              name: prefill-worker
              resources:
                limits:
                  cpu: "12"
                  memory: 50Gi
                  nvidia.com/gpu: "2"
                requests:
                  cpu: "12"
                  memory: 50Gi
                  nvidia.com/gpu: "2"
              volumeMounts:
                - mountPath: /models/Qwen3-32B/
                  name: model
                - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                  name: dynamo-configs
                  subPath: qwen3.yaml
          volumes:
            - name: model
              persistentVolumeClaim:
                claimName: llm-model
            - configMap:
                name: dynamo-configs
              name: dynamo-configs
    - name: decoder
      replicas: 1
      template:
        spec:
          containers:
            - command:
                - sh
                - -c
                - cd /workspace/examples/llm; dynamo serve components.worker:VllmWorker -f ./configs/qwen3.yaml --service-name VllmWorker
              env:
                - name: DYNAMO_NAME
                  value: dynamo
                - name: DYNAMO_NAMESPACE
                  value: default
                - name: ETCD_ENDPOINTS
                  value: http://etcd:2379
                - name: NATS_SERVER
                  value: nats://nats:4222
                - name: DYNAMO_RP_TIMEOUT
                  value: "60"
              image: # The address of the Dynamo Runtime container image you built
              name: vllm-worker
              resources:
                limits:
                  cpu: "12"
                  memory: 50Gi
                  nvidia.com/gpu: "2"
                requests:
                  cpu: "12"
                  memory: 50Gi
                  nvidia.com/gpu: "2"
              volumeMounts:
                - mountPath: /models/Qwen3-32B/
                  name: model
                - mountPath: /workspace/examples/llm/configs/qwen3.yaml
                  name: dynamo-configs
                  subPath: qwen3.yaml
          volumes:
            - name: model
              persistentVolumeClaim:
                claimName: llm-model
            - configMap:
                name: dynamo-configs
              name: dynamo-configs
---
apiVersion: v1
kind: Service
metadata:
  name: dynamo-service
spec:
  type: ClusterIP
  ports:
    - port: 8000
      protocol: TCP
      targetPort: 8000
  selector:
    rolebasedgroup.workloads.x-k8s.io/name: dynamo-pd
    rolebasedgroup.workloads.x-k8s.io/role: processor

Apply the configuration.
```
kubectl apply -f rbg.yaml
```

Verify that the ScalingAdapter was created and bound successfully.

kubectl get rolebasedgroupscalingadapter

Expected output:

NAME                  PHASE   REPLICAS
dynamo-pd-prefill     Bound   2

For more detail, describe the resource:

kubectl describe rolebasedgroupscalingadapter dynamo-pd-prefill

Confirm that Status.Phase is Bound. This means the ScalingAdapter has successfully bound to the Prefill role.

Name:         dynamo-pd-prefill
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  workloads.x-k8s.io/v1alpha1
Kind:         RoleBasedGroupScalingAdapter
Metadata:
  Creation Timestamp:  2025-07-25T06:10:37Z
  Generation:          2
  Owner References:
    API Version:           workloads.x-k8s.io/v1alpha1
    Block Owner Deletion:  true
    Kind:                  RoleBasedGroup
    Name:                  dynamo-pd
    UID:                   5dd61668-79f3-4197-a5db-b778ce460270
  Resource Version:        1157485
  UID:                     edbb8373-2b9c-4ad1-8b6b-d5dfff71e769
Spec:
  Replicas:  2
  Scale Target Ref:
    Name:  dynamo-pd
    Role:  prefill
Status:
  Phase:     Bound
  Replicas:  2
  Selector:  rolebasedgroup.workloads.x-k8s.io/name=dynamo-pd,rolebasedgroup.workloads.x-k8s.io/role=prefill
Events:
  Type    Reason           Age   From                          Message
  ----    ------           ----  ----                          -------
  Normal  SuccessfulBound  25s   RoleBasedGroupScalingAdapter  Succeed to find scale target role [prefill] of rbg [dynamo-pd]

Step 2: Create a KEDA ScaledObject to monitor the message queue

Create a ScaledObject resource that targets the dynamo-pd-prefill ScalingAdapter and triggers scaling based on the NATS queue backlog.

Tune key parameters before deploying

The following parameters require tuning for your workload. Run a stress test (see Step 3) to determine the right values before deploying to production. Improper values can delay scale-out and degrade service performance, or cause over-provisioning and wasted resources.

Parameter	What it controls	How to tune
`lagThreshold`	Average number of pending messages per Prefill replica that triggers a scale-out	Set lower (for example, `3`) for latency-sensitive workloads where time-to-first-token (TTFT) matters; set higher for throughput-focused workloads that can tolerate queue buildup
`pollingInterval`	How often KEDA checks the queue depth, in seconds	Shorter intervals react faster but increase API server load; `30` seconds is a reasonable default for most LLM inference workloads
`minReplicaCount`	Minimum number of Prefill replicas KEDA maintains	Set to `1` to keep at least one warm replica at all times; set to `0` to enable scale-to-zero (requires tuning `activationLagThreshold`)
`maxReplicaCount`	Upper replica limit	Set based on available GPU capacity and budget

Create the ScaledObject

Create a scaledobject.yaml file with the following content.

The parameter values below are for demonstration only. Adjust them based on your workload before deploying to production.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: dynamo-prefill-scaledobject
spec:
  pollingInterval: 30        # How often KEDA polls the queue, in seconds
  minReplicaCount: 1         # Minimum Prefill replicas (default: 0; set to 0 to enable scale-to-zero)
  maxReplicaCount: 6         # Maximum Prefill replicas (default: 100)
  scaleTargetRef:
    apiVersion: workloads.x-k8s.io/v1alpha1
    kind: RoleBasedGroupScalingAdapter
    name: dynamo-pd-prefill  # Targets the Prefill role via its ScalingAdapter
  triggers:
  - type: nats-jetstream
    metadata:
      natsServerMonitoringEndpoint: "nats.default.svc.cluster.local:8222"  # NATS monitoring endpoint
      account: "$G"                    # Default NATS account when no account is configured
      stream: "dynamo_prefill_queue"   # NATS stream for Dynamo Prefill requests
      consumer: "worker-group"         # Consumer group name in Dynamo
      lagThreshold: "5"               # Scale out when average backlog per replica exceeds this value
      useHttps: "false"               # Set to "true" if your NATS endpoint uses HTTPS

Apply the configuration.
```
kubectl apply -f scaledobject.yaml
```

Verify the ScaledObject is ready.

The Cooldown Period: 300 in the output is KEDA's scale-to-zero cooldown — it controls how long KEDA waits after the last active trigger before scaling down to zero. Because minReplicaCount: 1 is set in this example, the cooldown has no effect. Scale-down from multiple replicas to one is controlled by the HPA's stabilizationWindowSeconds, not by this field.

kubectl describe so dynamo-prefill-scaledobject

Expected output

Name:         dynamo-prefill-scaledobject
Namespace:    default
Labels:       scaledobject.keda.sh/name=dynamo-prefill-scaledobject
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  ...
Spec:
  Cooldown Period:    300
  Max Replica Count:  6
  Min Replica Count:  1
  Polling Interval:   30
  Scale Target Ref:
    API Version:  workloads.x-k8s.io/v1alpha1
    Kind:         RoleBasedGroupScalingAdapter
    Name:         dynamo-pd-prefill
  Triggers:
    Metadata:
      Account:                          $G
      Consumer:                         worker-group
      Lag Threshold:                    5
      Nats Server Monitoring Endpoint:  nats.default.svc.cluster.local:8222
      Stream:                           dynamo_prefill_queue
      Use Https:                        false
    Type:                               nats-jetstream
Status:
  Conditions:
    Message:  ScaledObject is defined correctly and is ready for scaling
    Reason:   ScaledObjectReady
    Status:   True
    Type:     Ready
    Message:  Scaling is not performed because triggers are not active
    Reason:   ScalerNotActive
    Status:   False
    Type:     Active
    Status:   Unknown
    Type:     Fallback
  External Metric Names:
    s0-nats-jetstream-dynamo_prefill_queue
  Hpa Name:                keda-hpa-dynamo-prefill-scaledobject
  Original Replica Count:  1
  Scale Target GVKR:
    Group:            workloads.x-k8s.io
    Kind:             RoleBasedGroupScalingAdapter
    Resource:         rolebasedgroupscalingadapters
    Version:          v1alpha1
  Scale Target Kind:  workloads.x-k8s.io/v1alpha1.RoleBasedGroupScalingAdapter
Events:
  Type    Reason              Age   From           Message
  ----    ------              ----  ----           -------
  Normal  KEDAScalersStarted  3s    keda-operator  Started scalers watch
  Normal  ScaledObjectReady   3s    keda-operator  ScaledObject is ready for scaling

Confirm that Status.Conditions.Ready is True. KEDA automatically creates an HPA resource named keda-hpa-dynamo-prefill-scaledobject (recorded in Status.HpaName). View it with:

kubectl get hpa keda-hpa-dynamo-prefill-scaledobject

Step 3: (Optional) Stress test and verify scaling

Run a stress test to validate the scaling behavior and to determine the right lagThreshold and pollingInterval values for your workload before promoting the policy to production.

Deploy a benchmark instance.

Create a benchmark.yaml file.

Sample YAML code

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: llm-benchmark
  name: llm-benchmark
spec:
  selector:
    matchLabels:
      app: llm-benchmark
  template:
    metadata:
      labels:
        app: llm-benchmark
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - command:
        - sh
        - -c
        - sleep inf
        image: # The Dynamo container image used to deploy the inference service
        imagePullPolicy: IfNotPresent
        name: llm-benchmark
        resources:
          limits:
            cpu: "8"
            memory: 40Gi
          requests:
            cpu: "8"
            memory: 40Gi
        volumeMounts:
        - mountPath: /models/Qwen3-32B
          name: llm-model
      volumes:
      - name: llm-model
        persistentVolumeClaim:
          claimName: llm-model

Create the instance.
```
kubectl create -f benchmark.yaml
```

After the instance is running, exec into it and run the benchmark:

python3 $VLLM_ROOT_DIR/benchmarks/benchmark_serving.py \
        --backend openai-chat \
        --model /models/Qwen3-32B/ \
        --served-model-name qwen \
        --trust-remote-code \
        --dataset-name random \
        --random-input-len 1500 \
        --random-output-len 100 \
        --num-prompts 320 \
        --max-concurrency 32 \
        --host dynamo-service \
        --port 8000 \
        --endpoint /v1/chat/completions

For information about the benchmark tool and its parameters, see vLLM Benchmark.

While the stress test runs, open a new terminal and watch the HPA scaling events:

kubectl describe hpa keda-hpa-dynamo-prefill-scaledobject

SuccessfulRescale events in the output confirm that KEDA triggered scale-outs based on the NATS queue backlog.

Name:                               keda-hpa-dynamo-prefill-scaledobject
Namespace:                          default
Reference:                          RoleBasedGroupScalingAdapter/dynamo-pd-prefill
Min replicas:                       1
Max replicas:                       6
RoleBasedGroupScalingAdapter pods:  6 current / 6 desired
Events:
  Type     Reason             Age                   From                       Message
  ----     ------             ----                  ----                       -------
  Normal  SuccessfulRescale  2m1s  horizontal-pod-autoscaler  New size: 4; reason: external metric s0-nats-jetstream-dynamo_prefill_queue(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: dynamo-prefill-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target
  Normal  SuccessfulRescale  106s  horizontal-pod-autoscaler  New size: 6; reason: external metric s0-nats-jetstream-dynamo_prefill_queue(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: dynamo-prefill-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

Confirm that replica counts on the ScalingAdapter reflect the scale-out:

kubectl describe rolebasedgroupscalingadapter dynamo-pd-prefill

Spec.Replicas and Status.Replicas should increase from the initial value to the scaled-out value (for example, 6).

Name:         dynamo-pd-prefill
Namespace:    default
API Version:  workloads.x-k8s.io/v1alpha1
Kind:         RoleBasedGroupScalingAdapter
Metadata:
  Owner References:
    API Version:           workloads.x-k8s.io/v1alpha1
    Block Owner Deletion:  true
    Kind:                  RoleBasedGroup
    Name:                  dynamo-pd
Spec:
  Replicas:  6
  Scale Target Ref:
    Name:  dynamo-pd
    Role:  prefill
Status:
  Last Scale Time:  2025-08-04T02:08:10Z
  Phase:            Bound
  Replicas:         6
  Selector:         rolebasedgroup.workloads.x-k8s.io/name=dynamo-pd,rolebasedgroup.workloads.x-k8s.io/role=prefill
Events:
  Type    Reason           Age    From                          Message
  ----    ------           ----   ----                          -------
  Normal  SuccessfulBound  6m9s   RoleBasedGroupScalingAdapter  Succeed to find scale target role [prefill] of rbg [dynamo-pd]
  Normal  SuccessfulScale  4m40s  RoleBasedGroupScalingAdapter  Succeed to scale target role [prefill] of rbg [dynamo-pd] from 1 to 4 replicas
  Normal  SuccessfulScale  4m25s  RoleBasedGroupScalingAdapter  Succeed to scale target role [prefill] of rbg [dynamo-pd] from 4 to 6 replicas

Apply in production

Before promoting the scaling policy to production, complete the following:

Validate in a staging environment first. Run the stress test under conditions representative of your peak traffic.
Tune lagThreshold based on acceptable queue wait time. A lower threshold prevents queue buildup at the cost of more aggressive scale-outs; a higher threshold is more efficient but allows queues to grow under sudden load spikes.
Tune pollingInterval based on how quickly your workload ramps. A 30-second interval works for gradual load increases. For bursty workloads where load spikes within seconds, use a shorter interval.
Set maxReplicaCount to a value your cluster GPU capacity can support. Scaling beyond available node capacity leaves new pods in a Pending state.
Scale-to-zero considerations. If you set minReplicaCount: 0, the first request after a scale-to-zero event waits for a new Prefill pod to start (cold start latency). Set minReplicaCount: 1 to keep a warm replica available at all times.