Use Gateway with Inference Extension to queue and prioritize inference requests - Container Service for Kubernetes

Gateway with Inference Extension supports load-aware inference request queueing and priority scheduling. When backend model servers are saturated, requests in the queue are dispatched by model priority so high-priority models are served first. This topic explains how to enable request queueing and priority scheduling.

Important

This feature requires Gateway with Inference Extension version 1.4.0 or later.

How it works

Generative AI inference servers have a hard limit on GPU throughput. When too many concurrent requests arrive, resources such as KV cache fill up, degrading response times and token throughput.

Gateway with Inference Extension monitors each server's internal metrics to detect saturation. At capacity, the gateway queues requests centrally instead of letting them pile up on the server, then dispatches them in priority order—high-priority models are served first.

Priority levels

Assign a criticality level to each InferenceModel to control queue priority under saturation:

Criticality level	Priority order
`Critical`	Highest
`Standard`	Medium
`Sheddable`	Lowest

With queueing enabled and backend servers saturated, requests dispatch in priority order: Critical first, then Standard, then Schedulable.

Prerequisites

Make sure you have:

An ACK managed cluster with GPU node pools. Or install ACK Virtual Node to use ACS (Alibaba Cloud Container Compute Service) GPU computing power.
Gateway with Inference Extension 1.4.0 installed with Enable Gateway API Inference Extension selected. See Step 2: Install the Gateway with Inference Extension component.

For the image in this topic, use A10 cards for ACK clusters and GN8IS cards for ACS GPU computing power. The LLM image is large—transfer it to Container Registry beforehand and pull it over the internal network. Public-network pulls depend on cluster EIP bandwidth and may take longer.

Enable request queueing and priority scheduling

Step 1: Deploy a sample inference service

Create vllm-service.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: qwen
  name: qwen
spec:
  progressDeadlineSeconds: 600
  replicas: 5
  selector:
    matchLabels:
      app: qwen
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8000"
        prometheus.io/scrape: "true"
      labels:
        app: qwen
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/compute-qos: default
        alibabacloud.com/gpu-model-series: GN8IS
    spec:
      containers:
        - command:
            - sh
            - -c
            - vllm serve /models/Qwen-2.5-7B-Instruct --port 8000 --enable_prefix_caching --trust-remote-code --served-model-name /model/qwen --gpu-memory-utilization 0.95 --enforce-eager --enable-lora --max-loras 2 --max-cpu-loras 4 --lora-modules travel-helper-v1=/models/Qwen-TravelHelper-Lora travel-helper-v2=/models/Qwen-TravelHelper-Lora-v2
          image: registry-cn-hangzhou.ack.aliyuncs.com/dev/qwen-2.5-7b-instruct-lora:v0.1
          imagePullPolicy: IfNotPresent
          name: custom-serving
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 30
            successThreshold: 1
            tcpSocket:
              port: 8000
            timeoutSeconds: 1
          resources:
            limits:
              nvidia.com/gpu: "1"
              cpu: "8"
              memory: 30G
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /dev/shm
              name: dshm
      restartPolicy: Always
      volumes:
        - emptyDir:
            medium: Memory
            sizeLimit: 30Gi
          name: dshm
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: qwen
  name: qwen
spec:
  ports:
    - name: http-serving
      port: 8000
      protocol: TCP
      targetPort: 8000
  selector:
    app: qwen

Deploy the sample inference service.

kubectl apply -f vllm-service.yaml

Step 2: Configure inference routing

Create InferencePool and InferenceModel resources. The inference-epp-env.networking.x-k8s.io/experimental-use-queueing: "true" and inference-epp-env.networking.x-k8s.io/experimental-use-scheduler-v2: "true" annotations on InferencePool enable request queueing.

Create inference-pool.yaml.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  annotations:
    inference-epp-env.networking.x-k8s.io/experimental-use-queueing: "true"
    inference-epp-env.networking.x-k8s.io/experimental-use-scheduler-v2: "true"
  name: qwen-pool
  namespace: default
spec:
  extensionRef:
    group: ""
    kind: Service
    name: qwen
  selector:
    app: qwen
  targetPortNumber: 8000
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: qwen-model
spec:
  criticality: Critical
  modelName: qwen
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: qwen-pool
  targetModels:
  - name: qwen
    weight: 100
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: travel-helper-model
spec:
  criticality: Standard
  modelName: travel-helper
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: qwen-pool
  targetModels:
  - name: travel-helper-v1
    weight: 100

This defines two InferenceModel resources:

Model	Criticality
`qwen-model` (base model `qwen`)	`Critical`
`travel-helper-model` (LoRA model `travel-helper`)	`Standard`

Deploy the inference routing configuration.
```
kubectl apply -f inference-pool.yaml
```

Step 3: Deploy the gateway and routing rules

Configure a Gateway and HTTPRoute to route qwen and travel-helper requests to the qwen-pool InferencePool.

Create inference-gateway.yaml.

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: inference-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: inference-gateway
  listeners:
    - name: llm-gw
      protocol: HTTP
      port: 8081
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-route
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
    sectionName: llm-gw
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: qwen-pool
    matches:
    - headers:
      - type: Exact
        name: X-Gateway-Model-Name
        value: qwen
    - headers:
      - type: RegularExpression
        name: X-Gateway-Model-Name
        value: travel-helper.*
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: backend-timeout
spec:
  timeout:
    http:
      requestTimeout: 24h
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway

Deploy the gateway.
```
kubectl apply -f inference-gateway.yaml
```

Step 4: Validate queueing and priority scheduling

Use the vLLM benchmark tool to load-test the qwen and travel-helper models simultaneously, pushing servers to full capacity.

Deploy the benchmark workload.

kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: vllm-benchmark
  name: vllm-benchmark
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: vllm-benchmark
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: vllm-benchmark
    spec:
      containers:
      - command:
        - sh
        - -c
        - sleep inf
        image: registry-cn-hangzhou.ack.aliyuncs.com/dev/llm-benchmark:random-and-qa
        imagePullPolicy: IfNotPresent
        name: vllm-benchmark
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
EOF

Get the internal IP of the gateway.

export GW_IP=$(kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=inference-gateway -o jsonpath='{.items[0].spec.clusterIP}')

Run the load tests simultaneously in two separate terminals.

Important

Results are from a test environment and for reference only. Actual results may vary.

Terminal 1: Load test the `qwen` (Critical) model

kubectl exec -it deploy/vllm-benchmark -- env GW_IP=${GW_IP} python3 /root/vllm/benchmarks/benchmark_serving.py \
--backend vllm \
--model /models/DeepSeek-R1-Distill-Qwen-7B \
--served-model-name qwen \
--trust-remote-code \
--dataset-name random \
--random-prefix-len 1000 \
--random-input-len 3000 \
--random-output-len 3000 \
--random-range-ratio 0.2 \
--num-prompts 300 \
--max-concurrency 60 \
--host $GW_IP \
--port 8081 \
--endpoint /v1/completions \
--save-result \
2>&1 | tee benchmark_serving.txt

Expected output:

============ Serving Benchmark Result ============
Successful requests:                     293
Benchmark duration (s):                  1005.55
Total input tokens:                      1163919
Total generated tokens:                  837560
Request throughput (req/s):              0.29
Output token throughput (tok/s):         832.94
Total Token throughput (tok/s):          1990.43
---------------Time to First Token----------------
Mean TTFT (ms):                          21329.91
Median TTFT (ms):                        15754.01
P99 TTFT (ms):                           140782.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          58.58
Median TPOT (ms):                        58.36
P99 TPOT (ms):                           91.09
---------------Inter-token Latency----------------
Mean ITL (ms):                           58.32
Median ITL (ms):                         50.56
P99 ITL (ms):                            64.12
==================================================

Terminal 2: Load test the `travel-helper` (Standard) model

kubectl exec -it deploy/vllm-benchmark -- env GW_IP=${GW_IP} python3 /root/vllm/benchmarks/benchmark_serving.py \
--backend vllm \
--model /models/DeepSeek-R1-Distill-Qwen-7B \
--served-model-name travel-helper \
--trust-remote-code \
--dataset-name random \
--random-prefix-len 1000 \
--random-input-len 3000 \
--random-output-len 3000 \
--random-range-ratio 0.2 \
--num-prompts 300 \
--max-concurrency 60 \
--host $GW_IP \
--port 8081 \
--endpoint /v1/completions \
--save-result \
2>&1 | tee benchmark_serving.txt

Expected output:

============ Serving Benchmark Result ============
Successful requests:                     165
Benchmark duration (s):                  889.41
Total input tokens:                      660560
Total generated tokens:                  492207
Request throughput (req/s):              0.19
Output token throughput (tok/s):         553.41
Total Token throughput (tok/s):          1296.10
---------------Time to First Token----------------
Mean TTFT (ms):                          44201.12
Median TTFT (ms):                        28757.03
P99 TTFT (ms):                           214710.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          67.38
Median TPOT (ms):                        60.51
P99 TPOT (ms):                           118.36
---------------Inter-token Latency----------------
Mean ITL (ms):                           66.98
Median ITL (ms):                         51.25
P99 ITL (ms):                            64.87
==================================================

The results confirm priority scheduling under full load:

Metric	`qwen` (Critical)	`travel-helper` (Standard)
Mean TTFT (ms)	21,329	44,201
Successful requests	293 / 300 (97.7%)	165 / 300 (55%)
Mean TPOT (ms)	58.58	67.38

Under saturation, the Critical qwen model shows about 50% lower mean TTFT than Standard travel-helper, and successful requests are roughly 97.7% versus 55% (about 96% fewer failures for qwen). Standard-priority requests queue longer, as expected.Standard -priority requests queue longer, The Critical model also completed 293 of 300 requests (97.7%) compared with 165 of 300 (55%) for the Standard model. Critical requests. This is expected: Critical requests are served first; Standard requests wait.