Route requests to optimal pods by real-time queue and GPU cache metrics to cut time to first token (TTFT).
Features
Gateway with Inference Extension is an enhanced gateway component based on the Kubernetes community Gateway API and its Inference Extension specification. It provides Layer 4 and Layer 7 routing with additional capabilities for generative AI inference, simplifying inference-service management and optimizing load balancing across multiple inference workloads.
Supports Layer 4 and Layer 7 routing with these AI inference capabilities:
-
Inference-aware load balancing: Routes requests by real-time inference server metrics (queue depth and GPU KV cache utilization) instead of even distribution, keeping GPU utilization consistent and reducing TTFT latency.
-
Model-aware routing: Routes requests by model name per the OpenAI API specification. For a base model with multiple LoRA adapters, canary release directs traffic to specific adapters by adapter name.
-
Model criticality: Assigns a criticality level to each model.
Criticalrequests take priority over lower-criticality workloads.
Key concepts
Two custom resources extend the Kubernetes Gateway API:
| Resource | Purpose |
|---|---|
| InferencePool | Groups pods that share the same compute configuration, accelerator type, foundation model, and model server. Can span pods across ACK nodes for scalability and high availability. |
| InferenceModel | Defines the model name served by an InferencePool and its properties, including criticality level. |
The following diagram shows how InferencePool, InferenceModel, and Gateway API resources relate.
How it works
The gateway evaluates these real-time metrics on each pod to route incoming requests:
-
Request queue length (
vllm:num_requests_waiting): The number of queued requests. Pods with shorter queues process new requests sooner. -
GPU KV cache utilization (
vllm:gpu_cache_usage_perc): The percentage of GPU KV cache in use. Lower utilization means more capacity for new requests.
The gateway routes each request to the pod best able to handle it based on these metrics.
The following diagram illustrates the request flow.
Why inference-aware load balancing
Standard HTTP load balancing distributes requests evenly across pods. This works for stateless services but not for large language model (LLM) inference, where each request's compute cost is unpredictable.
LLM inference has two phases:
-
Prefill phase: Encodes the input.
-
Decoding phase: Proceeds in steps; each step decodes the prior input and outputs a token (roughly corresponding to a word).
Because output length is unpredictable, even distribution creates uneven GPU load—some pods bottleneck while others idle.
Inference-aware load balancing routes each request to the pod with the most available capacity based on real-time queue depth and GPU KV cache utilization, keeping GPU load balanced, reducing TTFT, and improving throughput.