Quick start: intelligent routing with Gateway with Inference Extension - Container Service for Kubernetes

Large language model (LLM) applications typically run on GPUs, and GPU-based nodes or virtual nodes are more expensive than CPU-based nodes. This document shows you how to use Gateway with Inference Extension to build a cost-effective, CPU-based mock environment. You can use this environment to test the intelligent load balancing capabilities for an inference service.

Prerequisites

Gateway with Inference Extension 1.4.0 installed with Enable Gateway API Inference Extension selected. For the installation entry point, see Step 2: Install the Gateway with Inference Extension component.

Important

The mock environment described in this document is intended only for evaluating the basic AI capabilities of Gateway with Inference Extension, such as canary release, circuit breaking, and traffic mirroring. It is not suitable for performance testing or for use in a production environment.

Procedure

Step 1: Deploy the mock model application

Create a file named mock-vllm.yaml with the following content.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mock-vllm
---
apiVersion: v1
kind: Service
metadata:
  name: mock-vllm
  labels:
    app: mock-vllm
    service: mock-vllm
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8000
  selector:
    app: mock-vllm
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mock-vllm
  template:
    metadata:
      labels:
        app: mock-vllm
    spec:
      serviceAccountName: mock-vllm
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/mock-vllm:v0.1.7-g3cffa27-aliyun
        imagePullPolicy: IfNotPresent
        name: mock-vllm
        ports:
        - containerPort: 8000

Deploy the sample application.
```
kubectl apply -f mock-vllm.yaml
```

Create a file named sleep.yaml with the following content.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image:  registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent

Deploy the test application to send test requests to the sample application.
```
kubectl apply -f sleep.yaml
```

Step 2: Deploy inference resources

Create a file named inference-rule.yaml with the following content.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: mock-pool
spec:
  extensionRef:
    group: ""
    kind: Service
    name: mock-ext-proc
  selector:
    app: mock-vllm
  targetPortNumber: 8000
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: mock-model
spec:
  criticality: Critical
  modelName: mock
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: mock-pool
  targetModels:
  - name: mock
    weight: 100

Deploy the InferencePool and InferenceModel resources.
```
kubectl apply -f inference-rule.yaml
```

Step 3: Deploy the gateway and routing rule

Installing Gateway with Inference Extension creates a GatewayClass resource by default. You can verify this by running the following command:
```
kubectl get gatewayclass
```
If the GatewayClass resource is not found, you must create it manually.
Create a GatewayClass
Save the following YAML content as gatewayclass.yaml, and then run the kubectl apply -f gatewayclass.yaml command.
```
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: ack-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
```

Create a file named gateway.yaml with the following content.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mock-gateway
spec:
  gatewayClassName: ack-gateway
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
    - name: llm-gw
      protocol: HTTP
      port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: mock-client-buffer-limit
spec:
  connection:
    bufferLimit: 20Mi
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: mock-gateway
---

Create a file named httproute.yaml with the following content.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mock-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: mock-gateway
    sectionName: llm-gw
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: mock-pool
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

Deploy the gateway and routing rule.

kubectl apply -f gateway.yaml
kubectl apply -f httproute.yaml

Step 4: Send a test request

Get the gateway IP address.

export GATEWAY_ADDRESS=$(kubectl get gateway/mock-gateway -o jsonpath='{.status.addresses[0].value}')
echo ${GATEWAY_ADDRESS}

Send a request from the test application.

kubectl exec deployment/sleep -it -- curl -X POST ${GATEWAY_ADDRESS}/v1/chat/completions \
  -H 'Content-Type: application/json' -H "Host: example.com" -v -d '{
    "model": "mock",
    "max_completion_tokens": 100,
    "temperature": 0,
    "messages": [
      {
        "role": "user",
        "content": "introduce yourself"
      }
    ]
}'

Expected output:

*   Trying 192.168.12.230:80...
* Connected to 192.168.12.230 (192.168.12.230) port 80
> POST /v1/chat/completions HTTP/1.1
> Host: example.com
> User-Agent: curl/8.8.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 184
> 
* upload completely sent off: 184 bytes
< HTTP/1.1 200 OK
< date: Tue, 27 May 2025 08:21:37 GMT
< server: uvicorn
< content-length: 354
< content-type: application/json
< 
* Connection #0 to host 192.168.12.230 left intact
{"id":"3bcc1fdd-e514-4a06-95aa-36c904015639","object":"chat.completion","created":1748334097.297188,"model":"mock","choices":[{"index":"0","message":{"role":"assistant","content":"As a mock AI Assitant, I can only echo your last message: introduce yourself"},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":76,"total_tokens":94}}

Step 5: Clean up the environment

If you no longer need this environment, clean it up.

Clean up cluster resources:

# Delete the gateway and routing rule
kubectl delete -f gateway.yaml
kubectl delete -f httproute.yaml
# Delete the test application
kubectl delete -f sleep.yaml
# Delete the backend application
kubectl delete -f mock-vllm.yaml
kubectl delete -f inference-rule.yaml

On the Add-ons page, search for Gateway with Inference Extension, and then click Uninstall on the component card.