構建mock環境快速體驗Gateway with Inference Extension的智能推理-Container Service Kubernetes 版 ACK-阿里雲

LLM應用通常需要使用GPU來啟動，而且GPU類型的節點或虛擬節點比CPU節點的開銷更高。為此，Gateway with Inference Extension組件團隊提供了一種使用CPU算力快速體驗大語言模型（LLM）推理情境的智能負載平衡能力。本文介紹如何基於Gateway with Inference Extension構建一個mock環境來快速體驗推理服務的智能負載平衡能力。

適用範圍

已安裝Gateway with Inference ExtensionGateway with Inference Extension並勾選啟用Gateway API推理擴充。操作入口，請參見安裝組件。

重要

本文構建的mock環境僅適用於快速體驗Gateway with Inference Extension組件的一些基礎AI能力，例如灰階發布、請求熔斷、流量鏡像等，不適用於需要進行壓測的測試情境，也不建議在生產環境中使用。

操作步驟

步驟一：部署mock模型樣本應用

建立mock-vllm.yaml。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mock-vllm
---
apiVersion: v1
kind: Service
metadata:
  name: mock-vllm
  labels:
    app: mock-vllm
    service: mock-vllm
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8000
  selector:
    app: mock-vllm
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mock-vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mock-vllm
  template:
    metadata:
      labels:
        app: mock-vllm
    spec:
      serviceAccountName: mock-vllm
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/mock-vllm:v0.1.7-g3cffa27-aliyun
        imagePullPolicy: IfNotPresent
        name: mock-vllm
        ports:
        - containerPort: 8000

部署樣本應用。
```
kubectl apply -f mock-vllm.yaml
```

使用以下內容建立sleep.yaml檔案。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image:  registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent

部署測試應用，用於後續對樣本應用發起測試請求。
```
kubectl apply -f sleep.yaml
```

步驟二：部署inference資源

建立inference-rule.yaml。

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: mock-pool
spec:
  extensionRef:
    group: ""
    kind: Service
    name: mock-ext-proc
  selector:
    app: mock-vllm
  targetPortNumber: 8000
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: mock-model
spec:
  criticality: Critical
  modelName: mock
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: mock-pool
  targetModels:
  - name: mock
    weight: 100

部署inferencePool和inferenceModel。
```
kubectl apply -f inference-rule.yaml
```

步驟三：部署網關和路由規則

安裝Gateway with Inference Extension會預設建立GatewayClass，可參考以下方式進行查看。
```
kubectl get gatewayclass
```
如未發現GatewayClass資源，需手動建立。
建立GatewayClass
將以下YAML內容儲存為gatewayclass.yaml，然後執行kubectl apply -f gatewayclass.yaml命令。
```
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: ack-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
```

建立gateway.yaml。

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mock-gateway
spec:
  gatewayClassName: ack-gateway
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
    - name: llm-gw
      protocol: HTTP
      port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: mock-client-buffer-limit
spec:
  connection:
    bufferLimit: 20Mi
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: mock-gateway
---

建立httproute.yaml。

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mock-route
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: mock-gateway
    sectionName: llm-gw
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: mock-pool
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

部署網關和路由規則。

kubectl apply -f gateway.yaml
kubectl apply -f httproute.yaml

步驟四：發起測試

擷取網關IP。

export GATEWAY_ADDRESS=$(kubectl get gateway/mock-gateway -o jsonpath='{.status.addresses[0].value}')
echo ${GATEWAY_ADDRESS}

從測試應用中發起訪問。

kubectl exec deployment/sleep -it -- curl -X POST ${GATEWAY_ADDRESS}/v1/chat/completions \
  -H 'Content-Type: application/json' -H "Host: example.com" -v -d '{
    "model": "mock",
    "max_completion_tokens": 100,
    "temperature": 0,
    "messages": [
      {
        "role": "user",
        "content": "introduce yourself"
      }
    ]
}'

預期輸出：

*   Trying 192.168.12.230:80...
* Connected to 192.168.12.230 (192.168.12.230) port 80
> POST /v1/chat/completions HTTP/1.1
> Host: example.com
> User-Agent: curl/8.8.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 184
> 
* upload completely sent off: 184 bytes
< HTTP/1.1 200 OK
< date: Tue, 27 May 2025 08:21:37 GMT
< server: uvicorn
< content-length: 354
< content-type: application/json
< 
* Connection #0 to host 192.168.12.230 left intact
{"id":"3bcc1fdd-e514-4a06-95aa-36c904015639","object":"chat.completion","created":1748334097.297188,"model":"mock","choices":[{"index":"0","message":{"role":"assistant","content":"As a mock AI Assitant, I can only echo your last message: introduce yourself"},"finish_reason":"stop"}],"usage":{"prompt_tokens":18,"completion_tokens":76,"total_tokens":94}}

步驟五：清理環境

若不再需要使用此環境，可進行清理：

叢集資源清理：

# 刪除網關和路由
kubectl delete -f gateway.yaml
kubectl delete -f httproute.yaml
# 刪除測試應用
kubectl delete -f sleep.yaml
# 刪除後端應用
kubectl delete -f mock-vllm.yaml
kubectl delete -f inference-rule.yaml

在組件管理頁面，搜尋Gateway with Inference Extension，然後在組件卡片中單擊卸載。