Use ACK Gateway with Inference Extension to implement traffic mirroring for inference services - Container Compute Service

Mirror live inference traffic to a shadow service to evaluate new LLM models under production load—without affecting your users.

Important

Before proceeding, make sure you understand InferencePool and InferenceModel.

How it works

Traffic mirroring splits every incoming inference request into two copies. The primary copy goes through the InferencePool to your production model with intelligent load balancing. The secondary copy is sent asynchronously to a shadow service running your new model.

Mirrored traffic is fire-and-forget: the shadow service's response is silently discarded, and only the primary service's response reaches the client. This means mirroring has zero impact on production latency or correctness.

Intelligent load balancing is not supported for mirrored traffic. The shadow service must use a regular Kubernetes Service, not an InferencePool.

Prerequisites

Before you begin, ensure that you have:

An ACK managed cluster with a GPU node pool, or an ACK managed cluster with the ACK Virtual Node component installed to use Alibaba Cloud Container Compute Service (ACS) GPU compute power
ACK Gateway with Inference Extension installed, with Enable Gateway API Inference Extension selected during cluster creation (see Step 2: Install the Gateway with Inference Extension component)

Note For the images used in this tutorial, use A10 GPU cards for ACK clusters and GN8IS GPU cards for ACS GPU compute power. Because LLM images are large, transfer them to Container Registry in advance and pull them using an internal network address. Pulling from the public network depends on your cluster's elastic IP address (EIP) bandwidth and may result in longer wait times.

What this example deploys

This tutorial deploys the following resources:

Resource	Name	Role
Inference service (primary)	`vllm-llama2-7b-pool`	Serves production traffic via InferencePool
Inference service (shadow)	`vllm-llama2-7b-pool-1`	Receives mirrored traffic via a regular Service
Gateway	`example-gateway`	ClusterIP gateway (can be changed to LoadBalancer)
GatewayClass	`example-gateway-class`	Binds to the Envoy controller
HTTPRoute	`mirror-route`	Routes primary traffic and mirrors a copy to the shadow service
InferencePool	`vllm-llama2-7b-pool`	Enables intelligent load balancing for the primary service
InferenceModel	`inferencemodel-sample`	Maps the model name to the InferencePool
Service	`vllm-llama2-7b-pool-1`	ClusterIP Service for the shadow deployment
Test client	`sleep`	Sends test requests from within the cluster

The following diagram shows the traffic flow:

The client sends a request to the gateway. The HTTPRoute matches the request using a prefix rule (/).
The matched request is split:
- The primary copy is forwarded to the InferencePool, which applies intelligent load balancing before sending it to vllm-llama2-7b-pool.
- The RequestMirror filter sends an asynchronous copy to the vllm-llama2-7b-pool-1 Service, which forwards it to the shadow deployment.
Both services process the request independently. The gateway uses only the InferencePool's response. The shadow service's response is silently discarded.

Deploy traffic mirroring for inference services

Step 1: Deploy the inference services

The following YAML deploys the primary inference service (vllm-llama2-7b-pool). The shadow service (vllm-llama2-7b-pool-1) uses the same configuration—replace every occurrence of vllm-llama2-7b-pool with vllm-llama2-7b-pool-1 before applying it a second time.

Expand to view YAML content

# =============================================================
# inference_app.yaml
# =============================================================
apiVersion: v1
kind: ConfigMap
metadata:
  name: chat-template
data:
  llama-2-chat.jinja: |
    {% if messages[0]['role'] == 'system' %}
      {% set system_message = '<<SYS>>\n' + messages[0]['content'] | trim + '\n<</SYS>>\n\n' %}
      {% set messages = messages[1:] %}
    {% else %}
        {% set system_message = '' %}
    {% endif %}

    {% for message in messages %}
        {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
            {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
        {% endif %}

        {% if loop.index0 == 0 %}
            {% set content = system_message + message['content'] %}
        {% else %}
            {% set content = message['content'] %}
        {% endif %}
        {% if message['role'] == 'user' %}
            {{ bos_token + '[INST] ' + content | trim + ' [/INST]' }}
        {% elif message['role'] == 'assistant' %}
            {{ ' ' + content | trim + ' ' + eos_token }}
        {% endif %}
    {% endfor %}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama2-7b-pool
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama2-7b-pool
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: '8000'
        prometheus.io/scrape: 'true'
      labels:
        app: vllm-llama2-7b-pool
    spec:
      containers:
        - name: lora
          image: "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/llama2-with-lora:v0.2"
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
          - "--model"
          - "/model/llama2"
          - "--tensor-parallel-size"
          - "1"
          - "--port"
          - "8000"
          - '--gpu_memory_utilization'
          - '0.8'
          - "--enable-lora"
          - "--max-loras"
          - "4"
          - "--max-cpu-loras"
          - "12"
          - "--lora-modules"
          - 'sql-lora=/adapters/yard1/llama-2-7b-sql-lora-test_0'
          - 'sql-lora-1=/adapters/yard1/llama-2-7b-sql-lora-test_1'
          - 'sql-lora-2=/adapters/yard1/llama-2-7b-sql-lora-test_2'
          - 'sql-lora-3=/adapters/yard1/llama-2-7b-sql-lora-test_3'
          - 'sql-lora-4=/adapters/yard1/llama-2-7b-sql-lora-test_4'
          - 'tweet-summary=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_0'
          - 'tweet-summary-1=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_1'
          - 'tweet-summary-2=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_2'
          - 'tweet-summary-3=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_3'
          - 'tweet-summary-4=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_4'
          - '--chat-template'
          - '/etc/vllm/llama-2-chat.jinja'
          env:
            - name: PORT
              value: "8000"
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          livenessProbe:
            failureThreshold: 2400
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 6000
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              nvidia.com/gpu: 1
            requests:
              nvidia.com/gpu: 1
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: shm
            - mountPath: /etc/vllm
              name: chat-template
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
        - name: chat-template
          configMap:
            name: chat-template

Step 2: Deploy the InferencePool, InferenceModel, and shadow Service

The following YAML creates the InferencePool and InferenceModel for the primary service, and a ClusterIP Service for the shadow deployment.

# =============================================================
# inference_rules.yaml
# =============================================================
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: vllm-llama2-7b-pool
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-llama2-7b-pool
  extensionRef:
    name: inference-gateway-ext-proc
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: inferencemodel-sample
spec:
  modelName: /model/llama2
  criticality: Critical
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: vllm-llama2-7b-pool
  targetModels:
  - name: /model/llama2
    weight: 100
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama2-7b-pool-1
spec:
  selector:
    app: vllm-llama2-7b-pool-1
  ports:
  - protocol: TCP
    port: 8000
    targetPort: 8000
  type: ClusterIP

Step 3: Deploy the Gateway and HTTPRoute

The following YAML creates a GatewayClass, a Gateway, an EnvoyProxy configuration, and the HTTPRoute with the RequestMirror filter.

The Gateway uses a ClusterIP Service and is only accessible from within the cluster. Change type: ClusterIP to type: LoadBalancer in the EnvoyProxy spec if you need external access.

# =============================================================
# gateway.yaml
# =============================================================
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: example-gateway-class
  labels:
    example: http-routing
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    example: http-routing
  name: example-gateway
  namespace: default
spec:
  gatewayClassName: example-gateway-class
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 80
    protocol: HTTP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mirror-route
  labels:
    example: http-routing
spec:
  parentRefs:
    - name: example-gateway
  hostnames:
    - "example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
      - group: inference.networking.x-k8s.io
        kind: InferencePool
        name: vllm-llama2-7b-pool
        weight: 1
      filters:
      - type: RequestMirror
        requestMirror:
          backendRef:
            kind: Service
            name: vllm-llama2-7b-pool-1
            port: 8000

Step 4: Deploy the test client

# =============================================================
# sleep.yaml
# =============================================================
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image:  registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Verify traffic mirroring

1. Get the gateway address

export GATEWAY_ADDRESS=$(kubectl get gateway/example-gateway -o jsonpath='{.status.addresses[0].value}')

2. Send a test request

kubectl exec deployment/sleep -it -- curl -X POST ${GATEWAY_ADDRESS}/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "host: example.com" \
  -d '{
      "model": "/model/llama2",
      "max_completion_tokens": 100,
      "temperature": 0,
      "messages": [
        {
          "role": "user",
          "content": "introduce yourself"
        }
      ]
  }'

The response comes from the primary service only. The expected output is similar to:

{"id":"chatcmpl-eb67bf29-1f87-4e29-8c3e-a83f3c74cd87","object":"chat.completion","created":1745207283,"model":"/model/llama2","choices":[{"index":0,"message":{"role":"assistant","content":"\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n        ","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":15,"total_tokens":115,"completion_tokens":100,"prompt_tokens_details":null},"prompt_logprobs":null}%

3. Check the application logs

If mirroring is working, both deployments log the same request. You should see POST /v1/chat/completions HTTP/1.1" 200 OK entries in both vllm-llama2-7b-pool and vllm-llama2-7b-pool-1—the presence of entries in the shadow deployment's log confirms that mirrored traffic is reaching it.

echo "original logs" && kubectl logs deployments/vllm-llama2-7b-pool | grep /v1/chat/completions | grep OK
echo "mirror logs" && kubectl logs deployments/vllm-llama2-7b-pool-1 | grep /v1/chat/completions | grep OK

The expected output is similar to:

original logs
INFO:     10.2.14.146:39478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.2.14.146:60660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
mirror logs
INFO:     10.2.14.146:39742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.2.14.146:59976 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Entries in both logs confirm that traffic mirroring is working correctly.