Prevent Fault Propagation with Circuit Breaker Rules for Inference Services on ACK Gateway - Container Service for Kubernetes

ACK Gateway with Inference Extension lets you configure circuit breaker rules alongside intelligent load balancing for inference services. When a backend becomes unhealthy, the circuit breaker cuts off connections to that backend automatically, preventing fault propagation across the system.

Important

Before proceeding, make sure you understand InferencePool and InferenceModel concepts.

Prerequisites

Before you begin, make sure you have:

An ACK managed cluster with a GPU node pool. Alternatively, install the ACK Virtual Node component to use ACS GPU compute power.
ACK Gateway with Inference Extension installed with Enable Gateway API Inference Extension selected. For the installation entry point, see Step 2: Install the ACK Gateway with Inference Extension component.

Note

The container image used in this guide requires more than 16 GiB of GPU memory. The T4 card (16 GiB) may not be sufficient. For ACK clusters, use an A10 GPU. For ACS GPU compute power, use GN8IS (8th-gen GPU B). Because Large Language Model (LLM) images are large, push the image to Container Registry (ACR) in advance and pull it over the internal network to avoid slow downloads caused by elastic IP address (EIP) bandwidth limits.

How it works

The circuit breaker uses a BackendTrafficPolicy with two thresholds: maxPendingRequests and maxParallelRequests. This guide sets both to 1 for demonstration — low values make it easier to observe circuit breaker behavior in a test environment. Adjust both values for your production workload.

The following diagram shows the request flow when the circuit breaker is active.

With maxParallelRequests: 1 and maxPendingRequests: 1:

Request ① arrives. No concurrent requests are in flight, so it is forwarded to the application.
Request ② arrives before ① completes. Request ① is already being processed, so the circuit breaker rejects ② and returns an overflow error ③ to the client.
The application processes ① and returns response ④ to the client.

Deploy the inference service and configure the circuit breaker

This guide uses the following resources:

Resource	Description
`vllm-llama2-7b-pool` Deployment	The vLLM-based inference service
Gateway (ClusterIP)	Receives traffic from within the cluster; change to LoadBalancer for external access
HTTPRoute (`test-httproute`)	Routes requests to the InferencePool
BackendTrafficPolicy (`circuitbreaker-for-route`)	Applies the circuit breaker rule to the HTTPRoute
InferencePool + InferenceModel	Enable intelligent load balancing for the inference service
`sleep` Deployment	Test client

Step 1: Deploy the inference service

Apply inference_app.yaml to deploy the vllm-llama2-7b-pool inference service.

Expand to view YAML content

# =============================================================
# inference_app.yaml
# =============================================================
apiVersion: v1
kind: ConfigMap
metadata:
  name: chat-template
data:
  llama-2-chat.jinja: |
    {% if messages[0]['role'] == 'system' %}
      {% set system_message = '<<SYS>>\n' + messages[0]['content'] | trim + '\n<</SYS>>\n\n' %}
      {% set messages = messages[1:] %}
    {% else %}
        {% set system_message = '' %}
    {% endif %}

    {% for message in messages %}
        {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
            {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
        {% endif %}

        {% if loop.index0 == 0 %}
            {% set content = system_message + message['content'] %}
        {% else %}
            {% set content = message['content'] %}
        {% endif %}
        {% if message['role'] == 'user' %}
            {{ bos_token + '[INST] ' + content | trim + ' [/INST]' }}
        {% elif message['role'] == 'assistant' %}
            {{ ' ' + content | trim + ' ' + eos_token }}
        {% endif %}
    {% endfor %}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama2-7b-pool
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama2-7b-pool
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: '8000'
        prometheus.io/scrape: 'true'
      labels:
        app: vllm-llama2-7b-pool
    spec:
      containers:
        - name: lora
          image: "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/llama2-with-lora:v0.2"
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
          - "--model"
          - "/model/llama2"
          - "--tensor-parallel-size"
          - "1"
          - "--port"
          - "8000"
          - '--gpu_memory_utilization'
          - '0.8'
          - "--enable-lora"
          - "--max-loras"
          - "4"
          - "--max-cpu-loras"
          - "12"
          - "--lora-modules"
          - 'sql-lora=/adapters/yard1/llama-2-7b-sql-lora-test_0'
          - 'sql-lora-1=/adapters/yard1/llama-2-7b-sql-lora-test_1'
          - 'sql-lora-2=/adapters/yard1/llama-2-7b-sql-lora-test_2'
          - 'sql-lora-3=/adapters/yard1/llama-2-7b-sql-lora-test_3'
          - 'sql-lora-4=/adapters/yard1/llama-2-7b-sql-lora-test_4'
          - 'tweet-summary=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_0'
          - 'tweet-summary-1=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_1'
          - 'tweet-summary-2=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_2'
          - 'tweet-summary-3=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_3'
          - 'tweet-summary-4=/adapters/vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm_4'
          - '--chat-template'
          - '/etc/vllm/llama-2-chat.jinja'
          env:
            - name: PORT
              value: "8000"
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          livenessProbe:
            failureThreshold: 2400
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 6000
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              nvidia.com/gpu: 1
            requests:
              nvidia.com/gpu: 1
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: shm
            - mountPath: /etc/vllm
              name: chat-template
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
        - name: chat-template
          configMap:
            name: chat-template

Step 2: Deploy InferencePool and InferenceModel

Apply inference_rules.yaml to create the InferencePool and InferenceModel resources that enable intelligent load balancing.

# =============================================================
# inference_rules.yaml
# =============================================================
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: vllm-llama2-7b-pool
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-llama2-7b-pool
  extensionRef:
    name: inference-gateway-ext-proc
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: inferencemodel-sample
spec:
  modelName: /model/llama2
  criticality: Critical
  poolRef:
    group: inference.networking.x-k8s.io
    kind: InferencePool
    name: vllm-llama2-7b-pool
  targetModels:
  - name: /model/llama2
    weight: 100

Step 3: Deploy the Gateway, HTTPRoute, and circuit breaker policy

Apply gateway.yaml to create the Gateway, HTTPRoute, and BackendTrafficPolicy. The BackendTrafficPolicy attaches to the HTTPRoute and enforces the circuit breaker thresholds.

The Gateway Service type is ClusterIP, which limits access to within the cluster. Change it to LoadBalancer if you need external access.

# =============================================================
# gateway.yaml
# =============================================================
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
  name: example-gateway-class
  labels:
    example: http-routing
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  labels:
    example: http-routing
  name: example-gateway
  namespace: default
spec:
  gatewayClassName: example-gateway-class
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 80
    protocol: HTTP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-httproute
  labels:
    example: http-routing
spec:
  parentRefs:
    - name: example-gateway
  hostnames:
    - "example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
      - group: inference.networking.x-k8s.io
        kind: InferencePool
        name: vllm-llama2-7b-pool
        weight: 1
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: circuitbreaker-for-route
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: test-httproute
  circuitBreaker:
    maxPendingRequests: 1
    maxParallelRequests: 1 # Limit concurrent requests to 1

Step 4: Deploy the test client

Apply sleep.yaml to deploy the sleep application, which serves as the test client.

# =============================================================
# sleep.yaml
# =============================================================
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image:  registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Verify the circuit breaker

This guide sets maxPendingRequests: 1 and maxParallelRequests: 1 for demonstration purposes only. Adjust both values for your production workload.

Verify resource status

Before sending test traffic, confirm that all resources are ready.

Check the Gateway status:

kubectl get gateway example-gateway

Check the BackendTrafficPolicy:

kubectl get backendtrafficpolicy circuitbreaker-for-route

Check the InferencePool:

kubectl get inferencepool vllm-llama2-7b-pool

Confirm that all resources are running and have no error conditions before proceeding.

Trigger the circuit breaker

Get the gateway address:

export GATEWAY_ADDRESS=$(kubectl get gateway/example-gateway -o jsonpath='{.status.addresses[0].value}')

Open two terminal windows. In terminal 1, send a request to the inference service:

kubectl exec deployment/sleep -it -- curl -X POST ${GATEWAY_ADDRESS}/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H "host: example.com" \
  -d '{
    "model": "/model/llama2",
    "max_completion_tokens": 100,
    "temperature": 0,
    "messages": [
      {
        "role": "user",
        "content": "introduce yourself"
      }
    ]
  }'

Before terminal 1 returns a response, run the same command in terminal 2.

Expected output in terminal 1 — the inference service responds successfully:

{"id":"chatcmpl-eb67bf29-1f87-4e29-8c3e-a83f3c74cd87","object":"chat.completion","created":1745207283,"model":"/model/llama2","choices":[{"index":0,"message":{"role":"assistant","content":"\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n         [INST] I'm a [/INST]\n\n        ","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":15,"total_tokens":115,"completion_tokens":100,"prompt_tokens_details":null},"prompt_logprobs":null}%

Expected output in terminal 2 — the circuit breaker rejects the second request:

upstream connect error or disconnect/reset before headers. reset reason: overflow

The circuit breaker trips as soon as concurrent requests exceed the configured limit. Any request that arrives while the limit is already reached is rejected with the overflow error.

Clean up

Remove all resources created in this guide:

kubectl delete -f sleep.yaml
kubectl delete -f gateway.yaml
kubectl delete -f inference_rules.yaml
kubectl delete -f inference_app.yaml

What's next

To adjust circuit breaker thresholds for production workloads, modify maxPendingRequests and maxParallelRequests in the BackendTrafficPolicy based on your service's capacity.
To learn more about intelligent load balancing, see InferencePool and InferenceModel.