Container Compute Service:Global rate limiting for generative AI requests based on token count

How it works

Token-based throttling combines two capabilities of Gateway with Inference Extension:

• Global throttling: Enforces rate limits using the token bucket algorithm. By default, each HTTP request consumes one token from the bucket. The cost per request is configurable.

• Generative AI observability plugin: Inspects responses from generative AI applications, extracts the token count, and exposes it as a metric that the throttling layer reads.

The example in this topic limits each user based on completion tokens — the tokens generated in each response. You can also track prompt tokens or total tokens by updating the metadata key in the cost field.

Token bucket algorithm

The system generates tokens at a fixed rate and adds them to a bucket with a finite capacity. Each incoming request must consume one or more tokens to proceed:

• If the bucket has enough tokens, the request succeeds and the tokens are deducted.

• If the bucket is empty, the request is queued or rejected.

This algorithm keeps the average request rate within the token generation rate while allowing short traffic bursts.

Prerequisites

Before you begin, ensure that you have:

• Gateway with Inference Extension 1.4.0 or later installed with the Enable Gateway API Inference Extension option selected. For installation instructions, see Install Gateway with Inference Extension

• The mock-vllm application deployed

• The generative AI observability plugin deployed

• Global throttling enabled

Step 1: Deploy a throttling policy

This example creates a throttling policy for the mock-vllm application that limits each user to 300 completion tokens per hour.

1. Create a file named token-ratelimit-test.yaml with the following content.

   apiVersion: gateway.envoyproxy.io/v1alpha1
   kind: BackendTrafficPolicy
   metadata:
     name: token-ratelimit-test
   spec:
     targetRefs:
     - group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: mock-route
     rateLimit:
       type: Global
       global:
         rules:
         - clientSelectors:
           - headers:
             - name: x-user-id
               type: Distinct
           limit:
             requests: 300
             unit: Hour
           cost:
             response:
               from: Metadata
               metadata:
               # This metric is set by the generative AI observability plugin
                 namespace: FILTER_STATE
                 key: wasm.gen_ai.completion.tokens

   The key fields in this configuration are:

   Field	Description
   clientSelectors[].headers[].name: x-user-id	Identifies each client by the x-user-id request header. Each distinct value gets its own token bucket.
   limit.requests: 300 / limit.unit: Hour	Sets the token budget to 300 per hour per user.
   cost.response.from: Metadata	Reads the token cost from response metadata rather than counting requests.
   metadata.namespace: FILTER_STATE	The Envoy metadata namespace where the observability plugin stores the token count.
   metadata.key: wasm.gen_ai.completion.tokens	The metadata key that holds the completion token count for each response.

2. Apply the throttling policy.

   kubectl apply -f token-ratelimit-test.yaml

Step 2: Test the deployment

1. Get the IP address of the gateway.

   export GATEWAY_ADDRESS=$(kubectl get gateway/mock-gateway -o jsonpath='{.status.addresses[0].value}')
   echo ${GATEWAY_ADDRESS}

2. Send 5 consecutive requests from a client pod.

   kubectl exec deployment/sleep -it -- curl -X POST ${GATEWAY_ADDRESS}/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H "host: example.com" \
     -d '{
       "model": "mock",
       "max_completion_tokens": 100,
       "temperature": 0,
       "messages": [
         {
           "role": "user",
           "content": "introduce yourself"
         }
       ]
     }' -v -H "x-user-id: one" | grep "^< HTTP"

   Expected output:

   < HTTP/1.1 200 OK
   < HTTP/1.1 200 OK
   < HTTP/1.1 200 OK
   < HTTP/1.1 200 OK
   < HTTP/1.1 429 Too Many Requests

   The mock-vllm application returns a fixed-length response of 76 completion tokens per request:

   • Requests 1–3 consume 228 tokens (3 × 76), staying within the 300-token limit.

   • Request 4 brings the total to 304 tokens (4 × 76), exceeding the limit.

   • Request 5 is rejected with 429 Too Many Requests.

What's next

For more advanced throttling strategies and configuration options, see:

• Monitor generative AI requests

• BackendTrafficPolicy RateLimit field configuration