Implement global rate limiting by using Gateway with Inference Extension - Container Service for Kubernetes

Gateway with Inference Extension supports global throttling to protect your cluster from instability under high traffic or burst conditions. This topic explains how to enable global throttling and test common rate-limiting scenarios using a BackendTrafficPolicy resource.

How it works

Throttling limits how many requests a server accepts within a given time window — for example, 300 requests per minute or 10 requests per second.

When you enable global throttling, Gateway with Inference Extension automatically deploys a centralized throttling service. This service enforces rate-limit policies across the gateway in real time by interacting with built-in filters (such as the rate limit filter) to apply your configured thresholds — such as maximum requests per second or concurrent connections.

Prerequisites

Before you begin, ensure that you have:

Gateway with Inference Extension version 1.4.0 or later installed
Completed the steps in Preparations

Step 1: Enable global throttling

The global throttling service requires Redis as a shared backend for distributed rate-limit counters. The following steps use a self-built Redis instance as an example. Alternatively, use Tair (Redis OSS-compatible) and update the ack-gateway-config ConfigMap in the envoy-gateway-system namespace accordingly. For the full EnvoyGateway configuration reference, see Envoy Gateway.

Create a file named redis-service.yaml to deploy Redis.

kind: Namespace
apiVersion: v1
metadata:
  name: redis-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: redis-system
  labels:
    app: redis
spec:
  serviceName: "redis"
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - image: registry-cn-hangzhou.ack.aliyuncs.com/dev/redis:6.0.6-for-ack-gateway
          name: redis
          ports:
            - containerPort: 6379
          resources:
            limits:
              cpu: 1500m
              memory: 512Mi
            requests:
              cpu: 200m
              memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: redis-system
  labels:
    app: redis
spec:
  ports:
    - name: redis
      port: 6379
      protocol: TCP
      targetPort: 6379
  selector:
    app: redis

Create a file named enable-global-rate-limit.yaml. This ConfigMap points the gateway to your Redis instance.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-gateway-config
  namespace: envoy-gateway-system
data:
  ack-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    rateLimit:
      backend:
        type: Redis
        redis:
          url: redis.redis-system.svc.cluster.local:6379

Deploy the Redis service and apply the configuration.

kubectl apply -f redis-service.yaml
kubectl apply -f enable-global-rate-limit.yaml

Step 2: Deploy a sample HTTPRoute

Create an HTTPRoute resource to use as the target for your throttling policy.

Create a file named httproute.yaml.

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-ratelimit
spec:
  parentRefs:
  - name: eg
  hostnames:
  - ratelimit.example
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - group: ""
      kind: Service
      name: backend
      port: 3000

Deploy the HTTPRoute resource.
```
kubectl apply -f httproute.yaml
```

Get the public IP address of the gateway.

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

Step 3: Configure and test throttling scenarios

The following scenarios demonstrate different BackendTrafficPolicy configurations. Each scenario edits or replaces the policy applied to the http-ratelimit HTTPRoute.

Scenario 1: Throttle by user ID

This scenario limits a specific user — identified by the request header x-user-id: one — to 3 requests per hour. Requests with any other x-user-id value are not affected.

Create a file named backendtrafficpolicy.yaml.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: policy-httproute
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: http-ratelimit
  rateLimit:
    type: Global
    global:
      rules:
      - clientSelectors:
        - headers:
          - name: x-user-id
            value: one
        limit:
          requests: 3
          unit: Hour

Apply the policy.

kubectl apply -f backendtrafficpolicy.yaml

Test by sending four requests with the x-user-id: one header.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:47:49 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 2
x-ratelimit-reset: 731

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:47:50 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 1
x-ratelimit-reset: 730

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:47:52 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 728

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 727
date: Tue, 27 May 2025 07:47:52 GMT
transfer-encoding: chunked

The first three requests return HTTP 200. The fourth returns HTTP 429, confirming the policy is active.

Test by sending four requests with the x-user-id: two header.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:50:11 GMT
content-length: 504

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:50:12 GMT
content-length: 504

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:50:14 GMT
content-length: 504

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:50:15 GMT
content-length: 504

All four requests return HTTP 200, confirming the policy does not apply to x-user-id: two.

Scenario 2: Throttle all users except administrators

This scenario applies a per-user limit of 3 requests per hour to every distinct user ID, while leaving requests with x-user-id: admin unrestricted. The Distinct type creates a separate counter for each unique header value. Setting invert: true excludes the admin value from the policy.

Edit the throttling policy.

kubectl edit BackendTrafficPolicy policy-httproute

Update the rateLimit section as follows:

...
  rateLimit:
    type: Global
    global:
      rules:
      - clientSelectors:
        - headers:
          - type: Distinct
            name: x-user-id
          - name: x-user-id
            value: admin
            invert: true
        limit:
          requests: 3
          unit: Hour

The policy takes effect immediately after saving.

Test by sending four requests with the x-user-id: one header.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:47:49 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 2
x-ratelimit-reset: 731

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:47:50 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 1
x-ratelimit-reset: 730

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:47:52 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 728

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 727
date: Tue, 27 May 2025 07:47:52 GMT
transfer-encoding: chunked

The first three requests return HTTP 200. The fourth returns HTTP 429.

Test by sending four requests with the x-user-id: two header.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:53:38 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 2
x-ratelimit-reset: 382

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:53:39 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 1
x-ratelimit-reset: 381

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:53:41 GMT
content-length: 504
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 379

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 378
date: Tue, 27 May 2025 07:53:41 GMT
transfer-encoding: chunked

The first three requests return HTTP 200. The fourth returns HTTP 429.

Test by sending four requests with the x-user-id: admin header.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" --header "x-user-id: admin" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:57:44 GMT
content-length: 506

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:57:45 GMT
content-length: 506

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:57:46 GMT
content-length: 506

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 07:57:47 GMT
content-length: 506

All four requests return HTTP 200, confirming that admin requests bypass the throttling policy.

Scenario 3: Throttle all requests

This scenario applies a single shared limit of 3 requests per hour across all traffic, regardless of client identity.

Edit the throttling policy.

kubectl edit BackendTrafficPolicy policy-httproute

Update the rateLimit section as follows:

...
  rateLimit:
    type: Global
    global:
      rules:
      - limit:
          requests: 3
          unit: Hour

The policy takes effect immediately after saving.

Test by sending four requests.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 08:02:53 GMT
content-length: 473
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 2
x-ratelimit-reset: 3427

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 08:02:55 GMT
content-length: 473
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 1
x-ratelimit-reset: 3425

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 08:02:56 GMT
content-length: 473
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 3424

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 3423
date: Tue, 27 May 2025 08:02:57 GMT
transfer-encoding: chunked

The first three requests return HTTP 200. The fourth returns HTTP 429, confirming the policy is active.

Scenario 4: Throttle by client IP address

This scenario limits each source IP address to 3 requests per hour based on its CIDR range.

Note

The IP range 0.0.0.0/0 is used for demonstration purposes. Adjust the range to match your requirements.

Edit the throttling policy.

kubectl edit BackendTrafficPolicy policy-httproute

Update the rateLimit section as follows:

...
  rateLimit:
    type: Global
    global:
      rules:
      - clientSelectors:
        - sourceCIDR:
            value: 0.0.0.0/0
            type: Exact
        limit:
          requests: 3
          unit: Hour

The policy takes effect immediately after saving.

Test by sending four requests.

for i in {1..4}; do kubectl exec deployment/sleep -it -- curl -I --header "Host: ratelimit.example" http://$GATEWAY_HOST/get ; sleep 1; done

Expected output:

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 08:02:53 GMT
content-length: 473
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 2
x-ratelimit-reset: 3427

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 08:02:55 GMT
content-length: 473
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 1
x-ratelimit-reset: 3425

HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 27 May 2025 08:02:56 GMT
content-length: 473
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 3424

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 3, 3;w=3600
x-ratelimit-remaining: 0
x-ratelimit-reset: 3423
date: Tue, 27 May 2025 08:02:57 GMT
transfer-encoding: chunked

The first three requests return HTTP 200. The fourth returns HTTP 429, confirming the per-IP throttling policy for 0.0.0.0/0 is active.

(Optional) Step 4: Clean up resources

Delete the throttling policy.

kubectl delete BackendTrafficPolicy policy-httproute

Delete the remaining resources created in this topic.

kubectl delete -f httproute.yaml
kubectl delete -f redis-service.yaml
kubectl delete -f enable-global-rate-limit.yaml