Server Load Balancer:Token Rate Limiting Practice

1. Create an ALB Extensible Edition instance

1. Log on to the ALB console, select the China (Wulanchabu) region, and click Create ALB.

2. On the purchase page, configure the following settings and click Create Now.

   • Region: Select China (Wulanchabu).

   • Instance Network Type: Select Internet.

   • VPC and availability zone: Select the target VPC, check the boxes for Wulanchabu Zone A and Wulanchabu Zone B, select the corresponding vSwitches, and select Automatically Assign Public IP.

   • Protocol version: Select IPv4.

   • Edition (Instance Fee): Select Extensible.

3. On the Confirm Order page, verify the instance configuration details and click Activate Now.

2. Create an AI-type server group

Create an AI-type server group to connect to Alibaba Cloud Model Studio.

1. In the Server Group console, click Create Server Group. For Server Group Type, select AI Service, enter a name such as sgp-ai-qwen, and click Create.

2. In the The server group is created dialog box, click Add Backend Server.

3. In the Add AI Service dialog box, configure the following settings and click OK.

   • Model provider: Select Alibaba Cloud Model Studio.

   • Endpoint: The system automatically populates this field after you select the Model provider.

   • Identity Credential: Select the identity credential that you created for the Alibaba Cloud Model Studio API key.

3. Create a listener

1. In the ALB console, click the target instance ID to go to the Instance Details page. On the Listener tab, click Create Listener.

2. In the Configure Listener step, set Listener Protocol to HTTPS and Listener Port to 443. Then, click Next.

3. In the Configure SSL Certificate step, select the server certificate that matches your custom domain name and click Next.

4. In the Select Server Group step, select the AI Service type and the server group sgp-ai-qwen. Then, click Next.

   The server group selected here serves as the listener's default rule, which handles requests that do not match any other forwarding rules. You can adjust this based on your requirements.

5. In the Configuration Review step, confirm your configurations and click Submit.

4. Create a service extension

Create a service extension and add the token rate limiting component to identify requests by HTTP header and apply a rate limiting policy.

1. In the Service Extension console, click Create Service Extension. In the Service Extension Configuration area, enter a Extension name such as ext-token-rate-limit.

2. The Extension Type defaults to Plug-in. From the Component name drop-down list, select Token Rate Limiting. Configure the rate limiting policy and click Create.

   • Throttling Condition: Select By HTTP Header, enter the parameter x-account-id, and set the matching method to Wildcard Match.

     The value of x-account-id is extracted from the HTTP request header and used as a rate limiting identifier. Token consumption is tracked separately for requests with different values, and rate limiting is calculated independently for each value. x-account-id is only an example. You can use other HTTP header field names or adjust the rate limiting condition type and matching method based on your actual needs.

   • Throttling Range: Select Total Tokens to set a rate of 100 Tokens per 1 minute. You can adjust the values based on your actual needs.

   • Timeout and Processing policy: The default values 1000 and Skip are used in this topic. You can adjust them as needed.

If a token rate limiting component has multiple policies, requests are matched sequentially from top to bottom. Once a request matches a policy, the corresponding rate limit is applied, and no further policies are checked.

5. Configure a forwarding rule

Create a forwarding rule for the listener, add a domain name match condition, and associate it with the service extension.

1. In the ALB console, click the target instance ID to go to the Instance Details page. Switch to the Listener tab, click the target listener ID to go to the Listener Details page, and then switch to the Forwarding Rules tab.

2. Click Add New Rule, complete the following configurations, and click OK.

   • Add Condition: Select Domain Name. Exact Match is used for the domain name that is used to access the ALB instance, such as ai.example.com.

     For production use, you can use other types of forwarding conditions as needed.

   • Service Extension (Optional): The default option is Use Existing Service Extension. Select ext-token-rate-limit from the drop-down list.

   • Action: Select Forward To the AI service server group sgp-ai-qwen.

After the forwarding rule is created, requests with the Host field in the HTTP header set to ai.example.com will match the rule. The service extension then identifies the request based on the value of the x-account-id HTTP header, performs a rate limiting check, and forwards the request to the sgp-ai-qwen server group if the threshold is not exceeded.

6. Set up DNS resolution

Point your custom domain name to the ALB instance's DNS name by using a CNAME record. This allows clients to access ALB through your custom domain.

This guide uses Alibaba Cloud DNS as an example. If your domain is not registered with Alibaba Cloud, you must first add your domain to the Alibaba Cloud DNS console.

1. In the ALB console, copy the Domain Name of the target instance.

2. Log on to the Alibaba Cloud DNS console. In the Actions column for the target domain, click Settings. On the Settings page, click Add Record.

3. Add a CNAME record with the following information and click OK.

   • Record Type: Select CNAME.

   • Hostname: Enter a domain prefix, such as ai. If your root domain is example.com, the domain name used to access the ALB instance is ai.example.com. Note that this domain name must match the matching domain configured in Step 5.

   • Query Source and TTL: Keep the default values.

   • Record Value: Enter the DNS name of the ALB instance.

4. In the Change Resource Record Confirmation dialog box that appears, verify the DNS information and click OK.

7. Test and verify

Use the cURL command to send requests and verify the token rate limiting feature. The requests must meet the following conditions:

• Accessing the ALB through ai.example.com: The value of the Host field in the HTTP header is ai.example.com.

• Rate limiting policy matching headerx-account-id: <any value>: The rate limiting policy identifies requests by this header and calculates a separate rate limit for each header value to determine whether to forward the request to the backend AI service sgp-ai-qwen.

• Compliant with the OpenAI compatibility protocol: The request path is /v1/completions, /v1/chat/completions, or /v1/embeddings, and the overall format conforms to the specification.

The domain name ai.example.com in the following test command is an example. For the actual test, replace it with the actual domain name that you configured in Step 6. You must ensure that the domain name resolution has taken effect.

curl -v \
    -H "x-account-id: id" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen-turbo",
        "messages": [
            {
                "role": "user", 
                "content": "Who are you?"
            }
        ]
    }' \
    https://ai.example.com/v1/chat/completions

{
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "Hello! I am Qwen, a large-scale language model developed by the Tongyi Lab at Alibaba Group..."
            },
            "finish_reason": "stop",
            "index": 0
        }
    ],
    "object": "chat.completion",
    "usage": {
        "prompt_tokens": 14,
        "completion_tokens": 53,
        "total_tokens": 67
    },
    "model": "qwen-turbo"
}

for i in {1..3}; do
    curl -v \
        -H "x-account-id: 12345" \
        -H "Content-Type: application/json" \
        -d '{
            "model": "qwen-turbo",
            "messages": [
                {
                    "role": "user", 
                    "content": "Who are you?"
                }
            ]
        }' \
        https://ai.example.com/v1/chat/completions
    sleep 2
done

HTTP/2 429 
x-tokenratelimit-reset: 52
content-length: 17
content-type: text/plain
date: Wed, 21 Jan 2026 07:59:38 GMT

Too Many Requests

Billing

• ALB Extensible Edition: Currently in public beta and available for free.

• Internet access fees: The public NAT gateway charges for instance fees and capacity unit (CU) fees. The NAT gateway and the EIPs bound to the ALB Extensible Edition instance have separate billing rules, and the EIPs are billed separately.

• Domain name and DNS resolution fees: In addition to the domain fees from your provider, you will incur public authoritative DNS fees for configuring DNS resolution on Alibaba Cloud.

• Certificate fees: Purchasing a certificate from or uploading a certificate to Alibaba Cloud incurs server certificate fees.

• Model Studio model fees: Calling the Model Studio LLM API incurs fees.

Regions that support ALB Extensible Edition

Recommendations

• Rate limit threshold setting: Monitor your normal token consumption patterns and set a rate limit threshold slightly higher than the normal value. This avoids overly aggressive policies that might block legitimate requests.

• Time window selection:

  • For high-frequency call scenarios, use a shorter time window for more precise control. For low-frequency but high-consumption scenarios, use a longer time window to prevent single requests from being incorrectly limited.

  • Backend response time affects response-based rate limiting. The time window must be longer than the duration of a single model inference.

FAQ