Server Load Balancer:Implement token-based rate limiting with ALB Extensible Edition

Step 1: Create an ALB Extensible Edition instance

1. Go to the ALB console, select the Singapore region, and click Create ALB.

2. On the buy page, configure the following settings and click Buy Now.

   Parameter	Value
   Region	Singapore
   Network Type	Internet
   VPC and Zone	Select the target VPC. Select Zone A and Zone B, choose the corresponding vSwitches, and select Automatically assign EIP.
   IP Version	IPv4
   Edition	Extensible

3. On the Confirm Order page, review the instance configuration and click Activate Now.

Step 2: Create an AI Service-type server group

Create an AI Service-type server group to connect to Alibaba Cloud Model Studio.

1. Go to the Server Groups page and click Create Server Group.

2. Set Server Group Type to AI Service, enter a name such as sgp-ai-qwen, and click Create.

3. In the dialog box that appears, click Add Backend Server.

4. Configure the following settings and click OK.

   Parameter	Value
   Model provider	Alibaba Cloud Model Studio
   Endpoint	Auto-populated after you select the provider.
   API Key	Enter the API key for Alibaba Cloud Model Studio.

Step 3: Create a listener

1. In the ALB console, click the target instance ID to go to the instance details page. On the Listener tab, click Create Listener.

2. In the Configure Listener step, set Listener Protocol to HTTPS and Listener Port to 443, then click Next.

3. In the Configure SSL Certificate step, select the server certificate that matches your custom domain name and click Next.

4. In the Select Server Group step, select the AI Service type and the sgp-ai-qwen server group, then click Next.

   The selected server group is used for the default forwarding rule, which handles requests that do not match any other forwarding rule. Adjust this setting as needed.

5. In the Configuration Review step, confirm the settings and click Submit.

Step 4: Create a service extension

Create a service extension and add a token rate limiting component. The component identifies requests by an HTTP header and applies rate limiting.

1. Go to the Service Extensions page and click Create Service Extension.

2. In the Service Extension Configuration section, enter an Extension name such as ext-token-rate-limit.

3. Extension Type is set to Plug-in by default. From the Component name drop-down list, select Token Rate Limiting.

4. Configure the rate limiting policy and click Create.

The x-account-id value is extracted from the HTTP request header and used as the rate limiting identifier. Token consumption is tracked separately for each unique value, and rate limits are calculated independently. You can use other HTTP header fields or adjust the condition type and match type as needed.

When a service extension contains multiple rate limiting policies, requests are matched from top to bottom. The first matching policy is applied. Subsequent policies are not evaluated.

Step 5: Configure a forwarding rule

Create a forwarding rule for the listener. Add a domain name match condition and associate the service extension.

1. In the ALB console, click the target instance ID to go to the instance details page. Click the Listener tab, click the target listener ID, and go to the Forwarding Rules tab.

2. Click Add New Rule and configure the following settings, then click OK.

   Parameter	Value
   Add Condition	Select Domain Name. Set the match type to Exact Match. Enter the domain name used to access the ALB instance, such ai.example.com.
   Service Extension (Optional)	Select Use Existing Service Extension and choose ext-token-rate-limit from the drop-down list.
   Action	Select Forward to and choose the AI Service-type server group sgp-ai-qwen.

After the forwarding rule is created, requests with Host: ai.example.com in the HTTP header match this rule. The service extension checks the x-account-id header value against the rate limit. If the token consumption is within the threshold, the request is forwarded to the sgp-ai-qwen server group.

For production environments, you can use other types of forwarding conditions as needed.

Step 6: Configure DNS resolution

Create a CNAME record to point your custom domain name to the DNS name of the ALB instance. This allows clients to access the ALB instance through your domain name.

1. In the ALB console, copy the Domain Name of the target instance.

2. Go to the Alibaba Cloud DNS console. Find the target domain name, and click Settings in the Actions column. On the Settings tab, click Add Record.

3. Configure the following settings and click OK.

   Parameter	Value
   Record Type	CNAME
   Hostname	Enter the domain name prefix, such as ai. If your root domain name is example.com, the full domain name for accessing the ALB instance is ai.example.com. This must match the domain name configured in Step 5.
   Query Source and TTL	Keep the default values.
   Record Value	Enter the DNS name of the ALB instance.

4. In the confirmation dialog box, review the DNS record and click OK.

Step 7: Verify the configuration

Use curl commands to send requests and verify the token rate limiting feature. Requests must meet the following conditions:

• Domain name: Access the ALB instance through ai.example.com (the Host header value must match).

• Rate limiting header: Include x-account-id: . The rate limiting policy identifies requests by this header. Each unique header value has an independent token quota.

• OpenAI-compatible protocol: The request path must be /v1/completions, /v1/chat/completions, or /v1/embeddings, and the request body must comply with the protocol format.

Replace ai.example.com in the following commands with the actual domain name configured in Step 6. Make sure the DNS record has taken effect before testing.

curl -v \
-H "x-account-id: id" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen-turbo",
"messages": [
{
"role": "user",
"content": "Who are you"
}
]
}' \
https://ai.example.com/v1/chat/completions

{
"choices": [
{
"message": {
"role": "assistant",
"content": "Hello! I am Qwen, a large language model..."
},
"finish_reason": "stop",
"index": 0
}
],
"object": "chat.completion",
"usage": {
"prompt_tokens": 14,
"completion_tokens": 53,
"total_tokens": 67
},
"model": "qwen-turbo"
}

for i in {1..3}; do
curl -v \
-H "x-account-id: 12345" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen-turbo",
"messages": [
{
"role": "user",
"content": "Who are you"
}
]
}' \
https://ai.example.com/v1/chat/completions
sleep 2
done

How are requests handled if they do not match the rate limiting policy?

If a request does not include the x-account-id: header, it matches the forwarding rule but not the rate limiting policy. The request is forwarded directly without rate limiting.

Each unique x-account-id value has its own independent token quota.

I configured token rate limiting, but requests are not being limited. What should I check?

1. Forwarding rule: Verify that the forwarding condition matches the request, the rule has a high enough priority, and a service extension is associated.

2. Service extension: Verify that the token rate limiting component is configured correctly, the rate limiting condition matches the request, and the rate limit values are appropriate.

3. Request format: Verify that the request matches both the forwarding condition and the rate limiting condition.

4. Token consumption: Check the usage.total_tokens field in the response. The rate limiting policy counts cumulative token consumption within the time window. A single request may not reach the threshold.