Server Load Balancer:AI service fallback

1. Create an ALB Extensible Edition instance

1. Log on to the ALB console, select the China (Ulanqab) region, and then click Create ALB.

2. On the purchase page, complete the following configurations and click Create Now.

   • Region: Select China (Ulanqab).

   • Instance network type: Select Internet.

   • VPC and Availability zone: Select the target VPC. Then, select the checkboxes for Ulanqab Zone A and Ulanqab Zone B, select the corresponding vSwitch for each, and select Assign EIP.

   • IP version: Select IPv4.

   • Edition (instance fee): Select Extensible Edition.

3. On the Confirm Order page, review the instance configuration details and click Activate Now.

2. Create AI server groups

Create three AI service-type server groups for DeepSeek, OpenAI, and Model Studio (Bailian). To demonstrate fallback, the DeepSeek and OpenAI server groups reference identity credentials that contain incorrect API keys so that requests to them fail and trigger the fallback mechanism.

3. Create a listener

1. In the ALB console, click the ID of your target instance to go to the Instance Details page. On the Listener tab, click Create Listener.

2. In the Configure Listener step, set Listener Protocol to HTTPS and Listener Port to 443. Then, click Next.

3. In the Configure SSL Certificate step, select the server certificate that matches your custom domain and click Next.

4. In the Select Server Group step, select the AI Service type and the server group sgp-fake-deepseek. Then, click Next.

   The server group selected here is used for the listener's default rule, which processes requests that do not match any other forwarding rules. You can adjust this based on your needs.

5. In the Configuration Review step, confirm the configuration and click Submit.

4. Configure a forwarding rule

ALB evaluates forwarding rules in ascending order of priority. Once a request matches a rule, ALB executes that rule's action and skips the remaining rules.

1. On the Listener tab of the instance details page, click the ID of your target listener. On the Listener Details page, switch to the Forwarding Rules tab.

2. Click Add New Rule, complete the following configurations, and then click OK.

   1. Add Condition: Select AI Model and enter deepseek-chat for Model Name.

   2. Action: Forward To the server group sgp-fake-deepseek.

   3. Add Action: Configure a Fallback to the server group sgp-fake-openai and enter gpt-3.5 for the Model Name on the right.

   4. Under the Fallback section, click Add Server Group. Select the server group sgp-bailian and enter qwen-turbo for the Model Name on the right.

The AI Model condition matches all requests that use an OpenAI-compatible protocol, have a request path of /v1/completions, /v1/chat/completions, or /v1/embeddings. If you specify a Model Name, the model field in the request body must also match the specified name.

When you configure a model name for a fallback action, ALB automatically rewrites the model field in the request body to the specified name during fallback. This ensures the request body matches the model expected by the target service.

5. Set up DNS resolution

Create a CNAME record that points your custom domain to the DNS name of the ALB instance, so that clients can access ALB through your domain.

This guide uses Alibaba Cloud DNS as an example. If your domain is not registered with Alibaba Cloud, add the domain to the Alibaba Cloud DNS console first.

1. In the ALB console, copy the Domain Name of your target instance.

2. Log on to the Alibaba Cloud DNS console. In the Actions column for your target domain, click Settings. On the Settings page, click Add Record.

3. Add a CNAME record with the following information and click OK.

   • Record Type: Select CNAME.

   • Hostname: Enter a domain prefix, such as ai. If your root domain is example.com, the full domain for accessing ALB will be ai.example.com.

   • Query Source and TTL: Keep the default settings.

   • Record Value: Enter the DNS name of the ALB instance.

4. In the Change Resource Record Confirmation dialog box, review the DNS information and click OK.

6. Test and verify

• Use the curl command to verify that the request falls back to the correct large model service. Replace ai.example.com with your configured domain. Make sure the DNS record has propagated.

• The client request must use the OpenAI-compatible protocol. The request path must be /v1/completions, /v1/chat/completions, or /v1/embeddings, and the request body must include the model field. ALB uses the model value for routing.

curl -v \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek-chat",
        "messages": [
            {
                "role": "user",
                "content": "Who are you"
            }
        ]
    }' \
    https://ai.example.com/v1/chat/completions

A successful request returns the following response. Because sgp-fake-deepseek and sgp-fake-openai use incorrect API keys, ALB receives failed responses and falls back sequentially, ultimately forwarding the request to sgp-bailian. The response shows model as qwen-turbo and content containing an introduction from Qwen, confirming that the request fell back to Model Studio (Bailian) and the model name was rewritten to qwen-turbo.

{
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "I am Qwen, a large language model from Alibaba's Tongyi Lab. I can answer questions, create text, use logical reasoning, and write code. How can I help you?"
            },
            "finish_reason": "stop",
            "index": 0
        }
    ],
    "object": "chat.completion",
    "model": "qwen-turbo"
}

Billing

• ALB Extensible Edition: This feature is currently in public beta and is free to use.

• Internet access fees: NAT Gateway charges instance fees and capacity unit (CU) fees. The Elastic IP Addresses (EIPs) bound to the NAT Gateway and ALB Extensible Edition instance have separate billing rules, charged by the EIP service.

• Domain and DNS fees: In addition to domain registration fees, configuring DNS resolution with Alibaba Cloud incurs public authoritative DNS fees.

• Certificate fees: Purchasing a certificate from Alibaba Cloud or uploading a third-party certificate incurs server certificate fees.

• Model Studio fees: Calling the Model Studio large model API incurs usage fees.

Limitations

• Fallback is supported for the following server group types: Server, IP Address, Function Compute, FQDN, and AI Service. These server group types can be used as targets for both Forward To and Fallback actions.

• If you configure a fallback to a non-AI Service server group, ensure that the server group provides an OpenAI protocol-compatible model service.

• By default, you can configure up to five fallback services for a single forwarding rule. To increase this quota, submit a ticket.

• The trigger conditions for fallback (HTTP 4xx or 5xx status codes) are fixed and cannot be customized.

ALB Extensible Edition regions

Recommendations

• Plan the fallback order : ALB executes fallback actions in the order they are configured. Plan the priority of each fallback service based on model availability, cost, and response latency.

• Ensure model consistency: Output formats and capabilities vary between model providers. Select models with similar capabilities for each fallback service to minimize the impact on your business logic during fallback.

FAQ