Platform For AI:Deploy the LLM intelligent router to improve inference efficiency

Key concepts

The LLM intelligent router service consists of two core components:

• LLM Gateway: traffic entry and request processing. Receives user requests and forwards them to the target inference instance based on decisions from LLM Scheduler.

  • Supports HTTP (HTTP_SSE) and WebSocket protocols.

  • Anthropic API protocol conversion is enabled by default. Requests formatted to the Anthropic specification are automatically converted to OpenAI-compatible format, so you can use tools like Claude Code with model services that follow the OpenAI API standard without modifying your code. For details, see Use Claude Code with the router.

• LLM Scheduler: scheduling engine. Selects the optimal target instance for each request based on the configured scheduling policy (for example, prefix-cache routing).

How it works

The request flow proceeds as follows:

1. Receive request. The user request arrives at LLM Gateway. If all backend inference instances are at high load, the gateway queues the request.

2. Intelligent scheduling. LLM Gateway sends a scheduling request to LLM Scheduler. LLM Scheduler selects the optimal instance based on the configured policy and real-time metrics from each instance.

3. Forward request. After receiving the scheduling result, LLM Gateway forwards the user request directly to the selected instance.

Failover mechanism

The service includes multiple layers of fault tolerance to maintain availability:

• LLM Gateway: Deploy at least two instances. If one instance fails, traffic automatically shifts to healthy instances.

• LLM Scheduler: If the scheduler fails, LLM Gateway automatically falls back to round-robin routing (scheduling performance is reduced, but availability is maintained). When the scheduler recovers, intelligent scheduling resumes automatically.

• Inference instances: If an instance fails, LLM Scheduler immediately removes it from the available pool. No new traffic is routed to that instance until it recovers and is automatically re-added.

Step 1: Deploy the LLM intelligent router service

1. Log in to the PAI console and select the target region at the top of the page.

2. In the left navigation pane, click Elastic Algorithm Service (EAS), select the target workspace, and go to EAS.

3. Click Deploy Service, then select Scenario-based Model Deployment>Deploy LLM gateway.

4. Configure the parameters:

   Parameter		Description
   Basic Information	Service Name	A custom name for the service, for example, llm_gateway.
   Resource Information	Deployment Resources	Resource configuration for LLM Gateway. To ensure high availability, Number of Replicas defaults to 2; keep this default. The default resource allocation is 4 vCPUs and 8 GB of memory.
   	Scheduling configuration	Resource configuration for LLM Scheduler. The default is 2 vCPUs and 4 GB of memory.
   	Scheduling Policy	The load balancing policy used to select backend inference instances. The default is Prefix cache. For a detailed comparison and selection guidance, see Scheduling policy reference.
   Advanced Features	Redis config	Optional. Persists user information and usage statistics. Without Redis, the service uses local memory and data is lost when the service stops. For details, see Configure Redis persistent storage.

5. Click Deploy. When the service status changes to Running, the deployment is complete.

   After deployment, the system automatically creates a service group named group_<LLM intelligent router service name>. To view it, go to the Elastic Algorithm Service (EAS) page and click the Canary Release tab.

Step 2: Deploy the LLM service

Configure the intelligent router when creating a new LLM service. You can't add it to an existing service through Update.

The following example uses Qwen3-8B:

1. Click Deploy Service, then select Scenario-based Model Deployment> LLM Deployment.

2. Configure the following parameters:

   Parameter		Value
   Basic Information	Model Settings	Select Public Model, search for, and select Qwen3-8B.
   	Inference Engine	Select vLLM (recommended; OpenAI API-compatible). Note If the LLM intelligent router is configured with the Prefix cache scheduling policy and you select vLLM as the inference engine, make sure prefix caching is enabled in vLLM.
   	Deployment Template	Select Single node. The system automatically populates the recommended instance type, image, and other parameters based on the template.
   Features	LLM Intelligent Router	Enable the toggle and select the LLM intelligent router service deployed in Step 1 from the drop-down list.

3. Click Deploy. Deployment takes approximately 5 minutes. When the service status changes to Running, the deployment is complete.

Step 3: Test the service

Send all requests to the LLM intelligent router service's access URL, not to the individual backend inference service URLs.

1. Get access credentials.

   1. Click the LLM intelligent router service to open the Overview page. In the Basic Information section, click View Endpoint Information.

   2. On the Call information page, copy the Service-specific Traffic Entry Internet Endpoint, and copy the Token.

   Important

   The service token shown here is the administrator credential. If an administrator has assigned individual API keys to users through the WebUI workbench, each user should replace <YOUR_TOKEN> in the examples below with their own API key.

2. Build the request URL and send a call.

   • URL format: <LLM intelligent router URL>/<LLM service API path>

   • Example: http://********.pai-eas.aliyuncs.com/api/predict/group_llm_gateway.llm_gateway/v1/chat/completions

   Request example:

   # Replace <YOUR_GATEWAY_URL> and <YOUR_TOKEN> with your actual values
   curl -X POST "<YOUR_GATEWAY_URL>/v1/chat/completions" \
        -H "Authorization: Bearer <YOUR_TOKEN>" \
        -H "Content-Type: application/json" \
        -N \
        -d '{
              "messages": [{"role": "user", "content": "Hello"}],
              "stream": true
            }'

   Example response:

   data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
   data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"content":"<think>","tool_calls":[]}}]}
   
   ...
   data: [DONE]

After the service is deployed, access the WebUI workbench for visual management, including live monitoring and configuration management. For details, see Use the LLM intelligent routing workbench.

Step 1: Provision a Tair (Redis) instance

1. Go to the Tair purchase page and purchase an instance.

   Important

   Select the same virtual private cloud (VPC) and vSwitch as the LLM intelligent router service you plan to deploy. Configure other options as needed; start with the minimum specification.

2. Add the vSwitch CIDR block of the LLM intelligent router service to the instance allowlist. For details, see Configure an allowlist.

3. Click the target instance ID to open the details page and view the connection information. You'll need this information when deploying the LLM intelligent router service.

   • Connection address and port: In Instance information > Connection information, get the VPC connection address and port number.

   • Username and password: On the Account management page, view the default account. If no password was set when the instance was created, set or reset the password there. You can also create a new account instead of using the default.

Step 2: Configure Redis during deployment

In addition to the basic configuration, specify the VPC settings and Redis connection parameters to enable persistent storage.

• Configure the VPC: The VPC and vSwitch must match those of the Tair instance from Step 1, so the LLM intelligent router service can reach the Tair instance.

• Configure Redis:

  Console

  In the Advanced Features section, enable Redis config and fill in:

  Parameter	Description
  Redis Address	Connection address of the Tair instance, in the format <VPC address>:<port>, for example r-uxxx.redis.rds.aliyuncs.com:6379.
  Redis Username	Username for the Tair instance.
  Redis Password	Password for the Tair instance.

  JSON

  In the llm_gateway node of the JSON body, add the following parameters:

  Parameter	Description
  enable_user_management	Enables user management. Defaults to true.
  redis_addrs	Connection address of the Tair instance, in the format <VPC address>:<port>, for example r-uxxx.redis.rds.aliyuncs.com:6379.
  redis_username	Username for the Tair instance.
  redis_password	Password for the Tair instance.

  JSON example:

  {
      "llm_gateway": {
          "enable_user_management": true,
          "redis_addrs": "r-uxxx.redis.rds.aliyuncs.com:6379",
          "redis_username": "xxx",
          "redis_password": "xxx"
      }
  }

Configuration example

{
    "cloud": {
        "computing": {
            "instance_type": "ecs.c7.large"
        }
    },
    "llm_gateway": {
        "max_queue_size": 128,
        "retry_count": 2,
        "wait_schedule_timeout": 5000,
        "wait_schedule_try_period": 500
    },
    "llm_scheduler": {
        "cpu": 2,
        "memory": 4000,
        "policy": "prefix-cache"
    },
    "metadata": {
        "group": "group_llm_gateway",
        "instance": 2,
        "name": "llm_gateway",
        "type": "LLMGatewayService",
         "rpc": {
            "disable_auth": true
        }
    }
}

Parameters

Use Claude Code with the router

1. Set Claude Code to use the base URL and token provided by the EAS intelligent router service.

   # Replace <YOUR_GATEWAY_URL> and <YOUR_TOKEN> with your actual values
   export ANTHROPIC_BASE_URL=<YOUR_GATEWAY_URL>
   export ANTHROPIC_AUTH_TOKEN=<YOUR_TOKEN>

2. Run Claude Code.

   claude "Write a Python Hello World program"

Scheduling policy reference

Selecting the right scheduling policy is key to getting the most out of the LLM intelligent router. The following table compares each policy's logic, ideal use cases, advantages, and considerations.

Performance benchmark

Tests on Distill-Qwen-7B, QwQ-32B, and Qwen2.5-72B show improvements in inference speed and throughput when the LLM intelligent router is used. The test environment and results are described below.