Platform For AI:Improve inference efficiency with the LLM intelligent router

Core components

The LLM intelligent router service consists of the following two core components:

• LLM Gateway: The traffic entry point and request processing center. It receives user requests and forwards them to the target inference instance based on decisions from the LLM Scheduler.

  • Supports the HTTP (HTTP_SSE) and WebSocket protocols.

  • By default, the Anthropic API protocol conversion feature is enabled. This feature automatically converts Anthropic-compliant requests into an OpenAI-compatible format. This allows you to use ecosystem tools, such as Claude Code, to call model services that follow the OpenAI API standard without code modifications. For more information, see Use Claude Code to make calls.

• LLM Scheduler: The intelligent scheduling engine. It selects the optimal target instance for each request based on a scheduling policy, such as prefix-cache routing.

Workflow

The following is the key workflow:

1. Receive request: A user request arrives at the LLM Gateway. If backend inference instances experience high load, the gateway queues the request.

2. Intelligent scheduling: The LLM Gateway sends a scheduling request to the LLM Scheduler. The LLM Scheduler then selects the optimal instance based on the scheduling policy and real-time metrics from each instance.

3. Forward request: After receiving the scheduling decision, the LLM Gateway forwards the user request directly to the selected instance.

Failover mechanism

A multi-layered fault tolerance mechanism ensures service stability:

• LLM Gateway: We recommend deploying at least two instances. If an instance fails, traffic automatically fails over to a healthy one.

• LLM Scheduler: If the scheduler fails, the LLM Gateway automatically falls back to round-robin routing. This ensures availability at the cost of intelligent scheduling. When the scheduler recovers, it automatically resumes intelligent scheduling.

• Inference instance: If an inference instance fails, the LLM Scheduler immediately removes it from the available pool and stops routing new traffic to it. After the instance recovers, the scheduler automatically adds it back to the pool.

Step 1: Deploy the LLM intelligent router

1. Log on to the PAI console and select the target region at the top of the page.

2. In the left-side navigation pane, click Elastic Algorithm Service (EAS). On the EAS page, select the target workspace.

3. Click Deploy Service, and then select Scenario-based Model Deployment > Deploy LLM gateway.

4. Configure the parameters:

   Parameter		Description
   Basic Information	Service Name	Enter a custom name for the service, for example, llm_gateway.
   Resource Information	Deployment Resources	Specifies the resource configuration for the LLM Gateway. For high availability, the Number of Replicas is set to 2 by default; we recommend keeping this setting. The default configuration is 4 vCPUs and 8 GB of memory.
   	Scheduling configuration	Specifies the resource configuration for the LLM Scheduler. The default is 2 vCPUs and 4 GB of memory.
   	Scheduling Policy	The load balancing policy for backend inference instances. The default is Prefix cache. For a detailed comparison to help you choose, see Scheduling policy reference.
   Advanced Features	Redis config	Optional. Persists user information and usage statistics. If not configured, the service uses local memory, and the data is lost when the service stops. For details, see Configure Redis persistent storage.

5. Click Deploy. When the service status changes to Running, the deployment is successful.

   After deployment, the system automatically creates a service group named group_<LLM intelligent router service name>. To view the group, go to the Elastic Algorithm Service (EAS) page and click the Group Service tab.

Step 2: Deploy an LLM service

You must configure the intelligent router feature when you deploy a new LLM service. You cannot add this feature to an existing service with the Update action.

The following steps use the deployment of Qwen3-8B as an example:

1. Click Deploy Service, and then select Scenario-based Model Deployment > LLM Deployment.

2. Configure the following key parameters:

   Parameter		Value
   Basic Information	Model Settings	Select Public Model, and then search for and select Qwen3-8B.
   	Inference Engine	Select vLLM (recommended; compatible with the OpenAI API). Note If the LLM intelligent router service uses the Prefix cache scheduling policy and you select vLLM as the inference engine, ensure the prefix caching feature is enabled for the engine.
   	Deployment Template	Select Single node. The system then automatically populates the recommended instance specifications, image, and other parameters from the template.
   Features	LLM Intelligent Router	Turn on the toggle and select the LLM intelligent router service deployed in Step 1 from the drop-down list.

3. Click Deploy. The deployment takes about 5 minutes. When the service status changes to Running, the deployment is successful.

Step 3: Test the service

All requests must be sent to the access endpoint of the LLM intelligent router service, not to the backend inference services.

1. Get access credentials.

   1. Go to the Overview page of the LLM intelligent router service. In the Basic Information section, click View Endpoint Information.

   2. On the Endpoint Information page, under Service-specific Traffic Entry, copy the Internet Endpoint and token.

   Important

   The service token here is the administrator API key. If an administrator has assigned you a separate API key through the WebUI workbench, use your own API key to replace <YOUR_TOKEN> in the example below.

2. Build the request URL and call the service.

   • URL format: <LLM intelligent router access endpoint>/<LLM service API path>

   • Example: http://********.pai-eas.aliyuncs.com/api/predict/group_llm_gateway.llm_gateway/v1/chat/completions

   Request example:

   # Replace <YOUR_GATEWAY_URL> and <YOUR_TOKEN> with your actual values
   curl -X POST "<YOUR_GATEWAY_URL>/v1/chat/completions" \
        -H "Authorization: Bearer <YOUR_TOKEN>" \
        -H "Content-Type: application/json" \
        -N \
        -d '{
              "messages": [{"role": "user", "content": "Hello"}],
              "stream": true
            }'

   Example response:

   data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
   data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"content":"<think>","tool_calls":[]}}]}
   
   ...
   data: [DONE]

After you deploy the service, you can use the WebUI workbench for tasks like live monitoring and configuration management. For details, see Use the LLM intelligent router workbench.

Step 1: Provision a Tair (Redis) instance

1. Purchase an instance from the Tair purchase page.

   Important

   When you create the instance, select the same VPC and vSwitch as your LLM intelligent router service. For other options, you can start with the minimum specifications.

2. Add the CIDR block of the vSwitch for the LLM intelligent router service to the instance's allowlist. For more information, see Configure an allowlist.

3. Click the target instance ID to open the details page. You will need the connection information when you deploy the LLM intelligent router service.

   • Connection address and port: In the Instance information section, find the VPC connection address and port number under Connection information.

   • Username and password: On the Account management page, find the default account. If you did not set a password when you created the instance, you can set or reset it here. You can also create a new account instead of using the default one.

Step 2: Configure Redis

In addition to the basic configuration, you must configure the VPC and add the Redis connection parameters to enable persistent storage.

• Configure VPC: The VPC and vSwitch must be the same as those of the Tair instance provisioned in Step 1. This ensures that the LLM intelligent router service can connect to the Tair instance.

• Configure Redis:

  Console

  In the Advanced Features section, enable Redis config and enter the following parameters:

  Parameter	Description
  Redis Address	The connection address of the Tair instance, in the format <VPC address>:<port>. For example, r-uxxx.redis.rds.aliyuncs.com:6379.
  Redis Username	The username for the Tair instance.
  Redis Password	The password for the Tair instance.

  JSON

  In the llm_gateway node of the JSON body, add the following parameters:

  Parameter	Description
  enable_user_management	Enables user management. Defaults to true.
  redis_addrs	The connection address of the Tair instance, in the format <VPC address>:<port>. For example, r-uxxx.redis.rds.aliyuncs.com:6379.
  redis_username	The username for the Tair instance.
  redis_password	The password for the Tair instance.

  JSON example:

  {
      "llm_gateway": {
          "enable_user_management": true,
          "redis_addrs": "r-uxxx.redis.rds.aliyuncs.com:6379",
          "redis_username": "xxx",
          "redis_password": "xxx"
      }
  }

Configuration example

{
    "cloud": {
        "computing": {
            "instance_type": "ecs.c7.large"
        }
    },
    "llm_gateway": {
        "max_queue_size": 128,
        "retry_count": 2,
        "wait_schedule_timeout": 5000,
        "wait_schedule_try_period": 500
    },
    "llm_scheduler": {
        "cpu": 2,
        "memory": 4000,
        "policy": "prefix-cache"
    },
    "metadata": {
        "group": "group_llm_gateway",
        "instance": 2,
        "name": "llm_gateway",
        "type": "LLMGatewayService",
         "rpc": {
            "disable_auth": true
        }
    }
}

Parameters

Use Claude Code

1. Configure Claude Code to use the BASE URL and TOKEN from the EAS intelligent router service.

   # Replace <YOUR_GATEWAY_URL> and <YOUR_TOKEN> with your actual values
   export ANTHROPIC_BASE_URL=<YOUR_GATEWAY_URL>
   export ANTHROPIC_AUTH_TOKEN=<YOUR_TOKEN>

2. Run Claude Code.

   claude "Write a Python Hello World program"

Scheduling policies

This table compares the logic, use cases, advantages, and considerations for each scheduling policy.

Performance benchmark

Benchmarks on the Distill-Qwen-7B, QwQ-32B, and Qwen2.5-72B models show that the LLM intelligent router significantly improves inference speed and throughput.