Platform For AI:Improve inference efficiency with the LLM intelligent router

Architecture overview

The LLM intelligent router consists of three core components that provide intelligent traffic distribution and management for the backend LLM inference instance cluster: LLM Gateway, LLM Scheduler, and LLM Agent.

• LLM Gateway: Serves as the traffic entry point. It receives all user requests and forwards them to the designated backend inference instance based on decisions from the LLM Scheduler. The gateway supports HTTP (HTTP_SSE) and WebSocket protocols and can cache requests when backend inference instances are under a high load.

• LLM Scheduler: The brain of the intelligent router. It executes the scheduling algorithm by gathering real-time metrics from all LLM Agents. It then calculates the optimal target instance for each incoming request based on a predefined scheduling policy, such as prefix caching.

• LLM Agent: Deployed as a sidecar container alongside each inference instance. The agent collects performance metrics from the inference engine, maintains a heartbeat with the LLM Scheduler, and reports the health status and load data of the instance.

Implementation flow

The LLM intelligent router is a special type of EAS service that must be deployed in the same service group as the inference service to function correctly. After you deploy the LLM intelligent router and the inference service, the router intelligently schedules requests to the backend inference service. The process is as follows:

1. Instance registration: After the inference service starts, the LLM Agent waits for the inference engine to be ready. It then registers the instance with the LLM Scheduler and periodically reports its health status and performance metrics.

2. Traffic ingress: User requests first arrive at the LLM Gateway. The gateway supports HTTP (SSE) and WebSocket protocols.

3. Scheduling request: The LLM Gateway sends a scheduling request to the LLM Scheduler.

4. Intelligent scheduling: The LLM Scheduler selects the optimal backend instance based on its scheduling policy and real-time metrics from each LLM Agent.

5. Request forwarding: The LLM Scheduler returns its decision to the LLM Gateway. The LLM Gateway then forwards the original user request to the target instance for inference.

6. Request buffering: If all backend instances are under a high load, the LLM Gateway temporarily queues new requests. These requests wait for the LLM Scheduler to find a suitable time to forward them, which prevents request failures.

Failover mechanism

The system is designed with multiple layers of fault tolerance to ensure service stability:

• LLM Gateway: As a stateless traffic ingress layer, the `LLM Gateway` should be deployed with at least two instances. If one instance fails, traffic is automatically routed to other healthy instances. This ensures service continuity and high availability (HA).

• LLM Scheduler: As the request scheduling component, the `LLM Scheduler` runs as a single instance to achieve global scheduling. If the LLM Scheduler fails, the LLM Gateway automatically enters a degraded mode after a heartbeat failure. It then reverts to a polling policy to forward requests directly to backend instances. This ensures service availability but sacrifices scheduling performance. After the LLM Scheduler recovers, the LLM Gateway automatically switches back to smart routing mode.

• Inference instance or LLM Agent: If an inference instance or its associated LLM Agent fails, the heartbeat between the LLM Agent and the LLM Scheduler is interrupted. The LLM Scheduler immediately removes the failed instance from the list of available services and stops assigning new traffic to it. After the instance recovers and resumes sending heartbeats, it is automatically added back to the service list.

Support for multiple inference engines

Different LLM inference engines expose different metrics through their /metrics interfaces. The `LLM Agent` collects these metrics, formats them into a unified structure, and reports them. This design allows the `LLM Scheduler` to focus on scheduling algorithms based on unified metrics without needing to know the implementation details of specific inference engines. The currently supported LLM inference engines and their corresponding collected metrics are as follows:

Step 1: Deploy an LLM intelligent router service

The following two deployment methods are supported:

Step 2: Deploy a large language model (LLM) service

When you deploy an LLM service, you must associate it with the intelligent router service that you created. This topic uses the EAS scenario-based deployment of an LLM as an example. The procedure is as follows:

1. Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).

2. On the Inference Service page, click Deploy Service, and then select LLM Deployment. In the Features section, find and turn on the LLM Intelligent Router switch. From the drop-down list, select the LLM intelligent router service that you deployed in Step 1. For more information about the other parameters, see LLM Deployment.

   Note

   When you use vLLM for accelerated deployment and the Scheduling Policy of the LLM intelligent router service is set to Prefix Cache, you must enable the prefix caching feature of the engine.

3. After you configure the parameters, click Deploy.

Obtain access credentials

1. On the Elastic Algorithm Service (EAS) page, find the deployed LLM intelligent router service.

2. Click the service name to go to the Overview page. In the Basic Information section, click View Endpoint Information.

3. On the Invocation Method page, copy the Internet Endpoint and Token from the Service-specific Traffic Entry section.

Construct and send a request

The final request URL is a combination of the endpoint of the LLM intelligent router and the API path of the model service.

• URL structure: <LLM intelligent router endpoint>/<LLM service API path>

• Example: http://********.pai-eas.aliyuncs.com/api/predict/group_llm_gateway.llm_gateway/v1/chat/completions

Request example:

# Replace <YOUR_GATEWAY_URL> and <YOUR_TOKEN> with your actual information.
# Replace <model_name> with the actual model name.
curl -X POST "<YOUR_GATEWAY_URL>/v1/chat/completions" \
     -H "Authorization: Bearer <YOUR_TOKEN>" \
     -H "Content-Type: application/json" \
     -N \
     -d '{
           "model": "<model_name>",
           "messages": [{"role": "user", "content": "Hello"}],
           "stream": true
         }'

The following is an example of the returned result:

data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"content":"<think>","tool_calls":[]}}]}

...
data: [DONE]