Alibaba Cloud AI Gateway Intelligent Routing Is Now Live

By Zhang Jianchuan (Congyan)

The more models there are, the harder it becomes to choose. Alibaba Cloud AI Gateway Intelligent Routing is now live: it first identifies which models can handle the task well, then makes a clear choice based on your priorities—overall balance, cost, speed, or task fit. In multi-turn conversations and Agent tool-call chains, it also knows when not to make unnecessary changes.

Most developers building large language model applications have experienced this: they start by integrating one model and eventually manage an entire set. A single business application may involve simple question answering, content summarization, complex reasoning, code generation, and Agent tool calls, while each model differs in capability, price, latency, and availability. This creates a new engineering question: Which model should handle this particular request?

There are several common approaches, each with its own challenges. Sending everything to a flagship model delivers consistent results, but costs and rate-limiting pressure keep rising. Hard-coding selection logic in the application means redeploying whenever a model changes, prices shift, or operational conditions fluctuate. Reassessing every request can also cause an Agent's tool-call chain to "switch brains mid-task," losing context affinity and adding latency.

With Intelligent Routing, we do not want to add another layer of complex configuration. Instead, we bring model selection into AI Gateway: sufficient capability is a prerequisite, you define the optimization objective, and each request ultimately goes to just one model.

Meet the Task Requirements First, Then Optimize Price, Speed, and Fit

One Request, One Decision

Once a valid Auto request enters the gateway, Intelligent Routing evaluates it in a fixed sequence:

Determine whether it is a standalone request, a regular user message, or a continuation of tool/model state.
First exclude candidates that clearly fail requirements for protocol, modality, tool capabilities, context capacity, authorization, availability, and other conditions.
Understand the relationship between the current task and the previous task, as well as the task's difficulty and indispensable capabilities.
Use ModelCard metadata and the platform's model capability references to form a candidate set that is "capable enough to complete the task."
Apply the Profile selected by the requester within this set.
Produce a single routing decision and send the request to the model that will actually execute it.

We would like to clarify two boundaries up front.

First, the task advisor only describes "what the task requires"; it does not directly specify the final model. Gateway-side policies still make the unified decision based on facts such as protocol, capability, cost, and operational status. If the advisor is temporarily unavailable, an initial or standalone request can use local scoring to make a controlled decision, provided deterministic hard constraints are satisfied. Requests with trusted continuity state instead favor conservatively reusing the original execution model.

Second, candidate evaluation results are not a Fallback order. Intelligent Routing only selects the execution model for the current primary request; it does not generate an automatic candidate execution chain. Retries and fallback after a model call fails remain the responsibility of the separate AI Fallback capability. The two capabilities can work together, but their responsibilities remain distinct.

Lower Cost, Faster Responses, or Greater Accuracy—You Decide

When using Intelligent Routing, the requester must explicitly select a mode:

Mode	Suitable priorities	What it means in practice
`auto/balanced`	Most general-purpose scenarios	Balances task fit, cost, and operational status
`auto/cheap`	High-volume, cost-sensitive tasks	Favors a lower cost measure only among candidates with sufficient capability
`auto/fast`	Interactive, low-latency scenarios	Favors lower latency to the first non-empty response body and better health status, based on valid operational samples
`auto/quality`	Complex reasoning and critical tasks	Prioritizes task fit among candidates with sufficient capability

Note that all four Profiles represent optimization preferences, not an SLA guaranteeing a theoretical optimum. cheap will not choose an underqualified model merely because it is inexpensive, and quality does not guarantee the correctness of an individual answer. When the underlying facts are similar, it is normal for different Profiles to select the same model.

Custom weights are not available in the initial release, primarily because we do not want every team to maintain its own routing formula that is difficult to validate. The platform centrally evolves the Profile templates; you only need to express your business preferences.

In the Agent Era, Knowing When Not to Reselect Is Just as Important

Three Types of Calls, Three Routing Cadences

Simple Model API requests, multi-turn chats, and Agent calls may all look like "requests to a model," but they require completely different routing cadences.

Standalone Requests: Reselection Is Possible Every Time

Requests such as summarization, rewriting, content review, and format conversion are independent of one another and lack reliable session evidence. The gateway treats each request as a new decision unit, allowing each task to select a model according to its own requirements.

Multi-Turn Conversations: Switch Only When Appropriate

When a user follows up with "Keep going" or "Expand on the second point," switching models abruptly may invalidate the context cache and cause the response style to drift. Intelligent Routing first determines whether the current message continues the context, begins an independent new task, or cannot be confidently classified:

If the context continues or the classification is uncertain, conservatively reuse the model that actually executed the previous turn.
Make a new decision at the user-message boundary only when a new task is identified with high confidence.
The gateway overrides soft affinity and reselects only if new protocol, modality, tool, or context requirements make the current model ineligible.

Agent Tool-Call Chains: No Mid-Task Brain Swaps

A single user task may trigger multiple rounds of model requests, tool calls, and tool-result returns. Intelligent Routing makes one decision at the start of the task and keeps subsequent tool continuations on the same actual execution model. Existing correlation information for tool and model states in standard protocols is used to restore the correct execution relationships.

An Agent therefore does not repeat the full routing process simply because a tool result is returned, nor does it switch models within the same task solely because of short-term cost or health fluctuations. This avoids additional latency while preserving context and cache affinity as much as possible.

Of course, continuity does not pretend to have unlimited memory. When a client compresses the context, loses protocol-level causal information, or sends only ordinary requests that cannot distinguish sessions, the guarantees degrade accordingly. The gateway does not force all requests under the same consumer identity into a single session.

Platform Models and Your Own Models Share Clear Selection Boundaries

The Candidate Pool: Ready to Use or Fully Defined by You

You can select up to 5 custom candidates from existing model services, or directly enable the platform's built-in economy, standard, and flagship model tiers. Both types of candidates can also be compared within the same decision.

Each candidate has a ModelCard that provides structured facts: supported protocols, input and output modalities, tool and reasoning capabilities, context capacity, and the Credit used for routing comparisons. Intelligent Routing does not infer capabilities from ambiguous model names or allow low cost to compensate for missing hard capabilities.

The platform's built-in models are exposed through stable tiers. As the underlying models evolve, you do not need to integrate each new model individually. Custom models continue to use your existing services and credentials, with ownership unchanged.

One point deserves special emphasis: ModelCard Credit is a routing preference measure, not a billing price. You can maintain Credit based on your own contracted prices or internal cost accounting criteria. Actual model charges are settled separately according to the corresponding model service and product billing rules. Thus, auto/cheap compares candidates within the same pool using your own cost criteria, rather than imposing uniform platform pricing.

A Single Fluctuation Does Not Trigger a Model Switch

Intelligent Routing continuously monitors candidate availability and latency. Candidates that are clearly unavailable are excluded at the next safe decision boundary. Once the minimum threshold for valid samples is reached, latency and health signals are incorporated into local scoring for fast and other Profiles.

When samples are insufficient, the system will not label a model "fastest" or "worst" based on one or two isolated requests. Strong continuations already within the same Agent tool-call chain will not switch models solely because of transient soft signals. If the original model violates a deterministic hard boundary, the gateway handles it with explicit failure semantics; business-level fallback remains the responsibility of a separate Fallback policy.

Simple Integration, No New SDK Required

When creating or updating a Model API in the console, select the "Intelligent Routing" backend scenario, configure custom candidates, and enable built-in platform models as needed. It is ready to use after publishing. Callers do not need any additional routing SDK; simply set model to an explicit mode, for example:

{
  "model": "auto/balanced",
  "messages": [
    {"role": "user", "content": "Summarize this material and list three action items."}
  ]
}

If a request continues to specify a particular model name, Intelligent Routing is completely bypassed, preserving existing application behavior. Bare auto is not interpreted as any default policy—the caller must explicitly specify auto/balanced, auto/cheap, auto/fast, or auto/quality.

A Model API using Intelligent Routing enforces consumer authentication, isolating continuity state across tenants through trusted identities. Multiple sessions under the same consumer are still distinguished by protocol-level causal information; the consumer identity itself is not treated as a session ID. Routing state stores only bounded task summaries and necessary correlation information, without persisting complete Prompts, complete histories, or credentials. Bounded summaries are still protected as sensitive business content and are not included in ordinary Traces or logs.

A Few Boundaries

To keep the initial release predictable, we would like to make the following clear up front:

The initial release is available for AI Gateway Model APIs that support this capability. MultiTenantServerless multi-tenant gateways are not supported for now.
Intelligent Routing handles only explicit auto/<profile> requests; requests naming a specific model retain their original path.
Each decision produces only one primary execution model and does not automatically generate a retry or Fallback candidate chain.
The four Profiles express optimization preferences; they do not promise to select the theoretically cheapest, fastest, or most capable model globally on every request.
Continuity depends on standard-protocol evidence observable in the current request. If the context is compressed or causal information is lost, the capability degrades conservatively rather than guessing across sessions.
The built-in platform models, supported protocols, available regions, supported versions, and billing methods are subject to the console and official website announcements at the time of the official launch.

Closing Thoughts

In the multi-model era, the real costs extend beyond Tokens to the selection logic scattered across applications, ongoing parameter tuning, and the risk of failures.

We want Intelligent Routing to bring these responsibilities into a single, clear entry point: give independent tasks room to choose, maintain the stability that multi-turn sessions need, and avoid repeated decisions in Agent tool-call chains. First ensure sufficient capability, then select a lower-cost, faster, or better-suited model according to your business preferences.

Intelligent Routing has already been launched first on AI Gateway dedicated instances. Try it out.

Community

Alibaba Cloud AI Gateway Intelligent Routing Is Now Live

One Request, One Decision

Lower Cost, Faster Responses, or Greater Accuracy—You Decide

Three Types of Calls, Three Routing Cadences

Standalone Requests: Reselection Is Possible Every Time

Multi-Turn Conversations: Switch Only When Appropriate

Agent Tool-Call Chains: No Mid-Task Brain Swaps

The Candidate Pool: Ready to Use or Fully Defined by You

A Single Fluctuation Does Not Trigger a Model Switch

Simple Integration, No New SDK Required

A Few Boundaries

Closing Thoughts

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Token Plan

API Gateway

AgentBay

QwenWork