Community Blog Model Routing Explained: Send Each Request to the Cheapest Model That Passes

Model Routing Explained: Send Each Request to the Cheapest Model That Passes

Every platform team serving Qwen at volume faces the same choice: send every request to the flagship, or route each one to the cheapest model that pas...

Model Routing Explained: Send Each Request to the Cheapest Model That Passes, Qwen Enterprise AI Series cover

One million requests a month on qwen3.8-max: $7,000. The same traffic routed across Flash, Plus and Max, with a classifier and a fallback: $1,644. The traffic did not change. The address did.

One million requests a month cost $7,000 all on qwen3.8-max and $1,644 routed 60/30/10 across Flash, Plus and Max with a router and fallback

Price the request by what it needs, prove it with a gate, and pay the escalation tax only when the gate says so.

In this guide

The request path, step by step, from one endpoint to a logged decision

Rules, classifier or cascade: which routing strategy fits your traffic

A task-to-tier map for the Qwen ladder, with the gate that decides each route

Worked savings math, including the escalation tax most dashboards hide

A five-step start that ships one paying route this week

01 · What is LLM model routing, and why does it matter now?

LLM model routing is a decision layer in front of your models that sends each request to the least expensive model expected to pass your quality bar, and moves it up a tier when it does not.

It is the same idea as a card authorization: most of the work happens in milliseconds, before anyone sees a result, and the record settles afterward.

It matters now because inference, not training, is where the money goes. Gartner expects inference to take 55% of AI-optimized IaaS spending in 2026, $23.3B against $19B for training (Gartner, AI-optimized IaaS spending forecast, Aug 2026). Every default you set on the request path is multiplied by that curve.

Gartner expects inference to take 55% of AI-optimized IaaS spending in 2026, $23.3B against $19B for training

Platform teams sit between two fears. Send everything to the flagship and finance asks why ticket tagging runs on your most capable model. Send everything to the smallest model and the first bad answer on a hard request becomes an incident review. Both defaults are expensive, one in dollars and one in trust.

The answer is not a better default. It is no default: a router that matches each request to a tier, a gate that checks the output, and a log that tells you whether the match was right.

Before you write the first rule, get a routing read on your traffic mix from the team: which task types look like Flash work, and which genuinely need Max.

The flagship is a tier you escalate to, not a place you live.

02 · How does a routed request move, step by step?

A routed request takes seven steps, and most of them finish before anyone sees a result.

1. The request reaches one endpoint. On Alibaba Cloud Model Studio, the whole Qwen ladder sits behind one compatible-mode endpoint per region (Singapore: dashscope-intl.aliyuncs.com/compatible-mode/v1), so changing tiers means changing the model parameter, not the client.

2. The router reads the request's features. Task type, input length in tokens, tools or images involved, tenant tier, language.

3. The router picks a tier and a fallback. It returns two things: the first model to try, and the next model if that one fails the gate. A route without a declared fallback is a guess.

4. The model runs. Same prompt template, same output schema, on every tier.

5. The gate checks the output. Schema parses, required fields present, values in allowed sets, citations present where required. Fast checks run on every response; slower judge checks run on a sample.

6. Pass returns, fail escalates. A pass goes back to the caller. A fail goes up one tier, at most twice, then returns a safe failure or a human handoff. Unbounded escalation is how a cost control becomes a cost leak.

7. The decision is logged. Route, tier, tokens, gate result, latency, cost. This is your settlement file: it shows which routes pay and which bounce.

Seven-step routed request path: endpoint, features, tier and fallback, model run, gate, pass or escalate, log

If a step is not logged, you cannot tell a good route from a lucky one.

03 · Which LLM model routing strategy fits: rules, classifier, or cascade?

Step 3 can be decided three ways.

Start with the first; add the others where the data says to.

1. Rules. If-then logic on the features from step 2: "extraction tasks go to Flash; anything with a tool call goes to Plus; inputs over 256K tokens go to Max." Rules are free, instant and easy to audit. They break when difficulty varies inside a task type, which it usually does.

2. Classifier. A small, fast model labels the request's difficulty or intent, and the label picks the tier. qwen3.8-flash can run it. It adds a call per request, so keep it short: a few hundred input tokens and a one-word label.

3. Cascade with fallback. Try the cheapest plausible tier first, run the gate, and escalate on failure. It needs no prediction at all, only a reliable gate.

Rules, classifier and cascade compared on how they decide, added cost, added latency, best fit and failure mode

The strongest pattern is a hybrid: rules for the obvious cases, a classifier for mixed traffic, and a cascade fallback behind both so that a wrong route costs an escalation instead of a bad answer. Whether you write that layer yourself or run it on a managed gateway is the same build-or-partner call we broke down in Build vs. Buy, or Both? The Enterprise AI Decision: own the routing policy, and partner for the plumbing that is not your edge.

Rules first, classifier second, cascade always as the net.

04 · Which Qwen model should each task type get?

The Qwen ladder gives you three tiers on one bill.

List prices below are Model Studio, Singapore, per 1M tokens (Alibaba Cloud Model Studio pricing page, accessed Oct 5, 2026):

Task-to-tier map: classification, extraction and summaries start on qwen3.8-flash at $0.15/$0.47, grounded answers and policy drafts on qwen3.7-plus at $0.40/$1.60, multi-step reasoning on qwen3.8-max at $2.00/$6.00

Two price details change routes. First, qwen3.7-plus steps up to $1.20 in and $4.80 out per 1M tokens once input passes 256K tokens, while qwen3.8-max stays at $2.00 / $6.00 flat to 1M. For very long inputs the gap narrows, so a length rule belongs in the router. Second, qwen3.8-flash carries a 1M-token context and multimodal input, so long or image-bearing requests do not automatically need a bigger tier.

Route on the task's hardest step, not its longest input.

05 · What does routing save on 1M requests a month?

Here is one illustrative month of traffic, with every assumption on the table.

Assumptions (illustrative).
Volume: 1,000,000 requests per month.

Tokens: every request averages 2,000 input tokens and 500 output tokens on every tier (a simplification; output length varies by model and prompt).

Mix: traffic splits 60% Flash, 30% Plus, 10% Max after routing.

Classifier: runs on qwen3.8-flash with 300 input tokens and 5 output tokens.

Escalation: 10% of Flash requests fail the gate and escalate to Plus.

Prices: Singapore list prices, no batch or cache discounts.

Worked math: $0.0070, $0.0016 and $0.000535 per request on Max, Plus and Flash; routed bill $1,548, or $1,644 with $96 of escalations, against $7,000 all on Max

Result: routing takes the monthly bill from $7,000 to $1,644, 76.5% lower, after paying for every escalation.

The $96 is the escalation tax: what you pay for running a cheap attempt that did not pass. It is the honest price of routing, and worth watching more closely than the headline saving.

Where the cascade stops paying. A Flash-first attempt costs $0.000535. If it escalates to Plus, the pair costs $0.002135 against $0.0016 for going to Plus directly. Flash-first is cheaper on average as long as fewer than 66.6% of attempts escalate (1 - $0.000535 / $0.0016).

For a Flash-to-Max cascade the break-even escalation rate is 92.4% (1 - $0.000535 / $0.007).

A Flash-first cascade stays cheaper while fewer than 66.6% of attempts escalate to Plus, or 92.4% to Max

On price, cascades almost always win. On latency they can lose, because every escalated request waits for two calls. That is why the escalation rate, not the price list, is the number to set targets on.

See your escalation tax before you build a router
Your mix is not 60 / 30 / 10, and the math moves fast. Send your task types, token averages and current tier, and the team will price your routed mix, the escalation tax and the cascade break-even on your own traffic. Price my routing mix on real traffic →

Report cost per passed request, and keep the escalation tax on the same dashboard.

06 · How do you measure "passes"?

"The cheapest model that passes" is only a policy if "passes" is a number you set before you route.

Two kinds of gates do the work, and two habits keep them honest.

1. Offline gates decide which tier a task may start on. Build a golden set per task type from real, labeled traffic. Run every tier against it and score the results.

Model Studio's Model Evaluation supports LLM-as-judge numeric and classification scoring, string match, text similarity (ROUGE-L, BLEU, cosine) and human annotation, and rule-based scoring carries no judge fee (Model Studio Model Evaluation documentation, accessed Oct 5, 2026). A tier earns a task when it scores within a margin you choose of the top tier on that set. Write the margin down.

2. Online gates decide whether a single response passes. These run in the request path, so they must be cheap: schema validation, allowed values, required citations, length bounds, refusal detection. A slower judge check runs on a sample of traffic per route, not on every call.

3. Outcome signals check the gates themselves. User edits, re-asks within a session, handoffs to a human, and downstream errors. If a route passes the gate but users re-ask at twice the rate of the Max route, the gate is too lenient.

4. Re-run the offline set on every change. New model snapshot, new prompt template, new router threshold: each one re-earns its routes. A route that passed in March has not passed in October until you run it again.

Four ways to measure passes: offline gates, online gates, outcome signals and re-running the offline set on every change

A gate you cannot write as a test is an opinion, not a gate.

07 · What breaks in production, and when should you not route?

Five failures show up first.

1. Gate drift. The gate was tuned on last quarter's traffic. Inputs change, the gate stays still, and pass rates rise while quality falls. Fix: re-score a weekly sample against the golden set.

2. Escalation storms. A prompt change makes Flash fail a gate it used to pass, and 40% of traffic suddenly pays twice. Fix: alert on escalation rate per route, not just total spend.

3. Tail blindness. Averages look fine because most requests are easy. The few that matter fail quietly on the small tier. Fix: score the golden set by segment, and pin high-stakes segments to a higher tier by rule.

4. Latency stacking. Cascades double the wait on every escalated request. Fix: skip the cheap attempt when the user is waiting and the classifier is uncertain.

5. Prompt forking. Each tier gets its own tuned prompt, and three prompts drift apart. Fix: one template, one schema.

When not to route

Routing is a system with its own cost. Do not build it when:

Volume is small. At 10,000 requests a month, the all-Max bill in the example above is $70. A router will cost more to build and maintain than it saves.

There is one task type. If every request is the same kind of work, pick the cheapest tier that passes the offline gate and stop. That is routing with one route.

The top tier also fails. If qwen3.8-max does not pass your golden set, routing will only make a broken workload cheaper. Fix the task, the data or the prompt first.

One validated model is a requirement. Some reviewed workflows require a single, documented model for every output.

Latency budgets are tight and gates are slow. If you cannot gate inside the budget, route by rules only, with no cascade.

Route when the mix is wide and the volume is real; otherwise, choose once and measure.

08 · Start LLM model routing this week

You do not need a classifier on day one.

You need a log, a golden set and one rule.

1. Tag your traffic. Add a task-type field to every call and log tokens, latency and cost per request for one week.

2. Build three golden sets. Pick your three highest-volume task types and label a few hundred real examples each.

3. Score the ladder. Run qwen3.8-flash, qwen3.7-plus and qwen3.8-max on each set in Model Studio. New Singapore users get a free quota of 1M tokens per model for 90 days, which covers a first evaluation round (Model Studio pricing page, accessed Oct 5, 2026).

4. Ship rules for the clear wins. Move each task type to the cheapest tier inside your margin. Keep a fallback to the next tier, capped at one hop.

5. Put the escalation tax on the dashboard. Track cost per passed request and escalation rate per route. Add a classifier only when a single task type shows a wide difficulty spread.

Five-step start: tag traffic, build three golden sets, score the ladder, ship rules, track the escalation tax

When you outgrow hand-written rules, AgentRun's model gateway provides multi-model load balancing and fallback, with token-level cost attribution (AgentRun documentation, accessed Oct 5, 2026). Model Studio's call statistics and cost analysis give you the settlement view per model. One model family, one bill, three tiers: the router decides, the gate proves, the log pays.

Start with one route that pays, then earn the next one.


Key takeaways

Set no default tier. A router, a gate and a log send each request to the cheapest Qwen model that passes, and escalate only on a fail.

Watch the escalation tax. In the worked example routing takes $7,000 to $1,644 a month; the $96 of escalations is the number to set targets on.

Write "passes" as a test. Golden sets per task type, cheap online gates and outcome signals keep routes honest as traffic drifts.

Work through your first route and gate with the Qwen team

Turn three golden sets into your first paying route
Tell us your three highest-volume task types and how you judge a good answer today; rough counts are enough to start. The team will work through gate design, tier margins and a one-hop fallback with you, so the first route ships with a test behind it. Work through my first route and gate →

See more of the enterprise AI portfolio at qwen.ai. Lasting returns come from treating AI like infrastructure: planned over years, delivered in stages, with Qwen alongside you so the early choices hold up and the value keeps growing.


About Qwen. Qwen gives enterprises an open route into production AI: one model family covering language, vision, audio, coding, embeddings, and agents, as open weights and managed APIs. Built on curated data and ongoing research, Qwen supports creating, testing, and running AI systems for decisions that matter, from self-serve developer access to dedicated inference.

Sources: Gartner, Gartner Forecasts Worldwide AI-Optimized IaaS Spending to Grow 96% Through 2026 (Aug 2026); Alibaba Cloud Model Studio, Model pricing (accessed Oct 5, 2026); Alibaba Cloud Model Studio, Model evaluation (accessed Oct 5, 2026); Alibaba Cloud Model Studio, What is Model Studio (accessed Oct 5, 2026); Alibaba Cloud, What is AgentRun (accessed Oct 5, 2026).

Johnny Mai - Director, Gen AI Product Strategy & GTM, Qwen.


Disclaimer: This post is provided for general information only. Worked examples are illustrative and use stated assumptions; your costs will vary. Prices are Alibaba Cloud Model Studio list prices for the Singapore region as published in October 2026, vary by region, and may change over time. Figures from third-party research (Gartner 2026) are quoted as published in those reports and are not independently verified by Qwen or Alibaba Cloud. Product and service availability, features, and program terms vary by region and may change over time. Nothing in this post constitutes professional, legal, or investment advice. © 2026 Qwen. All rights reserved.

0 1 0
Share on

Johnny Mai

22 posts | 2 followers

Comments