Microsoft Foundry’s model router uses a trained model to analyze each prompt and choose an eligible LLM. Teams can also start with explicit rules.
Model routing pays for expensive capability only when the request is likely to benefit from it.
What signals should a model router use?
Start with hard requirements such as vision input, data residency, context length, tool support, or an approved provider list. These constraints define the eligible pool before optimization begins. Then consider task type, estimated difficulty, quality, price, and time to first token. Security and capability requirements must filter the pool before cost can influence the choice.
A simple router may use explicit rules. A learned router can inspect the prompt, conversation history, and tool definitions to estimate which eligible model is most likely to succeed. Neither approach removes the need for a fallback when the chosen model fails a check.
How does a router balance cost, latency, and quality?
The useful decision is not small model versus large model in the abstract. It is the least expensive eligible model that meets the acceptance criteria for this class of request. Routing needs a stated objective because cost, speed, and quality cannot all be maximized at once.
High-volume classification may favor a fast model. A vision task must reach a model that supports images. Complex reasoning or multi-step tool use may justify a stronger model. Direct routing remains appropriate when policy or reproducibility requires the same model every time.
How do you know whether routing works?
Compare the router with a fixed baseline on representative traffic. Measure quality, total cost, median and tail latency, and policy compliance for every route. Record which model handled every inference call. A router is successful only when workload-level results improve without hiding regressions in important categories.
A fallback can hide a weak first choice while doubling cost and latency. Track fallback and retry rates separately rather than crediting the first route for the eventual answer. LLMOps should also monitor how traffic distribution changes as prompts, models, and pricing evolve.
When is model routing unnecessary?
Use one model when the workload is narrow, the same capability is required every time, or the traffic volume is too small for routing savings to repay operational complexity. A router with one viable destination is only another failure point. A multi-provider AI architecture is justified only when each added route solves a measured requirement.
Reasoning models should be reserved for tasks where evaluation shows that extra computation changes the outcome. Our model profiles and context management guide shows a practical routing boundary inside an agent workflow.