Model Routing
Automatically sending each request to the most suitable model, cheap or capable, local or hosted, based on the task, its difficulty, cost limits or data rules.
Definition
No single LLM is best for every request. Frontier models reason well but cost more and respond more slowly; small models are fast and cheap but weaker on hard tasks; self-hosted models keep data in-house. Model routing is the layer that chooses. It inspects each request, or the feature it came from, and dispatches it to the model that meets the quality bar at the lowest cost and risk. The same layer usually provides fallbacks when a provider is down or rate-limited.
Routing logic can be static (this feature always uses this model), rule-based (long documents go to the large-context model, personal data stays on-premises), learnt (a classifier predicts difficulty) or cascaded (try the cheap model, escalate if it is unsure or fails validation). All of them depend on evals to know which model is good enough for which task.
Example: a support platform routes greeting and FAQ messages to a small model, complex billing disputes to a frontier model, and any message containing a card number to a self-hosted model that never leaves the company's network. Cost per conversation drops without visible quality loss.
Why it matters for your business
- It is the main lever on inference cost once prompts have been trimmed.
- It decouples your product from one provider, which protects against price changes and outages.
- It lets data rules be enforced in code: sensitive requests simply cannot reach an external model.
- It is easy to over-engineer; most teams need three tiers and clear rules, not a learnt router.
How Teksmobile uses it
We build routing and fallback into the platform layer of LLM applications from the start, and we monitor the cost and quality of each route through our LLMOps and monitoring service. The LLM cost calculator shows what a multi-tier routing policy saves before you build it.
Related terms
Part of the Teksmobile AI Glossary. See also our guide to small language models.
Frequently asked questions
How does the router decide?
Options range from fixed rules per feature, to a small classifier that predicts difficulty, to a cascade where a cheap model answers first and escalates when confidence is low. Rules per feature are the usual starting point.
Does routing add latency?
A rule-based router adds almost none. A classifier adds a few milliseconds. A cascade can add a full extra call when it escalates, so it suits workloads where most requests are easy.
Is a routing layer another thing to maintain?
Yes, and it should be treated as part of the platform, with its own evals so that a change in routing rules is tested like a change in prompts.