Intelligent routing: make every model call earn its cost
Plain routing sends every request to one fixed model. Intelligent routing first understands the request, then picks the best-fit model and endpoint under quality, cost, latency, privacy, and safety constraints.
Billing support questionLightweight instruct models
Multi-file code refactorFlagship coding models
Patient record summaryAttested private deployments
Long document synthesisLong-context flagship models
Same goal, lower cost with intelligent routing
The numbers below are local demo data used to illustrate the product advantage: without lowering the quality threshold, simple requests go to cheaper models while high-risk or hard tasks stay on dedicated lanes.
Billing support question
Versus routing everything to a fixed strong model, it keeps the quality threshold while cutting unnecessary large-model calls.
Signal panel
Fixed routing answers “can it run”; smart routing answers “is it worth it”
How plain routing runs
Every request goes to the same model or a set of static weights, so even simple tasks burn an expensive model.
Support, code, privacy, and long-document requests are all treated the same, with no differentiation.
All traffic goes through one lane — no signal analysis and no policy branching.
Quality is acceptable, but even the simplest tasks burn the most expensive compute.
How intelligent routing runs
Keep one complete routing chain: pick a task type on the left, match its rule and policy in the middle, and land on different model pools and endpoints on the right.
Classify task type, complexity, context, privacy, and cache similarity.
- Lightweight tasks
- General tasks
- Strong reasoning tasks
- Private tasks
Merge signals into a decision vector, then hit concrete rules by priority.
complexity = simple AND cache_similarity > 0.9Run semantic cache, cost gates, prompt injection, verification, and redaction.
- Semantic cache lookup
- Lightweight model pool
- Support knowledge injection
Select the candidate pool by quality, cost, latency, and deployment boundaries.
- Target model pool
- Lightweight instruct models
- Serving endpoint
- AnyInt low-cost pool
Signals in, policy out
Routing is a configurable orchestration layer, not random dispatch. Multi-dimensional signals profile every request, rules compose them into policies, and each policy binds the plugin chain that runs before, during, and after the model call.
Cost policies
When complexity and cache signals allow, check the semantic cache first, then climb from lightweight pools only as far as quality demands.
Quality policies
When a fact-check or schema signal fires, add verification and repair stages instead of blindly paying for a bigger model.
Compliance policies
When sensitive spans are detected, restrict the candidate pool to attested deployments, redact before logging, and write an audit record.