Inference routing is a systems problem

Inference routing selects an eligible model for each request or reasoning step. It is a systems optimization problem rather than a lookup against posted token prices. A useful policy needs workload records and live serving signals to balance completed-task cost, response time, and reliability. Context reuse, trajectory length, endpoint state, and failures can change every part of that balance.

muniment’s bring your own key (BYOK) model means an organization supplies its model provider credentials. Routing only selects among its eligible paths.

In IBM Research’s test of 417 AppWorld Test Challenge tasks run with the same CodeAct agent, Sonnet cost $79 total, or $0.19 per task, while GPT-4.1 cost $155 total, or $0.37 per task. The authors attribute that result to the price of reusing processed context: the workload reused large portions of context across steps, and Sonnet’s lower price for that reuse outweighed its higher posted base prices and roughly three times as many reasoning steps. That workload changed the cost comparison.

These are IBM Research’s benchmark observations, not universal model rankings or production guarantees. IBM Research, Hugging Face, and the named model providers are neither Muniment customers nor endorsers.

The baseline decides the saving

A model router is software that selects a model for each request. DigitalOcean reports that tiered routing saved about 39.6 percent against hardcoding Claude Sonnet 4.5. DigitalOcean reports that the same routing saved about 7.7 percent against a GPT-5 baseline. The baseline decides the saving, so a percentage without its baseline says little about another workload.

DigitalOcean reports that a router adds approximately 200 milliseconds of delay. DigitalOcean names the routing decision’s time, a second failure point, and a misroute as three router costs. DigitalOcean warns that a misrouted request returns an answer rather than an error, with nothing marking the answer as wrong.

A fallback is the second choice after a failure. It limits one failure path, but it does not remove the router’s delay or misroute risk. DigitalOcean is neither a Muniment customer nor an endorser.

A benchmark needs both sides of the route

The LLMRouter authors define model routing across single-turn, multi-turn, and personalized settings. Model routing means choosing a model for each request. Their xRouteBench benchmark evaluates routers on response quality and inference cost together. Inference cost means the provider charge for a model response.

LLMRouter’s authors report that learned routers beat the strongest fixed-model baseline by 14.6 percent relative. This result comes from their research, not a production outcome. The authors and their institutions do not use or endorse Muniment.

A joint benchmark supplies the score, but an operator still needs the decision record. Each route must connect the request, selected model, quality result, and provider charge. Without that record, a benchmark cannot show whether a live policy reproduced its measured tradeoff.

A production decision framework

Use a bounded evaluation before enabling a routing policy in production. Fix the task sample, evaluation period, and decision rules in advance so a promising result cannot quietly redefine success.

1. Set before launch. Define the minimum acceptable outcome or quality score for each task class, the maximum cost per completed task, and the maximum end-to-end response time. Set reliability requirements for successful completion, timeouts, retries, and fallback behavior. Apply data-region, retention, model-approval, contractual, and task-risk rules first; an ineligible model is not a candidate. Record what will stop or roll back the evaluation if any threshold is breached.
2. Measure during the bounded evaluation. For every attempted task, record its class, selected model and path, outcome, total cost through completion, end-to-end response time, retries, timeouts, partial failures, fallback use, context reuse, endpoint health, and the policy version. Report failed and incomplete tasks beside completed ones. Compare only eligible choices on the same task sample and serving conditions, without selecting a winner from token price or quality alone.
3. Review before launch and after policy changes. Check every threshold against the full sample, including zero-completion task classes and paths with too little evidence. Investigate whether endpoint load, stale health data, context reuse, or fallbacks produced the result. Approve, reject, or narrow the policy and record the evidence, owner, date, and rollback condition. Repeat the bounded evaluation after changes to models, prices, prompts, tools, eligibility rules, fallback order, or serving infrastructure.

No operating point is universally best. Enable a policy in production only when it meets the organization’s quality, completed-task cost, response-time, reliability, and governance requirements on its own workload.

The completed task is the accounting unit

A token table prices isolated input and output units. An agent task produces a trajectory: repeated context, reuse reads and writes, tool waits, retries, and sometimes an escalation to another model. Routing against the table alone assumes those paths are equivalent before the workload has supplied any evidence that they are.

A task record provides the required accounting unit. It records task class, selected model and path, newly processed and reused tokens, context-reuse rate, trajectory length, retries, end-to-end response time, outcome, total cost, and the governing policy context that made the path eligible. That joins model choice, context reuse, user-visible time, and policy in one record. Operators can then compare cost per accepted or otherwise completed task within their own workload. A cheap failed attempt that triggers an expensive retry has not lowered task cost.

Execution can change the routing problem

IBM Research notes that task difficulty can stay hidden until execution is underway. A contract summary can expand into retrieval, compliance checks, tool calls, and revisions after an initial path has been selected. Per-task routing pays for one decision and must act on the evidence available at the start. Per-step routing can adapt as the trajectory exposes difficulty and serving conditions, but each decision adds response time and operational complexity. Granularity is therefore another operating choice, not a free increase in routing intelligence.

IBM Research reports that its optimization used roughly 6 ms and 2 kB of memory per task. Those are measurements from IBM Research’s reported benchmark implementation, with the unit attached to the task. They are evidence that this optimization stayed lightweight in that test, not a general overhead guarantee for other policies, workloads, or routing granularities.

IBM Research’s response-time-oriented result makes the same point from another operating position.

IBM Research reports those values for its response-time-optimized configuration in its AppWorld test and compares them with an Opus-only baseline. The accuracy change is four percentage points, while the cost and response-time changes are relative percentages. It is one test operating point with an explicit quality trade, not a promise that the same routing policy will reproduce those changes on another workload or serving stack.

Serving state belongs inside the route

The IBM Research authors argue that hardware, availability of reusable processed context, and endpoint load can dominate end-to-end response time. A path therefore needs current infrastructure signals alongside historical task records. Queue depth, regional availability, rate limits, endpoint health, and location of reusable context can reverse a decision that looked sensible in a static comparison.

Reliability changes the calculation again. A candidate with a low nominal price but a high timeout or retry rate can raise both cost and user-visible response time. The routing policy needs recent success and error rates, bounded fallbacks, and a record of partial failures. Those signals support a local decision under current conditions. They do not establish a universally superior provider or model.

Governance narrows the eligible set before optimization begins. Data-region rules, retention policy, model approvals, contractual limits, and task-specific risk can exclude a path regardless of its benchmark score. Cost, quality, and response time should be optimized only among endpoints that the policy permits. Otherwise the policy has optimized an option the organization cannot use.

The operating question is concrete: for this class of task, under this policy and serving state, which eligible path produces an acceptable completed outcome at the best observed cost and response time? Token prices remain an input. Measured task data determines how much that input matters.