Model routing is a systems optimization problem

Posted token prices do not determine cost per completed task. Routing policy, the rules that select a model for each request or step, needs records from the workload and live signals from the serving system. Reuse of previously processed context, often called caching, trajectory length, endpoint state, and failures all affect the amount that operators actually pay.

In IBM Research’s test of 417 AppWorld Test Challenge tasks run with the same CodeAct agent, Sonnet cost $79 total, or $0.19 per task, while GPT-4.1 cost $155 total, or $0.37 per task. The authors attribute that result to the price of reusing processed context: the workload reused large portions of context across steps, and Sonnet’s lower price for that reuse outweighed its higher posted base prices and roughly three times as many reasoning steps. The workload changed the cost comparison.

These are IBM Research’s benchmark observations, not universal model rankings or production guarantees. IBM Research, Hugging Face, and the named model providers are neither Muniment customers nor endorsers.

A production decision framework

Use a bounded evaluation before enabling a routing policy in production. Fix the task sample, evaluation period, and decision rules in advance so a promising result cannot quietly redefine success.

1. Set before launch. Define the minimum acceptable outcome or quality score for each task class, the maximum cost per completed task, and the maximum end-to-end response time. Set reliability requirements for successful completion, timeouts, retries, and fallback behavior. Apply data-region, retention, model-approval, contractual, and task-risk rules first; an ineligible model is not a candidate. Record what will stop or roll back the evaluation if any threshold is breached.
2. Measure during the bounded evaluation. For every attempted task, record its class, selected model and path, outcome, total cost through completion, end-to-end response time, retries, timeouts, partial failures, fallback use, context reuse, endpoint health, and the policy version. Report failed and incomplete tasks beside completed ones. Compare only eligible choices on the same task sample and serving conditions, without selecting a winner from token price or quality alone.
3. Review before launch and after policy changes. Check every threshold against the full sample, including zero-completion task classes and paths with too little evidence. Investigate whether endpoint load, stale health data, context reuse, or fallbacks produced the result. Approve, reject, or narrow the policy and record the evidence, owner, date, and rollback condition. Repeat the bounded evaluation after changes to models, prices, prompts, tools, eligibility rules, fallback order, or serving infrastructure.

No operating point is universally best. Enable a policy in production only when it meets the organization’s quality, completed-task cost, response-time, reliability, and governance requirements on its own workload.

The completed task is the accounting unit

A token table prices isolated input and output units. An agent task produces a trajectory: repeated context, reuse reads and writes, tool waits, retries, and sometimes an escalation to another model. Routing against the table alone assumes those paths are equivalent before the workload has supplied any evidence that they are.

A task record provides the required accounting unit. It records task class, selected model and path, newly processed and reused tokens, context-reuse rate, trajectory length, retries, end-to-end response time, outcome, total cost, and the governing policy context that made the path eligible. That joins model choice, context reuse, user-visible time, and policy in one record. Operators can then compare cost per accepted or otherwise completed task within their own workload. A cheap failed attempt that triggers an expensive retry has not lowered task cost.

Execution can change the routing problem

IBM Research notes that task difficulty can stay hidden until execution is underway. A contract summary can expand into retrieval, compliance checks, tool calls, and revisions after an initial path has been selected. Per-task routing pays for one decision and must act on the evidence available at the start. Per-step routing can adapt as the trajectory exposes difficulty and serving conditions, but each decision adds response time and operational complexity. Granularity is therefore another operating choice, not a free increase in routing intelligence.

IBM Research reports that its optimization used roughly 6 ms and 2 kB of memory per task. Those are measurements from IBM Research’s reported benchmark implementation, with the unit attached to the task. They are evidence that this optimization stayed lightweight in that test, not a general overhead guarantee for other policies, workloads, or routing granularities.

IBM Research’s response-time-oriented result makes the same point from another operating position.

IBM Research reports those values for its response-time-optimized configuration in its AppWorld test and compares them with an Opus-only baseline. The accuracy change is four percentage points, while the cost and response-time changes are relative percentages. It is one test operating point with an explicit quality trade, not a promise that the same routing policy will reproduce those changes on another workload or serving stack.

Serving state belongs inside the route

The IBM Research authors argue that hardware, availability of reusable processed context, and endpoint load can dominate end-to-end response time. A path therefore needs current infrastructure signals alongside historical task records. Queue depth, regional availability, rate limits, endpoint health, and location of reusable context can reverse a decision that looked sensible in a static comparison.

Reliability changes the calculation again. A candidate with a low nominal price but a high timeout or retry rate can raise both cost and user-visible response time. The routing policy needs recent success and error rates, bounded fallbacks, and a record of partial failures. Those signals support a local decision under current conditions. They do not establish a universally superior provider or model.

Governance narrows the eligible set before optimization begins. Data-region rules, retention policy, model approvals, contractual limits, and task-specific risk can exclude a path regardless of its benchmark score. Cost, quality, and response time should be optimized only among endpoints that the policy permits. Otherwise the policy has optimized an option the organization cannot use.

The operating question is concrete: for this class of task, under this policy and serving state, which eligible path produces an acceptable completed outcome at the best observed cost and response time? Token prices remain an input. Measured task data determines how much that input matters.