# Small models belong at the front door [Journal](/journal/) July 20, 2026 · [evidence](/journal/#evidence) A local first reader can route each query by policy, cost, and latency, but its benchmark and inference stack must earn the right to decide. ![A query trace enters a compact verdigris routing dial, which selects one of three near-black model paths.](/_astro/small-models-at-the-front-door.DuEq486t_Z1mBkuD.webp) The first model to read a query should be the smallest model trusted to decide where that query goes. Put that decision on the user’s machine and routing becomes a control point for cost, latency, and governance before a large remote model sees the prompt. The benchmark evidence makes this architecture credible. It does not make the router ready to run unattended. Front-door routing gives one local decision three possible consequences: handle the request locally, send it through a governed proxy, or escalate it to a large model. Economics improve when expensive capacity is reserved for work that needs it. Latency improves only when classification overhead stays below the time it saves. Privacy improves structurally because the first reader can remain under the user’s control. The classifier still has to be right. A fast local mistake can select an inadequate model, cross a policy boundary, or trigger a costly retry. The front door deserves stricter validation than a convenience feature because every later path inherits its choice. ## The benchmark supports a candidate, not a verdict Johnson and Lee tested zero-shot task classification on a fixed corpus under matched serving conditions, then ran a synthetic-traffic experiment. Their [front-door routing benchmark reports the following comparison](https://arxiv.org/abs/2604.02367). Accuracy is exact match, and latency is measured at the classifier rather than across the downstream request. Study Model Exact-match accuracy Result Study 1 Qwen2.5-3B 0.783 Best accuracy and latency tradeoff among local candidates Study 2 Qwen2.5-3B 0.793 at 988 ms median Passes the latency gate and misses the accuracy gate Study 2 DeepSeek-V3 0.830 at 2,295 ms P95 Highest accuracy and fails both gates Study 2 Phi-4-mini 0.518 Misses the accuracy gate Qwen2.5-3B is the useful operating point here. It reached [0.783 exact-match accuracy in the harmonized study](https://arxiv.org/abs/2604.02367) and [0.793 accuracy with 988 ms median latency in the synthetic-traffic study](https://arxiv.org/abs/2604.02367). DeepSeek-V3 produced the [highest accuracy at 0.830](https://arxiv.org/abs/2604.02367), yet its [2,295 ms P95 latency failed the pre-registered gate](https://arxiv.org/abs/2604.02367). More model bought some accuracy and lost the operating constraint. The negative result matters most: [no tested model satisfied the standalone viability rule of at least 0.85 accuracy and less than 2,000 ms P95 latency](https://arxiv.org/abs/2604.02367). The paper identifies targeted prompt work, taxonomy revision, and supervised fine-tuning as remediation paths. Until one of those paths clears a representative gate, the safe architecture needs abstention, fallback, and a record of why each route was chosen. The paper also reports [Phi-4-mini at 0.518 accuracy](https://arxiv.org/abs/2604.02367), below the earlier Phi generation in a cross-study comparison. Its authors explicitly warn that model version, tokenizer differences, prompt sensitivity, and instruction tuning are confounded. Release sequence is not a routing policy. Microsoft and Alibaba appear here as the providers of benchmarked model families. Google appears as the publisher of the Gemma material below. These companies are neither Muniment customers nor endorsers; their public work is evidence, not approval. ## A router inherits its inference stack Model weights are only half of a local classifier. The serving implementation determines whether those weights execute as designed. A llama.cpp issue [for Gemma 4 E2B and E4B describes a particularly dangerous failure mode](https://github.com/ggml-org/llama.cpp/issues/22243): the loader reads the Per-Layer Embeddings metadata while the forward graph omits the residual signal from decoder layers. The models continue to run without crashing, with subtly degraded output quality. The same issue raises [quantization-sensitivity questions for the PLE weights](https://github.com/ggml-org/llama.cpp/issues/22243). Silent degradation is worse than a clean refusal at the front door. A crash trips a fallback. A plausible classification from an incomplete graph can quietly send work down the wrong path. Router qualification therefore binds the model artifact, quantization, inference engine, hardware backend, prompt, taxonomy, and test corpus into one versioned unit. Change any member and the evidence expires until the unit is tested again. That requirement will matter as the on-device field expands. Google introduced [Gemma 4 under the Apache 2.0 license in E2B, E4B, 26B mixture-of-experts, and 31B dense sizes](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/). More capable edge models widen the candidate set. They also widen the matrix of runtimes, conversions, quantization choices, and hardware paths that can invalidate an otherwise persuasive score. ## The first reader sets the boundary Muniment’s execution surfaces are being built around a required on-device model performing this routing role. Owners will define the map, while each query will meet a local classifier before an eligible model or proxy. That direction makes governance an input to routing rather than a review performed after disclosure. The evidence sets the bar for that direction. The deciding model needs a workload-specific evaluation, explicit latency and accuracy gates, bounded fallback behavior, and a durable route receipt. It also needs conformance tests for the exact inference stack delivered to users. A small model at the front door earns its place by deciding under policy and showing its work. Size alone grants no authority. ## Sources 1. [arXiv — Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment](https://arxiv.org/abs/2604.02367) arxiv.org 2. [llama.cpp on GitHub — Bug / Missing Feature: Gemma 4 E2B/E4B Per-Layer Embeddings (PLE) not implemented in forward graph](https://github.com/ggml-org/llama.cpp/issues/22243) github.com 3. [Google — Gemma 4: Byte for byte, the most capable open models](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/) blog.google ## Continue reading [All publications](/journal/) evidence · Jul 19, 2026 ### [Agent governance is a tiered operating model](/journal/agent-governance-is-a-tiered-operating-model/)