Local inference is becoming a real routing tier
Large open-weight models can now run on local machines, which gives governed routing policies a credible on-device tier with hard operational limits.
Local execution has become credible as the lowest tier in a governed routing policy. When an organization’s policy or contract requires work to remain on-device, a router can now send some jobs to very large open-weight models without pretending that every workload belongs there.
The evidence is narrow. DeepSeek publishes V4 Flash as a 284-billion-parameter mixture-of-experts model with 13 billion parameters active and MIT-licensed weights. Z.ai publishes GLM-5.2 weights under the MIT license. Two project authors have built local runtimes around those weights. Their measurements show feasibility on personal machines, not equivalence to closed frontier systems and not independent evaluation.
The floor now has evidence
DS4 uses an asymmetric 2-bit scheme that quantizes only routed mixture-of-experts weights while leaving shared experts, projections, and routing untouched. Its current documentation says the 2-bit Flash quant is 81 GB. That footprint fits a high-memory laptop class without reducing the claim to a hardware shopping list.
Colibri takes another route: it keeps GLM-5.2’s dense portion resident and streams routed experts from a roughly 370 GB int4 model on local storage. The consequence is a much lower demonstrated RAM floor and a much slower cold decode.
| Model and source | Scale | Active | Machine | Footprint | Speed | Status |
|---|---|---|---|---|---|---|
| DeepSeek V4 Flash via DS4 | 284B | 13B | M3 Max MacBook Pro, 128 GB | 81 GB q2 | 26.68 tok/s generation | measured |
| GLM-5.2 via Colibri | 744B | roughly 40B | WSL2, 12 cores, 25 GB RAM, NVMe via VHDX | roughly 370 GB int4; roughly 20 GB peak RSS | roughly 0.05 to 0.1 tok/s cold | measured |
| GLM-5.2 via Colibri | 744B | roughly 40B | Native Linux, 32 GB RAM, PCIe 4 NVMe | roughly 370 GB int4 | roughly 0.5 to 1 tok/s | project estimate |
DS4 labels the 26.68 tok/s figure as a single-run Metal CLI measurement on a 128 GB M3 Max MacBook Pro using q2, a short prompt, a 32,768-token context setting, non-thinking mode, greedy decoding, and 256 generated tokens. It is a project-author benchmark. It should travel with all of those conditions.
Colibri’s project-author baseline used WSL2, 12 cores, 25 GB RAM, and NVMe through VHDX; the project reports roughly 20 GB peak RSS and roughly 0.05 to 0.1 tok/s for cold decode. The same documentation puts native Linux with 32 GB RAM and PCIe 4 NVMe at roughly 0.5 to 1 tok/s, then explicitly calls that row an estimate rather than a measurement. The estimate belongs in capacity planning as a hypothesis to test, never as a proven baseline.
A routing tier has a narrow job
These results change the routing menu. A policy can reserve a local tier for data that must stay on-device, then keep larger or higher-throughput workloads in the data center. Local becomes an enforceable placement choice, not a claim that one machine should absorb the fleet.
Quality claims need the same discipline. DS4’s author writes that V4 Flash “feels quasi-frontier”. That is the author’s subjective impression. It is neither a benchmark result nor evidence that the model matches a closed frontier model.
The operational limits are substantial. Colibri’s proven baseline is slow because cold inference streams expert weights from storage, and both approaches demand large model footprints and meaningful RAM. DS4 currently describes its engine and model files as beta quality and its agent as alpha quality; Colibri describes a roughly 370 GB int4 model with roughly 20 GB peak RSS on its baseline. Neither project documentation resolves the training-data provenance of the underlying weights. Data-center inference keeps the larger and higher-throughput work.
DeepSeek, Z.ai, DS4, and Colibri are neither Muniment customers nor endorsers. Their public records establish something useful and smaller: governed routing now has a local floor worth testing.