# Local inference is becoming a real routing tier [Journal](/journal/) July 15, 2026 · [evidence](/journal/#evidence) Large open-weight models can now run on local machines, which gives governed routing policies a credible on-device tier with hard operational limits. ![A routing switch sends the selected path to a compact local machine while a dotted alternate path leads to remote servers.](/_astro/the-local-inference-tipping-point.DK7UsW8y_24jy3J.webp) Local execution has become credible as the lowest tier in a governed routing policy. When an organization’s policy or contract requires work to remain on-device, a router can now send some jobs to very large open-weight models without pretending that every workload belongs there. The evidence is narrow. DeepSeek publishes V4 Flash as a [284-billion-parameter mixture-of-experts model](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) with 13 billion parameters active and MIT-licensed weights. Z.ai publishes [GLM-5.2 weights under the MIT license](https://huggingface.co/zai-org/GLM-5.2). Two project authors have built local runtimes around those weights. Their measurements show feasibility on personal machines, not equivalence to closed frontier systems and not independent evaluation. ## The floor now has evidence DS4 uses an [asymmetric 2-bit scheme](https://github.com/antirez/ds4) that quantizes only routed mixture-of-experts weights while leaving shared experts, projections, and routing untouched. Its current documentation says the [2-bit Flash quant is 81 GB](https://github.com/antirez/ds4). That footprint fits a high-memory laptop class without reducing the claim to a hardware shopping list. Colibri takes another route: it [keeps GLM-5.2’s dense portion resident](https://github.com/JustVugg/colibri) and streams routed experts from a roughly 370 GB int4 model on local storage. The consequence is a much lower demonstrated RAM floor and a much slower cold decode. Model and source Scale Active Machine Footprint Speed Status [DeepSeek V4 Flash via DS4](https://github.com/antirez/ds4) 284B 13B M3 Max MacBook Pro, 128 GB 81 GB q2 26.68 tok/s generation measured [GLM-5.2 via Colibri](https://github.com/JustVugg/colibri) 744B roughly 40B WSL2, 12 cores, 25 GB RAM, NVMe via VHDX roughly 370 GB int4; roughly 20 GB peak RSS roughly 0.05 to 0.1 tok/s cold measured [GLM-5.2 via Colibri](https://github.com/JustVugg/colibri) 744B roughly 40B Native Linux, 32 GB RAM, PCIe 4 NVMe roughly 370 GB int4 roughly 0.5 to 1 tok/s project estimate DS4 labels the 26.68 tok/s figure as a [single-run Metal CLI measurement](https://github.com/antirez/ds4) on a 128 GB M3 Max MacBook Pro using q2, a short prompt, a 32,768-token context setting, non-thinking mode, greedy decoding, and 256 generated tokens. It is a project-author benchmark. It should travel with all of those conditions. Colibri’s project-author baseline [used WSL2, 12 cores, 25 GB RAM, and NVMe through VHDX](https://github.com/JustVugg/colibri); the project reports roughly 20 GB peak RSS and roughly 0.05 to 0.1 tok/s for cold decode. The same documentation puts [native Linux with 32 GB RAM and PCIe 4 NVMe at roughly 0.5 to 1 tok/s](https://github.com/JustVugg/colibri), then explicitly calls that row an estimate rather than a measurement. The estimate belongs in capacity planning as a hypothesis to test, never as a proven baseline. ## A routing tier has a narrow job These results change the routing menu. A policy can reserve a local tier for data that must stay on-device, then keep larger or higher-throughput workloads in the data center. Local becomes an enforceable placement choice, not a claim that one machine should absorb the fleet. Quality claims need the same discipline. DS4’s author writes that [V4 Flash “feels quasi-frontier”](https://github.com/antirez/ds4). That is the author’s subjective impression. It is neither a benchmark result nor evidence that the model matches a closed frontier model. The operational limits are substantial. Colibri’s proven baseline is slow because [cold inference streams expert weights from storage](https://github.com/JustVugg/colibri), and both approaches demand large model footprints and meaningful RAM. DS4 currently describes [its engine and model files as beta quality](https://github.com/antirez/ds4) and its agent as alpha quality; Colibri describes a [roughly 370 GB int4 model](https://github.com/JustVugg/colibri) with roughly 20 GB peak RSS on its baseline. Neither project documentation resolves the training-data provenance of the underlying weights. Data-center inference keeps the larger and higher-throughput work. DeepSeek, Z.ai, DS4, and Colibri are neither Muniment customers nor endorsers. Their public records establish something useful and smaller: governed routing now has a local floor worth testing. ## Sources 1. [DeepSeek on Hugging Face — DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) huggingface.co 2. [antirez on GitHub — ds4: DeepSeek 4 Flash and PRO local inference engine](https://github.com/antirez/ds4) github.com 3. [JustVugg on GitHub — colibri: Run GLM-5.2 on a 25GB-RAM consumer machine](https://github.com/JustVugg/colibri) github.com 4. [Z.ai on Hugging Face — GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) huggingface.co ## Continue reading [All publications](/journal/) evidence · Jul 14, 2026 ### [Measure discovery and verification separately](/journal/measure-discovery-and-verification-separately/)