Mobile inference needs phase-aware scheduling
A mobile inference study finds that prefill and decoding favor different hardware, making the execution phase part of the routing decision.
Efficient mobile model execution needs a phase-aware plan: prefill and decoding do not have the same hardware fit. In the authors’ version 1 measurement study, NPUs were strongest for compute-bound prefilling, while CPUs led the memory-bound decoding phase. For an operator, backend selection is therefore not one decision for an entire request. It is a scheduling decision that can change as the workload changes.
The measured split
The study evaluates five frameworks, llama.cpp, MLC-LLM, MLLM, MNN, and GENIE, across three backend classes: CPU, GPU, and NPU. The authors attribute the phase split to workload shape. Prefill supplies the large, fixed-shape work that suits an NPU. Decoding repeatedly presents smaller, dynamic kernels and memory-bound work that favored CPUs in their measurements.
| Request phase | Workload character in the study | Measured hardware fit | Operator implication |
|---|---|---|---|
| Prefill | Compute-bound, large and fixed shape | NPU strength | Evaluate an NPU route for this phase |
| Decoding | Memory-bound, small and dynamic kernels | CPU strength | Evaluate a CPU route for this phase |
This comparison records the paper’s phase-level finding, not a universal benchmark ranking. The paper reports framework-induced gaps of up to 10x on NPUs, which makes the runtime and its offloading, operator, and quantization choices part of the evidence. A backend label alone does not describe the execution path.
Measurements identify scheduling headroom
The authors measure up to 40% energy waste from suboptimal thread configuration, uncoordinated NPU sleep latency, and CPU polling intervals. That is a measured finding from their setup. Their response is proposed guidance: an energy-oriented configuration and scheduling approach that coordinates those controls rather than leaving each phase on an unexamined default.
Across three datasets, the authors estimate that their best-practice configuration could reduce energy consumption by up to 54.8% on the NPU backend. The 54.8% result is an estimate, not an independently replicated deployment result. It supports testing phase-aware configuration against a known baseline; it does not support assuming the same reduction on another device or workload.
The paper does not establish frontier-scale local inference or universal performance across devices and workloads. It is a version 1 preprint reporting the authors’ study results, not independent replication. Its useful operational claim is narrower: measure prefill and decoding separately, preserve framework and configuration in the record, and choose a route with the phase in view.
The paper’s authors, llama.cpp, MLC-LLM, MLLM, MNN, GENIE, Alibaba, and Qualcomm Technologies are neither Muniment customers nor endorsers.