LFM2.5-2.6B runs on OnePlus 13 CPU at 17 tok/s via 450KB engine
A 450KB custom engine pushes a 2.69B model to 17 tok/s on a phone
Deep Dive
Reddit user trikboomie submitted a post about their own inference engine, built from scratch at just 450KB, running the Q4_K_M GGUF of a 2.69B-parameter model with a 128K context window designed for multi-step agent workflows. The TUI is their own device probe suite running through ADB, and the engine supports other architectures including Qwen, Gemma, and Bonsai. They're currently trying to push it to ~30 tok/s.
Key Points
- LFM2.5-2.6B (Q4_K_M GGUF) runs at 17 tok/s on OnePlus 13's CPU with a 128K context window
- Custom 450KB inference engine built from scratch also supports Qwen, Gemma, and Bonsai model families
- Target performance is ~30 tok/s; current demo uses ADB-connected probe suite for real-time monitoring
Why It Matters
Shows efficient on-device agent inference without cloud, cutting costs and improving privacy for mobile AI applications.