Cactus Needle2: 14MB agentic LLM hits 800+ tok/s on edge devices

Cactus unveiled Needle2, a new agentic LLM specifically designed for low-cost edge hardware—phones under $200, Raspberry Pis, microcontrollers, wearables, and smart home devices. The model packs 45 million parameters into a 14MB binary using CQ2-bit compression, a lossless 2-bit quantization trained end-to-end from pretraining to deployment. This avoids the quality degradation typical of post-hoc quantization. On a Raspberry Pi 5, Needle2 achieves 800+ tokens/second prefill and 500+ tokens/second decode, with only 28MB session RAM. It uses a bespoke Simple Attention Network architecture co-designed with a dependency-free C++ inference engine that auto-probes CPU capabilities, running from Cortex-M to x86 to WebAssembly.

The key innovation is reframing device control as function mapping rather than open-ended chat. Needle2 translates messy user sentences into typed function calls with a byte-level grammar compiled from declared schemas, ensuring structural correctness. It also outputs a learned confidence score, enabling edge-cloud collaboration: routine actions execute locally and privately, while off-topic queries return an empty call, triggering escalation to the cloud when needed. Pebble, the wearable pioneer, already runs Needle2 locally in its Index 01 app, turning spoken requests into reliable actions without network dependency. Developers can fine-tune the 45M model on a Mac or PC in minutes to hours, customizing it to speak a device's specific tools.