EAGLE3 speculative decoding merges into llama.cpp for faster inference
New algorithm speeds up local LLM inference with guided multi-token prediction
Deep Dive
After half a year of development, EAGLE3 has been merged into llama.cpp. Like MTP but with a twist: the helper model gets extra guidance from the main model instead of guessing completely on its own.
Key Points
- EAGLE3 has been merged into llama.cpp after six months of development
- It improves upon multi-token prediction by giving the draft model guidance from the main model's hidden states
- Expected to deliver 2–4x speedups for local LLM inference on consumer hardware
Why It Matters
Faster local inference without quality loss enables larger models on everyday hardware, crucial for developers and AI hobbyists.