New Open-Source Trick Makes AI Think Harder Only When Needed
This could cut AI computing costs and speed up responses on your devices.
A developer shared an experiment using JEV logic to make LLM thinking adaptive. The method measures entropy for every single word, and when the model is unsure it uses a JEV-like gate to ask the model whether it should trigger a thinking process.
An opening thought happens anyway, followed by additional thinking blocks based on that logic. For the thinking blocks, there is a tree of parallel conversations where JEV logic is again used to define the best line.
The developer hopes it can inspire people, especially those running local models with the ability to run multiple streams, which are anyway bottlenecked by RAM speed for decode. The code is open source on GitHub.
- The AI only does extra 'thinking' when it's unsure, saving computing power.
- This could make local AI models faster and cheaper to run on personal devices.
- The method is open source, so developers can build on it, but it's not yet user-friendly.
Why It Matters
More efficient AI means lower costs, faster responses, and less energy use for everyone.