Gemma 4 agent architecture uses modular 'apps' to boost small model accuracy on local hardware

A developer experimenting with local AI agents identified a core frustration: small language models frequently mangle exact text like URLs or part numbers when given too many generic tools. Their solution was to replace a sprawling set of over 20 tools with just two dedicated 'applications' (workflows) that present the agent with scoped, menu-driven interfaces. For example, a web browsing app shows numbered links ("open 1, copy 2") instead of forcing the model to generate raw URLs. Each app maintains its own persistent context, so the agent can leave and return without resetting state. Only a brief reference to the app’s existence remains in main memory, with a tool to re-enter.

The system was tested on a single AMD RX6600XT using llama.cpp’s Vulkan backend, running Gemma 4 at two sizes: 26B (Q4_K_XL) and a smaller E4B variant (also quantized). Despite the larger model’s potential, the E4B performed better under this workflow—likely because the structured apps reduced the cognitive load of tool selection. Performance reached 70-85 tokens/s with multi-token prediction and 800 t/s prefill at a 10k token context. The design is open-ended: the agent can still access ~100 general tools when outside the apps, but for complex multi-step tasks (like finding rare car parts), the application approach dramatically improved reliability. The code is shared as a proof of concept for others building local agent frameworks.