[D] Why evaluating only final outputs is misleading for local LLM agents

A viral analysis by developer Kareem Rashed is challenging how the AI community evaluates local LLM agents. Using setups like Ollama and LangChain, Rashed observed that agents can produce a perfectly correct final answer while their internal process is a mess—calling the wrong tools first, performing unnecessary steps, getting stuck in loops, or nearly triggering forbidden actions. This means standard evaluation methods, which focus solely on the output, are dangerously misleading. Two agents can deliver the same correct summary, but one might do it in two clean steps while the other wastes resources on redundant searches and retries, hiding inefficiency and potential risk.

Rashed argues that for agents—AI systems that take actions—the process trace is where the real signal lies. To address this, he built 'rubric-eval,' an open-source tool designed for local evaluation. It uses another local LLM (via Ollama) as a judge to analyze the agent's execution trace, checking for correct versus forbidden tool usage, penalizing unnecessary loops and extra steps, and assessing the coherence of the reasoning path itself. This shift from output-based to process-based evaluation is crucial for deploying reliable, efficient, and safe autonomous AI systems, especially in private, local environments where data cannot be sent to external APIs for assessment.