Katanemo Labs' Signals finds best agent traces without LLM judges
82% informativeness vs 54% random – lightweight taxonomy for agent evaluation
Katanemo Labs, a DigitalOcean company, has released a research paper introducing Signals, a lightweight method for evaluating agent traces without relying on expensive LLM judges or human reviewers. The core insight is that most agent trajectories are not worth manual inspection – but finding the informative ones is costly. Signals computes structured indicators from live interactions, categorizing them into a simple taxonomy across interaction, execution, and environment patterns: misalignment, stagnation, disengagement, failure, looping, and exhaustion. No GPU or additional model calls required, and it leaves the agent's online behavior unchanged.
In an annotation study on τ-bench, Signals-based sampling achieved an 82% informativeness rate compared to 54% for random sampling, translating to a 1.52x efficiency gain per informative trajectory. The method is already implemented in the open-source project Plano on GitHub, making it immediately usable by developers building agentic systems. By enabling automated, cost-effective filtering of agent logs, Signals promises to accelerate debugging and iteration in production agent deployments.
- Signals taxonomy includes six patterns: misalignment, stagnation, disengagement, failure, looping, and exhaustion.
- Achieved 82% informativeness vs 54% random in τ-bench study, with 1.52x efficiency gain per informative trajectory.
- Implemented in open-source Plano (GitHub); requires no GPU or extra LLM calls.
Why It Matters
Makes evaluating agent traces cost-effective, enabling faster iteration on agentic systems without expensive LLM calls.