New paper 'Eta Given Delta' defines LLM tool efficiency metric
Directly measures how often agent tool calls actually add value.
Current LLM agent evaluation relies heavily on accuracy as a proxy for efficiency, but that misses how lean and cost-effective a tool suite really is. In a new arXiv paper, Nyx Iskandar tackles this gap head-on by introducing 'tool efficiency' – a quantitative metric that measures what fraction of tool calls actually contribute to the final answer. To make the metric well-defined, the paper also defines 'marginal tool utility' per call, which determines whether a tool is truly useful or can be safely removed without harming accuracy. The sign of marginal utility is determined using LLM-as-a-Judge, leveraging the same models being evaluated to score their own tool usage.
This framework is designed for post-hoc trajectory analysis, not real-time adjustments. The key insight: instead of optimizing accuracy only, future agent harnesses can now optimize for complementary metrics like tool efficiency. This could lead to cheaper, faster agents that use fewer API calls and simpler toolkits. The paper explicitly positions itself as a springboard for new benchmarks and agent engineering, particularly for building lean tool suites. It's a shift from 'does the agent get the right answer?' to 'does it get there efficiently?'
- Introduces 'tool efficiency' as the rate of useful tool calls over total calls in an LLM agent trajectory.
- Defines 'marginal tool utility' per call, determined by LLM-as-a-Judge to flag redundant tools.
- Enables post-hoc identification of unnecessary tool suite components without compromising accuracy.
Why It Matters
Paves the way for cheaper, faster LLM agents by quantifying and cutting unnecessary tool calls.