TACO: New AI training method optimizes tool use without external judge
Self-supervised credit assignment eliminates wasteful tool calls, boosting accuracy by...
TACO (Tool-Augmented Credit Optimization) addresses a critical flaw in current agentic multimodal models: code tool calls can be useful, redundant, or misleading, but outcome-only rewards can't distinguish them. Existing process rewards either fail to attribute final correctness to individual calls or require an external judge model. TACO solves this with a two-channel advantage system within a GRPO reinforcement learning framework.
First, DAPR (Differential Answer-Probe Reward) inserts probe tokens into the model's reasoning to elicit predictions with and without a specific tool call. The difference in outcome reward becomes that call's credit—positive if helpful, negative if misleading, zero if neutral. This reuses the existing answer checker without any auxiliary model and is robust to probe-hacking. Second, OGAR (Outcome-Gated Advantage Routing) distributes the final answer's outcome advantage only to responsible segments, suppressing wasted tool calls without additional cost terms. TACO is trained via a two-stage SFT+RL pipeline and tested across perception, reasoning, and general multimodal benchmarks, yielding consistent accuracy gains and more efficient tool invocation.
- DAPR self-assigns tool call credit by comparing probe responses with and without the tool, requiring no external judge.
- OGAR distributes final outcome reward only to responsible segments, reducing useless tool calls without extra penalties.
- Trained via SFT+RL pipeline, TACO shows consistent accuracy gains across perception, reasoning, and multimodal benchmarks.
Why It Matters
Enables more efficient and trustworthy AI agents that learn when to use tools, reducing costs and errors.