ToolChain-CRC catches AI agent failures mid-trajectory, not just final answers

Modern AI agents rely on retrieving documents, calling tools, and checking intermediate information before producing a final answer. This creates a risk-control blind spot: a final response may look acceptable even when retrieval was weak, a tool output was wrong, or an earlier step was unsupported. ToolChain-CRC, developed by Jeffery Opoku and David Banahene, addresses this by treating each agent run as a full trajectory of actions, observations, and final outputs. The method builds step-level risk scores, combines them into a trajectory risk score, and calibrates an accept-or-intervene rule. It also includes an anytime alarm that can stop risky runs before the final answer is produced, using a supermartingale construction for theoretical guarantees.

The authors prove trajectory-level risk control under exchangeable calibration runs and provide a drift-aware extension with auditable constants. Extensive experiments cover synthetic tool-chain drift, RAG/tool-use stress tests, public SQuAD-derived retrieval tasks, an API-free agentic QA case study, and a live RAG/tool-use agent benchmark. Across all settings, final-answer-only calibration missed retrieval and tool failures, while trajectory-level calibration kept accepted-trajectory risk below the target. The work has significant implications for building more reliable AI agents in production environments where tool and retrieval pipelines are prone to drift.