New framework lets AI agents decide when to call for human backup

Traditionally, decision support studied how humans use ML models to make better decisions. In modern agentic systems, the roles are reversed: AI agents act on behalf of users, while humans and tools become support mechanisms. This shift brings reliability concerns to the forefront, since agentic errors can be consequential. A new paper from Shayan Kiyani, Sima Noorani, George Pappas, and Hamed Hassani (University of Pennsylvania) reimagines decision support for this reversed world. They propose an optimization problem that minimizes support usage subject to controlling a 'counterfactual missed-support error' — the probability the agent acts alone on instances where support would have materially improved its output.

At the population level, the optimal policy is a threshold rule on the value of support. The team builds an online algorithm that adaptively thresholds such a score and uses randomized exploration to control the missed-support error without any distributional assumptions. They also introduce a calibration-on-the-fly method that further reduces unnecessary support calls. The framework is instantiated across information gathering, human-AI collaboration, and tool use. Experiments show it reliably controls the target error while substantially reducing support usage in practice, offering a principled way to make autonomous AI agents safer and more efficient.