When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

A team of researchers from Stanford University, including Lezhi Tan, Naomi Sagan, Lihua Lei, and Jose Blanchet, has published a groundbreaking paper titled 'When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions.' The work addresses a critical challenge in human-AI collaboration: how to optimally allocate limited human attention to correct AI-generated assessments that are often noisy or biased. The researchers propose a general decision-theoretic framework that treats AI outputs as factor-level signals and human judgments as costly information that can be selectively acquired.

In practical applications like AI-assisted peer review, their framework achieved remarkable efficiency. The system substantially outperformed LLM-only predictions and matched the performance of full human review while using only 20-30% of human information. The researchers found that simpler selection rules derived under linear models could significantly reduce computational costs without harming final prediction performance. The optimal rule in linear settings admits a closed-form expression with clear interpretations in terms of factor importance and residual variance, making it both interpretable and efficient.

The research demonstrates that principled human dispatching—strategically deciding when humans should intervene—can dramatically improve the efficiency of AI-assisted workflows. This has significant implications for fields like academic peer review, content moderation, medical diagnosis, and financial analysis where human expertise is valuable but scarce. The framework's ability to maintain high accuracy while drastically reducing human workload represents a major step toward practical, scalable human-AI collaboration systems.