Yunjin Tong's oversight game reveals 'avoidable harm' when AI knows better

A new paper from Yunjin Tong tackles a critical challenge in AI oversight: how should humans supervise an AI when both sides have private information? Building on Cooperative Inverse Reinforcement Learning (CIRL) and the Oversight Game, the author models a contextual-bandit team game where the human knows her own reward function, while the AI knows the true quality of its proposed action. This mirrors real-world scenarios like an autonomous robot inspecting a situation a human cannot directly assess. The bandit structure removes physical state transitions, allowing exact one-shot characterizations that would remain conjectural in full POMDP models.

Tong's key insight is a gap between two benchmarks: a team optimum and a behaviorally natural myopic rule. That gap represents 'avoidable harm' — a region where the AI privately knows its action is harmful and shutdown would help, yet a myopic human, trusting her prior, declines to oversee. The author shows this gap is the price of non-credible oversight communication and analyzes how it resolves dynamically over repeated rounds through passive learning and active signaling with a one-period-lagged oversight response. The work has direct implications for designing safer human-AI interfaces in autonomous systems.