Healthcare LLM benchmarks fail due to hidden user behavior assumptions

A new paper from researchers including Naveen Raman, Santiago Cortes-Gomez, and colleagues argues that healthcare LLM benchmarks are not inherently flawed – the real problem lies in unspoken assumptions about how users interact with models. The authors classify these assumptions into two categories: task assumptions (can be tested from conversation data alone) and outcome assumptions (require outcome data and behavioral studies). Crucially, outcome assumptions hinge on human behavior that even well-designed benchmarks cannot observe directly. Citing a retrospective analysis of a healthcare randomized controlled trial (RCT), the team found the evaluation–deployment gap splits roughly equally between task and outcome gaps.

To close this gap, the paper proposes two concrete tools. First, BenchmarkCards – structured artifacts that explicitly document every assumption underlying a benchmark's design. Second, staged evaluation – a procedure that systematically tests assumptions in order, starting with task assumptions and then moving to outcome assumptions using behavioral studies. The authors argue this framework makes the gap transparent and actionable, rather than attributing it to vague benchmark inadequacy. For professionals deploying LLMs in clinical settings, this work underscores that benchmark scores alone are insufficient; understanding and testing human-AI interaction patterns is equally critical for safe deployment.