Stanford study predicts clinical LLM rejection risk with 0.71 AUROC

A new paper from Stanford researchers introduces a deployment-centered evaluation approach for clinical LLM systems. Instead of relying on static benchmarks that measure correctness in aggregate, the team trains a pre-response classifier to estimate the risk that a user will reject the LLM's output. The model leverages both query content and deployment-specific context—such as provider type, department, and the specific language model used—to predict rejection before the response is generated.

Over 4.5 months of prospective analysis using real user feedback from an academic medical center's electronic health records, the classifier achieved an AUROC of 0.719. The study's key insight is that adding deployment-specific metadata significantly improves prediction accuracy compared to using query content alone. The authors demonstrate two downstream applications: triggering guardrails (e.g., flagging high-risk queries for human review) and abstention (withholding responses when rejection is likely).