ClinLens benchmark exposes AI's 56% accuracy gap in clinical data science
Top coding agents run 100% but only 56% get clinical answers right.
ClinLens is a new benchmark designed to evaluate AI agents on long-horizon clinical data-science tasks. Unlike existing benchmarks that focus on isolated question answering or structured-table reasoning, ClinLens requires agents to transform heterogeneous longitudinal patient records—spanning structured EHR data, clinical notes, electrocardiograms, chest radiographs, and echocardiograms—into auditable, correct analyses. The benchmark includes 200 executable tasks drawn from five linked MIMIC resources, organized by a 4×5 taxonomy that crosses four patient-time scopes (e.g., single visit, multi-visit) with five analysis capabilities (e.g., cohort selection, temporal reasoning, multimodal integration).
In testing 24 standardized model-scaffold configurations on a fixed 126-task subset, the best-performing agent achieved only 56.3% scope-macro STRICTPASS (fully correct analysis) despite 100% EXECSUCCESS (code ran without errors). A separately tuned coding agent solved 83 of 126 tasks, while five biomedical systems adapted to GPT-4o-mini reached at most 2.9% STRICTPASS. The results highlight a critical gap between code that executes and analyses that are clinically sound—a challenge for deploying autonomous AI in healthcare.
- ClinLens includes 200 tasks over five MIMIC data types: EHR, notes, ECGs, chest X-rays, and echocardiograms.
- Best agent achieved 56.3% STRICTPASS accuracy despite 100% execution success, showing code runs but logic fails.
- GPT-4o-mini-based biomedical systems scored ≤2.9% on the benchmark, indicating current LLMs struggle with complex clinical workflows.
Why It Matters
ClinLens reveals autonomous clinical AI still lacks reliable reasoning, risking errors in patient data analysis.