Study: LLMs match doctors on preventive care but underrate lifestyle advice

A new arXiv study led by Eden Avnat and colleagues evaluated whether large language models can replicate and even augment physician prioritization of preventive care services (PCS) under time constraints. Using a validated survey from Zhang et al. (JAMA Network Open 2020), the researchers simulated 137 physician personas matching cohort demographics and tested seven LLMs with three prompting strategies each. The primary outcome measured concordance with physician rankings via Spearman correlation, plus a novel Consensus-Stratified Agreement (CSA) metric that captured how well LLM selections aligned with physician consensus at different agreement levels.

Results showed LLMs mirrored physicians closely on average (mean Spearman = 0.83, SD = 0.11), with very high CSA at extreme agreement ranges (94%, 197/210) but strikingly low CSA in moderate ranges (21%, 30/140). The biggest divergence: LLMs consistently underprioritized lifestyle modifications—rating only 8.8% of these services as high-priority versus 38% for physicians (P < .001). Several LLMs actually outperformed physicians on life-years gained per prioritized choice and consistency, while being more selective. Time constraints affected both groups similarly, boosting selectiveness but reducing consistency. Augmentation—having LLMs revise physician rankings—varied by model, suggesting current systems need value-aligned training and explicit time-constraint modeling before they can reliably improve preventive care decisions.