The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare
New research finds 100% of healthcare AI systems require human oversight despite marketing claims of autonomy.
A Stanford research team led by Gabriela Aránguiz Dias and Kiana Jafari has published a groundbreaking study exposing the significant gap between marketing hype and operational reality for agentic AI in healthcare. The paper, "The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare," reveals that despite commercial claims of autonomous action, current systems operate under near-total human oversight due to safety, regulatory, and liability constraints.
Based on interviews with 20 stakeholders including developers, implementers, and end users, the researchers identified three mutually reinforcing tensions. First, conceptual fragmentation creates confusion about what 'agentic' actually means in practice. Second, an autonomy contradiction emerges where commercial promises of independent action exceed operational reality. Third, evaluation blind spots prioritize technical benchmarks over sociotechnical safety considerations, creating potential risks for patient outcomes.
The study argues that agentic AI functions as a 'site of contested meaning-making' where technical aspirations, commercial incentives, and clinical constraints intersect. This has material consequences for patient safety and the distribution of blame when systems fail. The research suggests that until accountability structures evolve to handle autonomous clinical reasoning, true agentic AI in high-stakes healthcare environments remains largely theoretical rather than practical reality.
This work provides crucial context for understanding why healthcare AI adoption has been slower than predicted, highlighting the structural barriers that go beyond technical capabilities. It serves as an important reality check for investors, developers, and healthcare administrators navigating the complex landscape of AI implementation in clinical settings.
- Study of 20 stakeholders reveals 100% of healthcare AI systems require human oversight despite autonomous marketing claims
- Identifies three key tensions: fragmented definitions, autonomy contradictions, and evaluation blind spots prioritizing technical metrics over safety
- Argues agentic AI serves as contested space where commercial incentives clash with clinical safety requirements
Why It Matters
Provides crucial reality check for AI healthcare investments and highlights safety barriers preventing true autonomous systems.