Adoption and Use of LLMs at an Academic Medical Center

Stanford Medicine researchers developed ChatEHR, an internal LLM platform that integrates directly with the electronic health record (EHR) system. Unlike standalone LLM tools that suffer from manual data entry friction, ChatEHR accesses the entire patient timeline spanning several years. It supports two modes: automations (static combinations of prompts and data for fixed tasks like pre-visit chart review or surgical site infection monitoring) and interactive use via a UI where clinicians can sift through records. The system is model-agnostic, allowing the team to match the best LLM (e.g., GPT-4, Claude, or open-source models) to each specific clinical or administrative task. This 'build-from-within' strategy gives the medical center full governance and vendor independence.

Adoption was rapid: over 1.5 years, 7 automations were built and 1,075 staff trained to use the UI. In the first 3 months after launch, users engaged in 23,000 sessions. The most frequent task was generating clinical summaries, with an observed 0.73 hallucinations and 1.60 inaccuracies per generation—highlighting the need for new monitoring methods beyond standard benchmarks. Financially, initial estimates peg savings at $6M in the first year, from cost and time savings plus revenue growth, without even quantifying improved care quality. The paper underscores that benchmark-based evaluations were insufficient for the UI's performance monitoring, requiring novel approaches to track real-world accuracy and safety.