Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction
Locally deployable models extract entities from dental notes without sending data to the cloud.
Clinical named entity recognition is notoriously difficult for dental progress notes, which are unstructured, domain-specific, and privacy-sensitive. A new research paper from Chuang et al. (arXiv:2605.04221) proposes a locally deployable framework that lets small language models (SLMs) self-generate, verify, and refine entity-specific prompts. Using 1,200 annotated dental notes, the team evaluated multiple open-weight models with multi-prompt ensemble inference, then fine-tuned them via QLoRA-based supervised fine-tuning and direct preference optimization (DPO).
Results varied significantly by model, underscoring the need for task-specific benchmarks. Qwen2.5-14B-Instruct led with micro/macro F1 scores of 0.864 and 0.837 after DPO, while Llama-3.1-8B-Instruct reached 0.806/0.797. Notably, these models run entirely on local hardware, eliminating the privacy risks of cloud-based APIs. The automated prompt optimization combined with lightweight post-training provides a scalable path for clinical information extraction without compromising patient data.
- Qwen2.5-14B-Instruct achieved 86.4% micro F1 and 83.7% macro F1 after Direct Preference Optimization (DPO).
- Automated prompt generation includes self-verification and refinement cycles, reducing manual prompt engineering.
- All 1,200 dental notes were annotated and processed locally, ensuring full data privacy for sensitive health records.
Why It Matters
Enables healthcare organizations to extract structured data from unstructured notes without cloud reliance, preserving patient privacy.