LLM framework with human-in-the-loop boosts skin toxicity detection by 14%

A team of researchers has introduced a novel human-in-the-loop framework leveraging retrieval-augmented generation (RAG) and multiple LLM agents to identify cutaneous immune-related adverse events (cirAEs) from unstructured clinical notes. These skin toxicities are common side effects of immunotherapy, and accurate identification is critical for patient management. The framework combines an LLM-based extraction pipeline with human reviewers who validate and correct outputs, creating a feedback loop that improves over time. By using RAG, the system retrieves relevant clinical context to enhance accuracy, while the multi-agent design allows different models to specialize in tasks like note parsing or toxicity classification.

The study, published on arXiv (2607.20428), compared the LLM-assisted workflow against traditional manual chart review. Results showed a significant jump in accuracy, with an F1 score of 0.88 versus 0.77 for manual review. Inter-rater agreement, measured by Cohen's kappa, improved dramatically from 0.50 (moderate) to 0.82 (almost perfect). Perhaps most impactful for clinical workflows, the average review time was cut by roughly 50%. The authors position this as a scalable, transparent approach not just for cirAEs but for any immune-related toxicity detection across organ systems, potentially transforming how adverse events are monitored in oncology.