LLM workflow prioritizes 21K health insurance pages for human review

A new arXiv paper from Martin Möller describes an LLM-assisted workflow for triaging content review on German statutory health insurance (SHI) websites. The system analyzed 56,198 pages across 84 SHI sites, combining deterministic screening, model-assisted triage, in-depth review, minimum evidence checks, and paired-model comparison. The workflow generated 35,998 review records and sent 21,452 to case review, with workload concentrated on transparency, legal framing, medical content, contradictions, and AI-related failure-mode signals. Notably, 31,347 quoted passages were locatable in the original page text, confirming literal occurrence rather than factual correctness.

Möller frames the work as a prioritization tool, not a validated AI detector. A stress test on 300 lower-priority pages surfaced a signal on 33.3% of pages, while 182 matched cases showed two models agreeing 75.8% of the time (Cohen's kappa = 0.532, 95% CI 0.415–0.649). The author warns that paired-model agreement measures consistency, not correctness, and that any public claims require human adjudication. The production code is proprietary, but reproducibility relies on frozen derived tables and paired-comparison artifacts—making this a practical audit framework rather than a generalizable benchmark.