New 'catch-n-repair' method slashes LLM podcast hallucinations by 40%

Large language models (LLMs) are increasingly used to turn documents into long-form conversational podcasts, but they often introduce made-up details. In the first systematic study of faithfulness in multi-speaker podcast generation, Soumya Dutta and colleagues from IBM and partner institutions created a dataset of over 1,500 documents spanning five domains (news, academic papers, technical docs, etc.) and generated podcasts using models including GPT-4o, Claude, and Llama. They developed an LLM-as-a-judge framework that evaluates each conversational turn against the source document, validated by human studies. Their analysis shows that even state-of-the-art models hallucinate frequently — GPT-4o showed 22% of turns containing unsupported claims.

To combat this, the team introduces 'catch-n-repair,' a model-agnostic pipeline that first detects unfaithful turns using a combination of natural language inference and structural pattern matching, then rewrites them while preserving conversational flow and narrative coherence. Experiments show consistent fidelity gains: the repair reduces hallucination rates by 35–50% across all tested models, both on in-domain data and on entirely new domains. The method is computationally lightweight and does not require retraining the underlying LLM. This work is under submission and provides a practical toolkit for anyone building AI-generated podcast or long-form audio content from text sources.