CaReCoS benchmark stumps AI models with medical sound spectrograms

A team of researchers from academia and industry has released CaReCoS (Cardiac, Respiratory, and Cough Sounds), a novel benchmark that challenges AI models to interpret spectrogram representations of medical acoustic signals. The benchmark draws from seven publicly available medical audio datasets, converting sounds into mel-spectrogram images and pairing them with clinically grounded multiple-choice questions. The goal: test whether modern vision and omni models can combine visual pattern recognition with medical domain knowledge to diagnose conditions from heart murmurs, wheezes, and cough patterns.

Evaluating nine state-of-the-art models—including GPT-4o, Gemini, and specialized medical imaging models—researchers found uniformly poor performance. The top model achieved just 51.2% accuracy, barely above random chance for the task. None of the models demonstrated the ability to reliably detect fine-grained acoustic features encoded in spectrograms or to correlate those visual patterns with clinical concepts. The results underscore a fundamental deficiency: today's AI systems cannot effectively bridge the gap between medical acoustics and visual reasoning, posing challenges for telehealth and automated diagnosis applications that rely on sound-based diagnostics. The authors call for new training datasets and model architectures specifically designed for medical sound visualizations.