Ishikawa and Duke audit depression detection benchmarks

In their paper 'A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks,' Takehiro Ishikawa and Jon Duke investigate the effectiveness of current benchmarks for clinical-interview depression detection. They utilize a lightweight hybrid text-plus-LLM-score model, achieving a macro-F1 score of 0.723, which is the highest reported under a stringent leave-one-subject-out cross-validation protocol. This score serves as a conservative reference point, emphasizing the need for robust evaluation methods in mental health AI applications.

The authors also explore the alignment between development-side cross-validation and official-test rankings, finding a significant disconnect. For instance, the best cross-validation configuration ranks twentieth in the official test, while the official-test winner ranks forty-first by cross-validation. This disparity raises concerns about the reliability of current benchmarks and their real-world applicability. Additionally, the study stresses the importance of symptom density in interviews, revealing that text models perform better on symptom-dense slices, while audio models show minimal improvement. This research aims to enhance the accuracy and reliability of depression detection tools, ultimately benefiting clinical practices.