Study diagnoses HOI detector failures, boosting F1 from 48.6 to 90.2

Human-object interaction (HOI) recognition is crucial for automatically analyzing student behavior in educational settings, but state-of-the-art detectors often degrade in real-world deployment due to domain-specific objects, occlusions, and complex visual conditions. Researchers from Vanderbilt University address this gap by introducing a diagnosis-driven framework that combines a triplet-level HOI error taxonomy with error-factor attribution analysis. They apply their method to the Critical Care Air Transport Team (CCATT) mixed-reality medical training dataset, where accurate HOI detection is essential for assessing trainee performance.

The framework first categorizes HOI errors into three types—missing detection, false interaction, and misclassification—then attributes each error to factors like occlusion, small object size, or ambiguous context. Based on the diagnosis, they perform targeted refinement of a pretrained CDN (Cascaded Detection Network) model, such as fine-tuning on specific error-prone examples or adjusting detection thresholds. The results are dramatic: macro-F1 score jumps from 48.6 to 90.2, a 41.6 point improvement. This underscores the value of systematic error diagnosis for adapting HOI models to specialized training domains, enabling more reliable automated analysis of student interactions in complex, real-world educational environments.