Flawed tests overstate AI room acoustics accuracy by an order of magnitude
Standard validation protocols let models cheat using position fingerprints, not true learning.
A new paper on arXiv (2607.15243) by Akın Oktav systematically dismantles the reported high accuracy of machine learning models for predicting ISO 3382-1 room acoustic parameters. The study used measurements from a 264-seat conference hall and a 180-seat concert hall, evaluating three model families under different protocols. The key finding: when validation splits are row-based (random) and input features include measured-at-test quantities like the actual impulse response, models achieve a mean R² of 0.81—matching inflated claims in the literature.
However, when the protocol is changed to match real-world deployment—grouping splits by receiver position and restricting inputs to geometry and environmental state (no test-time measurements)—the same models drop to R² between 0.09 and 0.57. The study shows that a hybrid CNN exploiting the target's own impulse response as input simply memorizes a position fingerprint, not transferable acoustic information. Under honest evaluation, the advantage of learned models over a simple inverse-distance weighting baseline shrinks by an order of magnitude. The only genuine strength remains interpolation at already measured positions (band means 0.80–0.88), a distinct and operationally useful task.
- Reported R² > 0.85 for room acoustics prediction is largely an artifact of row-based validation splits and inclusion of measured-at-test input features like the impulse response.
- Under deployment-consistent evaluation (grouped by receiver, no test-time inputs), performance drops to R² 0.09–0.57, reordering which parameters appear 'easy' to predict.
- Hybrid CNN using the target's own impulse response acts as a position fingerprint, offering no genuine transferable learning; gains for reverberation time were zero.
Why It Matters
Calls for rigorous benchmarking across AI for acoustics—many published results may not generalize to unmeasured positions in real rooms.