Google's Gemini models prove reliable as audio judges for voice agents at 100x lower cost

A new arXiv paper from Google researchers evaluates the reliability of Gemini models as LALM (Large Audio Language Model) audio judges for scoring full-duplex voice agent conversations directly from raw stereo waveforms. The study tested three models—Gemini 2.5 Flash, 3.5 Flash, and 3.1 Pro—on 209 stereo sessions scored by three calibrated human raters across 8 production dimensions. The dataset included 152 full-duplex conversations spanning 13 accent/condition strata and 57 adversarial defect-injected clips.

Results show that Gemini 2.5 Flash closely approximates human ratings: on 5 of 8 dimensions, the LALM-human Spearman correlation differs from human-human correlation by at most 0.07, with overlapping 95% bootstrap confidence intervals on 7 of 8 dimensions. The LALM agreed within 1 point of the human mean on 60-92% of sessions for 6 dimensions. Additionally, on 45 of 48 defect/dimension cells, the LALM was as sensitive as humans or better. Gemini 3.5 Flash improved simple agreement to all 8 dimensions, while 3.1 Pro rated several dimensions markedly lower despite comparable rank correlation. The paper emphasizes that model swaps require re-validation on calibration, not just rank-correlation. Crucially, the study estimates that human rating alone costs roughly two orders of magnitude (100x) more than the equivalent LALM workload, providing a strong economic incentive for adoption while noting four areas where deployment requires care.