Rate-Distortion Signatures of Generalization and Information Trade-offs

A team of researchers has published a novel framework that uses principles from information theory to fundamentally rethink how we measure AI generalization. In their paper 'Rate-Distortion Signatures of Generalization and Information Trade-offs,' Leyla Roksan Caglar, Pedro A.M. Mediano, and Baihan Lin propose treating a model's stimulus-response behavior as an effective communication channel. By analyzing confusion matrices, they derive a 'rate-distortion frontier' and summarize each system with two interpretable geometric signatures: slope (β), which captures the marginal cost of trading accuracy for robustness, and curvature (κ), which indicates the abruptness of that trade-off. This provides a more nuanced view than standard robustness metrics.

Applying this framework to human psychophysics data and 18 diverse deep vision models—including various architectures and training regimes—under controlled image perturbations yielded critical insights. The study found that while both biological and artificial systems follow a common lossy-compression principle, they occupy systematically different regions of the rate-distortion space. Humans exhibit smoother, more flexible generalization geometry. In contrast, modern deep networks like those tested operate in steeper and more 'brittle' regimes, meaning their performance degrades more sharply under perturbation, even when their baseline accuracy matches human levels. Furthermore, robustness training techniques induce specific but dissociable shifts in these β/κ signatures, revealing cases where improved metrics don't lead to more human-like generalization behavior. This compact, model-agnostic lens could guide the development of more robust and flexible AI systems.