arXiv study: Steps-to-failure better than loss curves for measuring adversarial robustness

Projected Gradient Descent (PGD) is the standard tool for evaluating adversarial robustness, but researchers typically only look at final accuracy under attack. Durgule's new paper, 'Geometry Is Not Robustness: A Trajectory-Level Study of PGD Evaluation,' digs deeper by recording the full 20-step PGD attack trajectory on 3000 clean-correct samples per model, using random initialization and multiple restarts. The study compares clean-trained and adversarially-trained convolutional neural networks on Fashion-MNIST, analyzing loss evolution, gradient alignment, and steps-to-failure across four robustness regimes.

The results show a clear robustness hierarchy across models — but not from every trajectory metric. Mean loss trajectories and gradient alignment patterns were quantitatively similar across adversarially-trained models that had substantially different robust accuracies. In contrast, steps-to-failure distributions provided a much sharper separation, directly reflecting the model's functional resistance to perturbation. This suggests trajectory-level diagnostics describe the optimization geometry of the attack, not the model's intrinsic robustness. Durgule cautions that interpreting these metrics requires context around the robustness regime, attack strength, and multi-metric evaluation. The paper concludes that trajectory-level analysis should complement — not replace — standard robustness measurements.