REALM benchmark unifies red-teaming for physical-world VLMs across 13 models

Vision-language models (VLMs) are increasingly powering perception-reasoning in physical systems like robots and autonomous vehicles, but their vulnerabilities remain poorly understood due to fragmented evaluation across datasets and metrics. REALM, proposed by researchers from the University of Central Florida and published on arXiv, is the first unified red-teaming benchmark designed specifically for physical-world VLMs. It standardizes 12 red-teaming methods (including text injection, typographic attacks, visual perturbations, and multimodal co-optimization), 3 model-agnostic defenses, and a suite of 13 VLMs — all under a practical black-box threat model with shared datasets and metrics. The benchmark introduces an agentic target-generation pipeline that constructs physically grounded, scene-specific attack objectives, enabling fair comparisons across diverse attack families.

The evaluation reveals several critical insights. Text and typographic injection attacks induce the most frequent failures across all models, while multimodal co-optimization (jointly perturbing text and images) yields the strongest transferability of visual perturbations. Surprisingly, single-pass attacks approach the effectiveness of iterative methods at much lower computational cost. Scale does not guarantee adversarial robustness: larger VLMs are not inherently more resistant to attacks. REALM provides a standardized framework for benchmarking physical-world VLM security, offering open-source code and datasets. The findings underscore that as VLMs deploy in safety-critical systems, systematic red-teaming is essential — and model size alone is not a defense.