New fusion method makes ViT ensembles 22% more robust to attacks
No domain knowledge needed: geometric rules match hand-crafted cues to 0.002 F1.
A new arXiv paper from Mario Leiva and colleagues at Arizona State University introduces an adversarially robust approach to fusing multiple pre-trained vision transformer (ViT) perception models. Their method, called Label Vector Pools (LVP), eliminates the need for hand-crafted domain knowledge by exploiting vector-space geometry. Each model's training embeddings form prototype-based error-detection rules that match traditional domain-knowledge rules to within 0.002 F1 on test sets. This keeps the system fully neurosymbolic, allowing optional integration of domain knowledge when available.
Tested on 15 weather-shifted aerial-imagery test sets with six ViT detectors, the approach frames fusion as a consistency-based abduction problem solved at test time via an exact Integer Program. Under a coordinated label-flipping attack with a 90% flip rate, LVP achieves 0.42 F1 versus 0.35 for the strongest majority-vote baseline (MV-Plurality)—a 22% relative improvement. Once the flip rate exceeds 0.4, LVP attains the highest F1 on every test set, showing robust performance where traditional combiners fail.
- LVP builds error-detection rules purely from each model's training embeddings, matching hand-authored domain rules within 0.002 F1.
- At a 90% label-flip attack rate, LVP averages 0.42 F1 vs 0.35 for MV-Plurality—a 22% relative gain.
- Achieves the highest F1 on every test set once flip rate exceeds 0.4, across 15 weather-shifted sets and six ViT detectors.
Why It Matters
For deployed AI perception in unpredictable environments, this enables robust ensemble fusion without costly domain expertise.