New method beats background bias in CLIP/SigLIP with 90%+ accuracy
No real-world debiasing needed—synthetic data alone fixes VLM background errors.
Vision-language models (VLMs) like CLIP and SigLIP 2 are notorious for learning spurious correlations between foreground objects and their backgrounds—e.g., associating a bird with water or land. In a new preprint, Youssef Zaazou and Mark Thomas show that these biases can be systematically removed by leveraging the linear structure inherent in VLM embedding spaces. Their key insight: scene representations can be decomposed linearly into foreground and background components. Using this property, they design a pre-training stage with synthetic data that forces the model to focus on foreground semantics while ignoring irrelevant background textures.
The results are striking. On the Waterbirds benchmark—where the training set has 100% spurious correlation (every bird on water is a waterbird, every bird on land is a landbird)—the method achieves over 90% worst-group accuracy, the first to break that barrier. No real-world debiased data is required, and the approach transfers well from synthetic to real images. This means VLMs can be made far more robust to background changes without expensive data collection or manual annotation, a critical step for deploying reliable image classification in unpredictable environments.
- First method to exceed 90% worst-group accuracy on Waterbirds under 100% spurious correlation
- Uses linear additivity in VLM embeddings to decompose scenes into foreground and background
- Requires only synthetic data for pre-training, no real-world debiased samples needed
Why It Matters
Makes VLMs far more robust to background shifts, enabling safer deployment in real-world vision tasks.