Contextual inference from single objects in Vision-Language models

A team of researchers including Martina G. Vilas has published a new paper, 'Contextual inference from single objects in Vision-Language models,' providing a systematic look at how AI models understand scenes. By presenting popular Vision-Language Models (VLMs) with images of single objects on masked backgrounds, the study probed their ability to infer both fine-grained scene categories (like 'bedroom' or 'kitchen') and coarse superordinate context ('indoor' vs. 'outdoor'). The key finding is that single objects do carry enough information for VLMs to make above-chance predictions at both levels, and their performance is modulated by object properties that also predict human scene categorization.

However, the research uncovered a critical fragility: accurate inference at one level (e.g., identifying a scene) neither requires nor guarantees accuracy at another (e.g., judging it's indoors). This partial dissociation varies significantly across different models. Mechanistic analysis revealed that object representations which remain stable when background context is removed are more predictive of successful inference. Furthermore, the way context is encoded differs fundamentally; scene identity is distributed across image tokens throughout the network's layers, while superordinate 'indoor/outdoor' information emerges only in later stages or not at all. This complex, layered organization suggests that the robustness of VLMs is more nuanced than simple benchmark accuracy scores indicate, with important implications for their reliability in real-world applications.