New L1–L4 framework tackles context-dependent image meanings in retrieval

A single image of two people in the rain could evoke hope and warmth in a reunion story or sorrow and finality in a farewell narrative. Researchers Ayuto Tsutsumi and Ryosuke Kohita investigate this context-dependent nature of image meaning and its implications for retrieval systems. Their key insight: context dependency correlates with semantic abstraction. Concrete elements like objects and actions remain consistent across contexts, while abstract elements like atmosphere or intent shift with the surrounding narrative. They formalize this as the L1–L4 framework, where L1 represents context-independent semantics (e.g., 'a red car') and L4 represents maximally context-dependent meanings (e.g., 'a bittersweet farewell'). To evaluate, they built synthetic story contexts and queries, allowing controlled comparison of how narrative injection affects retrieval at each abstraction level.

Results show that concrete queries (L1) are retrievable without any context, but abstract levels (L3–L4) increasingly require narrative grounding. Where context is injected matters: enriching the image embeddings (image-side) proved more effective than enriching the query alone. However, even with full narrative context, L4 queries—which capture highly subjective interpretations—remain difficult to retrieve accurately. This highlights context-dependent image retrieval as a significant open problem. The L1–L4 framework and experimental findings, presented as a short paper at SIGIR 2026, provide a structured approach for building future retrieval systems that go beyond keyword matching to understand the nuanced meanings images acquire in stories. It lays groundwork for applications in digital libraries, media archives, and AI-assisted storytelling.