PoVisLE: New Polish VQA benchmark exposes cultural blind spots in VLMs
1,117 Polish images and 2,366 manual VQA pairs test far beyond surface recognition
Vision-language models (VLMs) excel at captioning and visual question answering, but they are predominantly trained on English-centric data. This leads to failures when interpreting culturally specific symbols, regional contexts, and pragmatic language use. Most existing benchmarks for cultural competence are template-driven and only measure surface-level recognition, leaving deeper linguistic and visual understanding untested.
To address this gap, Anna Kołos and colleagues present PoVisLE, a new monocultural benchmark for Polish. It comprises 1,117 carefully selected images and 2,366 manually annotated VQA pairs, designed to evaluate how language interacts with visual context in culturally situated settings. The authors propose a grounded evaluation paradigm that requires models to reason about Polish cultural references, idioms, and context-dependent visual cues rather than simply matching objects. PoVisLE is offered as a controlled, challenging resource for advancing culturally aware multimodal AI.
- PoVisLE contains 1,117 Polish images and 2,366 manually annotated VQA pairs for culturally grounded evaluation.
- Benchmark targets shallow existing tests by requiring pragmatic, context-dependent understanding beyond surface recognition.
- Built to expose failures of English-centric VLMs on Polish symbolic content and region-specific visual cues.
Why It Matters
PoVisLE pushes VLM evaluation beyond English-centric surface tasks, enabling more culturally robust and locally relevant AI systems.