Ukrainian Visual Word Sense Disambiguation Benchmark
A new benchmark reveals AI models struggle with Ukrainian word meanings in images, lagging far behind English performance.
A team of Ukrainian researchers has published a new benchmark for evaluating AI's ability to perform Visual Word Sense Disambiguation (Visual-WSD) in the Ukrainian language. The task challenges models to identify the correct visual representation of an ambiguous word, like 'bank' (financial institution vs. river bank), from a set of ten images with minimal context. The benchmark was constructed semi-automatically and refined by domain experts, following a methodology established for similar benchmarks in English, Italian, and Farsi. This allows for direct cross-language performance comparisons, putting Ukrainian on the map for rigorous multimodal AI evaluation.
The researchers then put eight leading multilingual and multimodal large language models (LLMs) to the test. The results were stark: all models underperformed compared to a simple, zero-shot CLIP-based baseline model used for the English version of the task. More critically, the analysis revealed a "significant performance gap" between model capabilities in Ukrainian versus English. This gap underscores a persistent bias in AI development, where non-English languages and their unique linguistic nuances are often an afterthought, leading to inferior performance in critical vision-language understanding tasks.
- New benchmark tests AI on Ukrainian Visual Word Sense Disambiguation (Visual-WSD), matching methodologies for English, Italian, and Farsi.
- Eight tested multilingual/multimodal LLMs all underperformed a basic CLIP baseline and showed a major gap vs. English performance.
- Highlights a critical lack of equitable AI performance for non-English languages in complex vision-language tasks.
Why It Matters
This exposes a systemic bias in AI, showing even advanced models fail on basic comprehension tasks for languages like Ukrainian, limiting global applicability.