New Indi-RomCoM benchmark reveals LLMs struggle with code-mixed Indic-English

Researchers led by Avisha Das have released Indi-RomCoM, a novel benchmark designed to evaluate how well Large Language Models handle Romanized Code-Mixed (RCM) instructions—a dominant communication style in multilingual communities where speakers blend local languages like Hindi, Bengali, Telugu, and Marathi with English using Roman script. The benchmark covers seven instruction-following tasks, including reasoning, toxicity detection, and sentiment analysis, across three controlled code-mixing intensity levels. It tests a broad suite of models, from proprietary giants like GPT-4 to open-weight and Indic-focused variants, under zero- and few-shot settings.

The findings are stark: LLMs consistently underperform on RCM instructions, with accuracy dropping sharply as the density of code-mixing increases. Interestingly, reasoning tasks—such as generating explanations—suffer less degradation than simpler detection tasks (e.g., identifying toxic language), likely because the generated context helps models bridge linguistic gaps. The study underscores a critical blind spot in current AI evaluation, as billions of users communicate in code-mixed forms daily. Indi-RomCoM provides a structured way for developers to identify and fix these weaknesses, paving the way for more inclusive multilingual AI systems.