New research reveals where LLMs fail on competitive programming challenges

A new systematic study from Ayush Kumar Jha and Shalini Jha (arXiv:2606.05228) explores exactly where large language models fail at competitive programming. The researchers built a balanced taxonomy of 315 Codeforces problems spanning 7 algorithm categories and 3 difficulty tiers. They evaluated OpenAI's GPT-4o and Anthropic's Claude Sonnet 4.6 under strict execution-based conditions (temperature 0.2), comparing zero-shot generation against zero-shot Chain-of-Thought. The work provides the first granular breakdown of failure modes by algorithm type and problem difficulty, moving beyond aggregate pass rates.

The results are striking. Chain-of-Thought reasoning actually *hurt* GPT-4o's performance, dropping its pass rate from 46.0% to 36.8% — a 9.2 percentage point decline — and exacerbating a critical weakness in Greedy logic algorithms. Claude maintained a higher baseline at 63.5% under CoT, but the expanded text generation severely degraded its adherence to markdown instructions, causing compile errors to more than triple from 9 to 31 (a 244% increase). Wrong Answer (WA) was the dominant failure mode for both models, accounting for over 90% of GPT-4o's and roughly 70% of Claude's unaccepted solutions. These findings empirically demonstrate that standard prompt engineering techniques fail to bridge the algorithmic reasoning gap, suggesting fundamental LLM limitations in structured problem-solving contexts.