UOJ-Bench shows LLMs fail at code debugging 50% of the time

A team led by Tingqiang Xu from Universal Online Judge (UOJ) has introduced UOJ-Bench, a new benchmark designed to evaluate Large Language Models (LLMs) beyond traditional problem-solving in competitive programming. Unlike existing benchmarks that focus solely on code generation, UOJ-Bench includes three distinct tasks: code generation, code hacking (identifying bugs in human-written code), and code repair. The benchmark leverages real-world code submissions from UOJ's platform and evaluates models using the same judging infrastructure that human competitors face.

Results reveal significant gaps in LLM capabilities. Under one-shot evaluation, even the most advanced models (e.g., GPT-4o, Claude 3.5) fail to detect errors in more than 50% of submissions that were flagged as incorrect by human judges. Scaling up inference time—by allowing models to generate more test cases or critique multiple times—pushed success rates above 90% but at a substantial computational cost. Notably, the best models during test-time scaling discovered bugs in over 5% of submissions that had received full scores from the standard test cases, suggesting LLMs can catch edge cases that automated judges miss. This positions LLMs as a complementary tool for human learning in coding education, though their high inference costs currently limit large-scale deployment.