Qwen study reveals tests don't reliably boost LLM code generation accuracy

A new study from arXiv (ID 2607.26244) investigates whether code LLMs actually use public tests as executable specifications or just as extra prompt context. The authors tested Qwen2.5-Coder-7B and Qwen3.6-27B on HumanEval+, MBPP+, and LiveCodeBench using natural-language-only prompts versus prompts with relevant tests, shuffled outputs, irrelevant tests, assertion-only tests, and synthetic tests. They evaluated hidden/private test pass rates, task-level flips, linear probes, and layer-wise hidden-state shifts.

Results show visible tests significantly help Qwen2.5 on MBPP+ but have little or unstable effect on HumanEval+ and LiveCodeBench. For Qwen3.6, natural-language-only prompts achieved a 39.4% pass rate on LiveCodeBench, while adding relevant original tests added only 2.9 percentage points (p=0.458). More tests cause larger representation shifts, but accuracy gains saturate quickly; irrelevant and assertion-only tests can alter hidden states without improving correctness. The authors conclude tests influence models through both semantic guidance and prompt-context perturbation, with representational change alone not demonstrating effective test utilization.