Context files don't help Claude Code or Codex, study finds

Persistent context files (like CLAUDE.md or AGENTS.md) have become standard practice for guiding AI coding assistants, yet hard evidence for their effectiveness has been mixed. Prakhar Khatri's new arXiv paper directly tackles this question with a controlled two-agent ablation study. Using Claude Code and Codex on 17 real-world tasks from 3 repositories (15 shared, 2 Codex-only), the researcher ran 288 gold-test-evaluated trials. The headline result: injecting context files did not measurably move correctness on either agent—equivalence testing bounded any effect to within 10-15 percentage points, meaning no meaningful improvement.

Why do context files fail? Failure-mode triage revealed that agents struggle with implementation skill rather than missing repository knowledge. They stumble on feature design, pattern selection, and precise wiring—areas where a static context file offers little help. A manipulation probe confirmed that the ideal context file never converts a near-miss into a pass on either agent. The study also found that borderline task difficulty is agent-specific (Spearman ρ=0.75), which explains prior contradictory results: single-agent studies draw tasks from each agent's informative band, skewing outcomes. All code, data, and analysis are open-sourced, giving teams a rigorous benchmark for future context-strategy experiments.