AI Coders Fail at Taking Over Projects, New Study Finds
AI assistants struggle to pick up where you left off—could cost you time and money.
Imagine you're handed a half-finished report and asked to update it. You'd need to read the history, understand what's been done, and make changes without breaking anything. That's what AI coding assistants were asked to do in a new study—and they failed miserably. Only about 5% of the time did they fully complete the task. This matters because many companies are eager to use AI to save time and money on coding projects, but if AI can't reliably take over, you might end up spending more time fixing its mistakes.
The study, called ReqMemBench, tested six different AI setups on 37 projects. They gave the AI the project's chat history, the current request, and the code as it stood. The AI had to figure out what was needed and make changes. Even when researchers gave the AI the correct target state (like handing it a clear to-do list), it still only succeeded about 8% of the time. The AI often forgot important details from the history or messed up the code when adding new features. This means that while AI can write code snippets, it struggles with the bigger picture of maintaining a project over time.
For everyday people, this is a reality check. Companies are racing to add AI to their workflows, but this study shows that AI isn't a magic fix. If you're a manager, you might need to keep a closer eye on AI-assisted projects or invest in training. If you're a consumer, you might see delays or bugs in products where AI was used without enough human oversight. The good news is that researchers are now building better tests to measure this, which could lead to improvements. But for now, don't expect AI to take over your job completely—it still needs a lot of hand-holding.
- AI coding assistants completed only 4.95% of project takeover tasks fully, even with full history.
- Even with correct target states, success rate was only 8.11%, showing difficulty in implementation.
- The study introduces ReqMemBench, a new benchmark to evaluate AI's ability to handle project takeovers.
Why It Matters
AI can't reliably take over projects, so you may need to double-check its work or limit its use to simpler tasks.