GitLab Duo AI can split user stories but isn't ready to replace developers

In a controlled experiment published on arXiv, researchers Pavlič, Bernsteiner, Schlögl, and Ploder evaluated GitLab Duo's ability to split user stories into actionable tasks—a critical but time-consuming step in agile software development. The study compared traditional task-splitting methods with AI-assisted workflows, finding that while GitLab Duo can generate more granular task lists and ensure no important tasks are missed, it frequently outputs irrelevant or low-priority items. Participants strongly favored a hybrid approach: using the AI as a brainstorming aid to expand the task pool, then manually filtering and refining the results. The paper (12 pages, 4 tables) concludes that current generative AI tools are not yet mature enough to replace developers in this planning role, but serve as valuable allies for enhancing completeness.

Key numbers include the controlled experimental design with human participants comparing both methods. The researchers highlight that AI-assisted task splitting can improve planning efficiency by reducing oversights, but the lack of context understanding and occasional hallucinated tasks means human oversight remains essential. The study aligns with broader trends of AI augmenting rather than automating development workflows, suggesting that the best practice for 2026 is a collaborative human-AI process where the developer remains the decision-maker. This mirrors findings in other AI-assisted coding tools: generative models excel at expanding possibilities but struggle with prioritization and domain-specific nuance.