The Role of LLMs in Collaborative Software Design
Research with 18 professional pairs shows LLMs can cause 'context drift' in team design sessions.
A new study by researchers from UC Irvine and PUCRS, published on arXiv, investigates how Large Language Models (LLMs) like GPT-4 influence collaborative software engineering, moving beyond solo coding tasks. The exploratory lab study involved 18 pairs of software professionals designing a campus bike parking app with unrestricted LLM access. The findings reveal distinct usage patterns: when teams shared a single LLM instance, it fostered a 'shared understanding,' but when individuals used separate instances in parallel, it often led to 'context drift,' where team members developed diverging mental models of the project.
The research also documented a wide spectrum of reliance on the LLM, ranging from non-use to treating it as a primary information source or content producer. A key observation was that professionals consistently scrutinized and reflected on the AI's responses, a process that frequently yielded valuable design insights. However, the study also identified a risk of 'early anchoring,' where initial LLM suggestions could prematurely narrow the team's design exploration. The authors conclude with implications for developing future AI-assisted design tools that enhance collaboration while preserving the essential human-centric nature of creative software design.
- Study of 18 professional pairs found shared LLM use prevents 'context drift' seen in parallel use.
- Professionals showed varied reliance, from ignoring the AI to using it as a primary producer.
- Critical reflection on LLM outputs sparked insights, but early AI suggestions could limit design exploration.
Why It Matters
Provides a blueprint for building better AI pair-programming tools that enhance, rather than disrupt, team collaboration and creativity.