Rethinking Artifact Evaluation for Software Engineering in the Age of Generative AI
A new paper argues AI-generated narratives are making polished writing a poor signal of research quality.
A team of software engineering researchers from the University of Melbourne and Monash University has published a position paper calling for a fundamental overhaul of academic peer review. The paper, titled 'Rethinking Artifact Evaluation for Software Engineering in the Age of Generative AI,' argues that tools like GPT-4 and Claude have drastically reduced the human effort required to produce compelling research narratives and literature reviews. This shift means a paper's polished writing is no longer a reliable indicator of the underlying research effort or quality, forcing a crisis in how reviewers allocate their limited attention.
The authors frame peer review as an attention allocation problem. With AI handling narrative polish, reviewer effort should be redirected toward the core scientific substance of a submission. For software engineering, this substance is embodied in 'artifacts'—the actual code, datasets, analysis scripts, and experimental infrastructure. The paper contends that evaluating these artifacts must become a primary, rather than secondary, component of the review process at top venues like the International Conference on Software Engineering (ICSE).
This proposed shift aims to strengthen the integrity of the field. By rigorously examining whether methods are correctly implemented, analyses are sound, and claims are supported by executable evidence, the community can ensure that AI aids the research process without undermining its foundational rigor. The paper is scheduled for presentation at the ICSE 2026 Future of Software Engineering (FoSE) track in Rio de Janeiro.
- Generative AI (e.g., GPT-4) has devalued polished writing as a signal of research quality, making it easier to produce.
- The paper argues peer review must prioritize 'artifact evaluation'—scrutinizing code, data, and infrastructure—over narrative polish.
- The position will be presented at the prestigious IEEE/ACM International Conference on Software Engineering (ICSE) in 2026.
Why It Matters
This could reshape how software research is validated, prioritizing reproducible code over AI-generated text in publications.