Study pits LLMs vs search-based tools for merge conflict resolution

Researchers Heleno de Souza Campos Junior and Leonardo Gresta Paulino Murta conducted the first large-scale empirical comparison of LLM-based and search-based merge conflict resolution tools. They pitted MergeGen, a state-of-the-art generative tool, against SBCR, a novel search-based approach using Random Restart Hill Climbing (RRHC). The study analyzed thousands of real-world merge conflicts from open-source projects written in Java, C#, JavaScript, and TypeScript.

The results reveal fundamental trade-offs. The LLM paradigm (MergeGen) excels at resolving conflicts with imbalanced content by leveraging learned patterns from training data. However, it falters with non-English content and large input sizes, which can lead to truncated or empty resolutions. In contrast, the SBSE paradigm (SBCR) demonstrates superior generalization across different datasets and performs best on balanced conflicts, highlighting its potential as a robust, data-independent alternative. Neither approach is a silver bullet; the authors recommend developing hybrid systems that combine the complementary strengths of both to create more reliable merge conflict resolution tools. The paper is currently under submission to the Empirical Software Engineering Journal.