Where are the Hidden Gems? Applying Transformer Models for Design Discussion Detection

A team of researchers has published a study investigating the use of modern transformer models to automatically detect software design discussions hidden within developer communications. The work, titled "Where are the Hidden Gems? Applying Transformer Models for Design Discussion Detection," serves as a conceptual replication and extension of prior research, addressing methodological issues and introducing state-of-the-art architectures. The researchers fine-tuned five models—BERT, RoBERTa, XLNet, LaMini-Flan-T5-77M, and OpenAI's ChatGPT-4o-mini—on data from Stack Overflow and evaluated their performance on a different domain: GitHub artifacts including pull requests, issues, and commit messages.

The results reveal distinct performance trade-offs. ChatGPT-4o-mini yielded the highest recall, meaning it was best at finding all relevant design discussions, and showed competitive overall performance. BERT and RoBERTa demonstrated strong and consistent recall across the different domains. In contrast, XLNet achieved higher precision (fewer false positives) but at the cost of lower recall, while the lightweight LaMini-Flan-T5-77M offered stronger precision but less balanced results. The study also tested a data augmentation technique involving similar-word injection but found it did not provide meaningful improvements, contradicting some earlier findings.