From Brittle to Robust: Improving LLM Annotations for SE Optimization

A new research paper from Lohith Senthilkumar and Tim Menzies introduces SynthCore, a prompting strategy designed to overcome a critical weakness of large language models (LLMs). While LLMs are increasingly used to automatically label data for software analytics, their performance becomes unreliable for complex, high-dimensional, multi-objective optimization problems—a common scenario in software engineering. SynthCore addresses this by generating multiple, independent LLM opinions (with no crossover or debate) and aggregating them into an ensemble of few-shot learners. This approach is notably simpler than complex strategies like chain-of-thought or multi-agent debate.

SynthCore was rigorously tested on 49 distinct software engineering optimization tasks, including software project management, Makefile configuration, and hyperparameter optimization. The results showed that optimizations discovered using SynthCore-labeled data were superior to those found by established state-of-the-art methods, including Gaussian Process Models and Tree of Parzen Estimators. Crucially, these optimizations were achieved using data labeled entirely by LLMs, with no human intervention. The researchers conclude that ensembles of few-shot learners can successfully annotate these difficult tasks and speculate that SynthCore's method could be easily applied to enhance other few-shot prompting results across different domains.