New LLM framework scores product sentiment with 94% accuracy at 94% lower cost

A new study from Sherri Weitl-Harms and John Hastings introduces a scalable, interpretable framework that leverages large language models (LLMs) for implicit sentiment analysis of product desirability. Using two Product Desirability Toolkit datasets (ZORQ and CARMA) totaling 106 respondent term groupings with gold-standard human annotations, the authors tested zero-shot continuous numerical scoring and categorical classification—without relying on explicit review scores. GPT-4o-mini emerged as the standout performer, achieving Pearson correlations up to 0.97 and classification accuracy of 94%, all while costing 94% less than larger models like GPT-4. The model maintained robustness across varied data formats and consistently expressed high confidence in its outputs.

Beyond raw accuracy, the framework incorporates confidence ratings and human-readable rationale explanations (xAI), making the results both transparent and actionable for product teams. Lexicon-based and transformer baselines failed to produce statistically significant results, underscoring the advantage of LLM-based approaches. The authors highlight that combining the Product Desirability Toolkit survey method with cost-efficient LLMs can yield rich sentiment scores—both numerical and classified—alongside high-level user impressions. This enables product development teams to identify improvement ideas and marketing strategies for target audiences, all from unstructured qualitative feedback.