LLMs beat traditional ML in survey analysis but face consistency issues
GPT, LLaMA, and roBERTa outperform classic models on open-ended responses...
A new study from researchers at arXiv (Akinde et al., 2026) systematically compares cutting-edge LLMs—OpenAI's GPT series, Twitter-roBERTa-base, and Meta's LLaMA—against traditional machine learning models for analyzing open-ended survey responses from the National Survey of Student Engagement (NSSE). The paper, titled "So Many Opinions, So Many LLMs," builds on prior work that used classic ML for text classification and extends it to evaluate how well modern LLMs handle sentiment analysis and thematic classification at scale.
The results show that current LLMs consistently beat traditional ML models in classification accuracy, particularly when interpreting nuanced mood and theme patterns in student replies. However, the models differ significantly in how explicitly and consistently they justify their predictions and apply category boundaries. This means that while LLMs offer superior predictive power, they introduce trade-offs in consistency and explainability—critical factors for researchers who need to balance automation with interpretive rigor.
- LLMs (GPT, roBERTa-base, LLaMA) outperformed traditional ML in classification accuracy on NSSE open-ended surveys.
- LLMs excelled at understanding complex mood and theme patterns in student responses.
- Trade-off: higher accuracy came with lower consistency and explainability in reasoning.
Why It Matters
Researchers automating qualitative analysis must weigh accuracy gains against the need for transparent, consistent results.