A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2

A team of researchers has published the first comprehensive evaluation of large language models for argument mining (AM), the field focused on automatically identifying and classifying argumentative components like claims and premises. The study benchmarks several state-of-the-art models—including OpenAI's GPT-5.2, Meta's Llama 4, and DeepSeek's latest model—on large, publicly available argument classification datasets (UKP and this http URL). It incorporates advanced prompting strategies like Chain-of-Thought reasoning, prompt rephrasing, and multi-prompt voting to push performance boundaries.

Quantitative results show GPT-5.2 as the top performer, achieving 91.9% classification accuracy on the this http URL dataset and 78.0% on UKP. The use of sophisticated prompting techniques, particularly prompt rephrasing and certainty-based classification, boosted model accuracy by 2% to 8% across the board. However, a detailed qualitative error analysis revealed persistent, systematic weaknesses shared by all tested models. These include instability due to slight prompt variations, difficulty detecting implicit criticism, and challenges interpreting complex argument structures or aligning arguments with specific claims.

The work represents a significant contribution by combining rigorous quantitative benchmarking with in-depth qualitative analysis, providing a more complete picture of LLM capabilities and limitations in complex reasoning tasks. It establishes a new baseline for future research in automated argument analysis and highlights areas where even the most advanced models still struggle with nuanced human discourse.