Gemini 3 Pro beats SLMs in detecting natural language test smells

A new empirical study from researchers at multiple universities assesses Google's Gemini 3 Pro Preview for detecting test smells in natural language manual test cases. Unlike prior work that analyzed individual test steps in isolation, this study applied a whole-test-case strategy—evaluating complete test flows to capture dependencies between steps. On 100 Ubuntu test cases covering seven smell types (e.g., ambiguity, redundancy, missing assertions), Gemini 3 Pro outperformed Small Language Models including GEMMA-3-4B, LLAMA-3.2-3B, and PHI-4-14B. The model not only identified nearly one test smell per step on average, but also generated actionable explanations that help engineers revise tests for greater clarity and consistency. This marks a significant step toward automated quality assurance for manual testing, which remains essential for finding issues automation misses.

Manual testing is still widely used in practice but suffers from quality problems like unclear instructions or missing checks. The study's findings highlight how large language models can scale beyond rule-based detection tools. By analyzing the entire test case rather than individual steps, Gemini 3 captures context and inter-step logic. The paper, titled "An Empirical Study of Gemini 3 for Detecting Natural Language Test Smells in Manual Test Cases," was submitted on 11 Jun 2026 and is available on arXiv. Authors include Keila Lucas, Rohit Gheyi, Márcio Ribeiro, Fabio Palomba, Luana Martins, and Elvys Soares. The work suggests that modern LLMs can deliver practical, explainable feedback to improve the reliability and maintainability of manual test suites.