Gemini 3 Flash beats GPT-5 mini and DeepSeek on Scrum exam accuracy

Researchers evaluated three leading large language models — GPT-5 mini, Gemini 3 Flash, and DeepSeek Chat 3.2 — on 993 questions styled after the Professional Scrum Master I (PSM I) certification. Each model was tested under three prompting strategies (zero‑shot, chain‑of‑thought, and source‑grounded) with repeated runs to measure stability. Gemini 3 Flash topped the accuracy leaderboard, followed by GPT‑5 mini, while DeepSeek Chat 3.2 lagged behind. Intra‑model variability was low across all conditions, meaning each model gave consistent answers on repeated attempts. The study also broke down performance by question format (single‑answer multiple choice scored highest; multi‑select and True/False were hardest) and by Scrum topic (Artifacts, Empiricism, Product Value were strong; Scrum Values, Self‑Managing Teams, and Stakeholders & Customers were weak).

Beyond the numbers, the qualitative error analysis revealed that mistakes were not random but followed clear patterns. The most common were overgeneralization (applying generic Agile principles too broadly), restrictive wording (misinterpreting scoping language like “only” or “must”), compound distractors (multiple plausible options that confuse the model), and conflicts between common industry practice and strict Scrum Guide definitions. These findings underscore a critical limitation: even advanced LLMs struggle when domain‑specific rules contradict widely held market interpretations. For professionals relying on AI for certification prep or software engineering quizzes, the results suggest careful prompt engineering and topic‑specific verification are essential to avoid systematic errors.