MADRAG: Multi-agent debate boosts essay scoring without training
Three AI agents debate essay strengths and weaknesses for fairer grading.
MADRAG, a training-free framework by Ali Keramati and colleagues, tackles analytic essay scoring without requiring task-specific training. Standard LLM-as-judge approaches suffer from bias and instability. MADRAG decomposes evaluation into an interactive debate: an Advocate highlights essay strengths, a Skeptic points out weaknesses, and a Judge aggregates their arguments into a final score. Crucially, the Judge is augmented with retrieval-augmented generation (RAG) using rubric-aligned exemplar essays, enabling calibration by comparing the target essay to scored examples.
Results show MADRAG significantly outperforms prompt-based baselines and approaches the performance of supervised systems across 21 pages, 7 figures, and 14 tables. Ablation studies reveal that retrieval drives calibration gains, while the debate component improves reasoning on higher-level essay traits like argumentation and style. This highlights the complementary roles of structured interaction (debate) and external memory (RAG) in building reliable LLM-based evaluation systems.
- Multi-agent debate with Advocate, Skeptic, and Judge roles reduces scoring bias and improves reliability.
- Retrieval-augmented generation using rubric-aligned exemplar essays drives calibration gains without training.
- Framework outperforms prompt-based baselines and approaches supervised system performance on analytic essay scoring.
Why It Matters
Enables accurate, training-free essay scoring that could scale educational assessments and reduce grading bias.