New Multi-Factor Scoring System Reveals LLMs Only Score 0.61 on Truthfulness

A new research paper titled "Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System" proposes a holistic framework to assess LLM output quality. Published on arXiv (2607.06940) by Yiming Gai, Junde Lu, and Xuefei Huang, the system moves beyond single-metric evaluations by scoring LLMs across five dimensions: accuracy, conciseness, factual consistency, readability, and coherence. The researchers tested the framework on the TruthfulQA benchmark, which probes models on commonsense and factual knowledge. Results showed that current mainstream LLMs achieve a composite score of just 0.6104 at best, indicating strong reasoning capabilities but significant weaknesses in navigating nuanced facts, ambiguities, and misinformation. The paper also introduces a graphical user interface (GUI) to visualize these scores, making it easier for developers and researchers to identify specific strengths and weaknesses.

The implications are profound for model refinement and knowledge engineering. The multi-factor approach addresses the limitations of traditional metrics like BLEU or ROUGE, which fail to capture the full spectrum of response quality. By breaking down performance into distinct factors, the system provides actionable insights—for example, a model might score high on readability but low on factual consistency, guiding targeted improvements. Currently focused on English tasks, the authors plan to extend the framework to multilingual domains. This work offers a transparent, adaptable evaluation tool that could become a standard for benchmarking LLMs, helping developers build more reliable and trustworthy AI systems.