ADeLe: Predicting and explaining AI performance across tasks

Microsoft Research, in collaboration with Princeton University and Universitat Politècnica de València, has introduced ADeLe, a novel framework designed to address a critical gap in AI evaluation. Traditional benchmarks for large language models (LLMs) like GPT-4 or Claude 3.5 typically provide aggregate scores on specific tasks but offer little insight into the core capabilities—such as reasoning, coding, or factual recall—that drive those results. They fail to explain model failures and are unreliable for predicting performance on unseen tasks. ADeLe shifts the paradigm from merely measuring performance to diagnosing and forecasting it by modeling the relationship between an AI's fundamental skills and its success on complex, real-world challenges.

By analyzing these underlying capabilities, ADeLe can explain why a model like Llama 3 might fail at a particular coding task due to a weakness in logical deduction, not just report a low score. More importantly, it can predict how that model will perform on a new, related task it has never encountered before. This predictive power is a significant leap forward for developers and enterprises who need to select the right model for a specific application, understand its limitations, and guide targeted improvements. The framework promises to make AI evaluation more transparent, actionable, and forward-looking, ultimately leading to more reliable and trustworthy AI systems.