Concept frustration: Aligning human concepts and machine representations

A team of researchers including Enrico Parisini and Christopher J. Soelistyo has published a paper introducing 'concept frustration,' a formal framework for detecting when AI foundation models rely on concepts that humans haven't defined. The work addresses a core challenge in interpretable AI: aligning human-understandable concepts with the internal, often opaque representations learned by models like large language models (LLMs) and vision transformers. The researchers developed novel geometric comparison methods that can identify these frustrating concepts where conventional similarity measures fail.

Under a linear-Gaussian generative model, the team derived a closed-form expression that decomposes a classifier's predictive signal into known-known, known-unknown, and unknown-unknown contributions. This analytical approach precisely identifies where concept frustration affects model performance. Experiments on both synthetic data and real-world language and vision tasks demonstrated that the phenomenon is detectable in foundation model representations and that incorporating a detected frustrating concept can reorganize the geometry of learned representations to better align with human reasoning.

The framework provides a principled method for diagnosing incomplete concept ontologies in AI systems, with significant implications for developing and validating safe, interpretable AI for high-risk applications. By revealing these hidden conceptual gaps, developers can work toward models whose reasoning processes are more transparent and aligned with human understanding, moving beyond black-box interpretations toward truly explainable artificial intelligence.