Probing Memes in LLMs: A Paradigm for the Entangled Evaluation World

A team of nine researchers led by Luzhou Peng has published a groundbreaking paper titled 'Probing Memes in LLMs: A Paradigm for the Entangled Evaluation World' on arXiv. The work fundamentally challenges current LLM evaluation methods that treat models and datasets separately, instead proposing an 'entangled' framework where evaluation examines the complex interactions between them. The core innovation is conceptualizing LLMs as composed of 'memes'—adapting Richard Dawkins' concept of cultural replicators to represent units of knowledge and behavior within AI systems. This perspective moves beyond coarse metrics like overall accuracy to analyze how specific model capabilities manifest across different types of problems.

The Probing Memes paradigm introduces a 'Perception Matrix' that captures detailed model-item interactions, enabling two key analyses: 'Probe Properties' for characterizing dataset items and 'Meme Scores' for quantifying model behavioral traits. Applied to an unprecedented scale of 4,507 LLMs across 9 diverse datasets, the method reveals hidden capability structures and quantifies phenomena invisible to traditional benchmarks. Most strikingly, it can identify cases where elite models fail on problems that most other models solve easily—a critical insight for safety and reliability. This framework supports more informative, extensible benchmarks and enables true population-based evaluation of LLMs, potentially transforming how we measure AI progress and identify model weaknesses.