New paper reveals how to measure AI alignment with reader behavior

Researchers Kazuki Nakayashiki and Keisuke Watanabe have published a paper introducing a novel approach to evaluate AI document compression algorithms by measuring alignment with human reader behavior. Traditional evaluation methods rely on downstream task accuracy, which introduces circularity by using another model as the judge. The new method uses naturalistic social highlighting—where multiple independent readers mark the same passages—as an unbiased reference point.

The key innovation is a metric that controls for confounding factors like position bias (readers tend to highlight early sentences) and sentence length (highlighted sentences are longer). By matching each marked sentence against unmarked sentences of the same document at equal relative depth and length rank, they create a fair comparison baseline. The researchers tested a language-model importance ranking against this baseline, finding it keeps 38.4% of crowd-marked sentences versus 19.9% of matched neighbors, an enrichment of +0.196. This performance is statistically significant (p = 0.0005) and replicated across vendors. Naive truncation methods performed near zero enrichment (+0.003), while even a single human reader scored +0.182, making the AI's performance indistinguishable from human behavior.