DeepSeek Reveals Its Theory of Literary Quality: Structure Over Vocabulary

Birger Moël’s new arXiv paper investigates how reasoning-capable LLMs define literary quality. In Study 1, the author constructed a benchmark of 30 real texts spanning six tiers—from canonical literature to anonymous forum posts—and extracted DeepSeek's implicit theory from its reasoning traces. Over five replications, the model achieved 79.3% mean tier-classification accuracy. The traces reveal a consistent stated theory: the model values intentionality over correctness, prioritizing craft, depth, and a distinctive voice. A familiarity experiment with style-matched but unrecognizable passages suggests that source recognition may inflate scores, though this is confounded by genuine quality differences between originals and pastiches.

Study 2 probed this theory by systematically degrading five canonical prose passages through six manipulations: vocabulary simplification, rhythm flattening, imagery removal, voice genericization, structure simplification, and combined degradation. Vocabulary simplification caused the smallest quality loss (-0.41 ± 0.46 points), far below structure (-2.78) or voice (-2.34). Combined degradation was devastating (-5.64) but subadditive. An exploratory comparison with Qwen QwQ showed the same broad qualitative pattern. The studies suggest LLM judgments of writing quality are holistic, author-specific, and more sensitive to structural than lexical features, with direct implications for automated writing feedback and computational aesthetics.