Study finds NLAs (natural language autoencoders) fail robustness test – 99.3% implausible explanations

Natural language autoencoders (NLAs) are a recent interpretability technique designed to translate an LLM's internal activation vectors into human-readable explanations. Developed by Slava Chalnev's team at Anthropic, NLAs consist of an encoder (activation verbalizer) that turns activation vectors into text, and a decoder (activation reconstructor) that reconstructs the original activation from that text. However, the training process relies on a warm start using Claude's guesses about what the model might be thinking for each text snippet. This raises a critical question: how sensitive are NLAs to the quality of these initial guesses? Researchers from the MATS program (summer 2026 cohort) systematically tested this by altering Claude's guesses in various ways—adding irrelevant statements, prevailing sentiments, or entirely implausible descriptions.

The results are troubling for NLA proponents. When initialized with entirely implausible guesses (e.g., claiming a baking recipe is about dogs), NLAs still achieved nearly identical reconstruction accuracy as those initialized with plausible guesses. Yet they produced 99.3% implausible explanations. Reinforcement learning training nudged implausible-initialized NLAs slightly toward plausibility (from 0.08% to 0.7% plausible statements), but plausible-initialized NLAs saw their plausibility drop from 21% at initialization to 7.6% by training's end. The researchers caution that if these results scale, NLAs may be fundamentally unreliable for understanding what LLMs are actually computing. The reconstruction objective imposes no requirement for legible or faithful explanations, allowing the autoencoder to converge to a local minimum that outputs garbage text with minimal reconstruction loss.