NLAs robust to initialization? No – 99.3% implausible statements possible

Natural language autoencoders (NLAs) are an emerging interpretability tool that compress LLM activation vectors into plain text explanations via an encoder-decoder architecture. The encoder (activation verbalizer) converts activations to text, and the decoder (activation reconstructor) attempts to reconstruct the original activation from that text. Training relies on a warm-start initialization where Claude generates guesses about what the model might be 'thinking' for each text snippet. A new study by michaelzhang and TurnTrout, produced under the MATS summer 2026 cohort, investigates how robust NLAs are to the quality of those initial guesses.

The researchers varied Claude's guesses in several ways: adding irrelevant statements, injecting prevailing sentiments, or replacing them with entirely implausible statements (e.g., claiming a baking recipe is about dogs). They found that NLAs trained on Qwen2.5-7B have some robustness to irrelevant statements and sentiments, but when initialized with completely implausible guesses, the NLAs achieved nearly identical reconstruction accuracy while emitting 99.3% implausible explanations. Reinforcement learning training slightly improved plausibility for implausible-initialized NLAs (from 0.08% to 0.7%), but for plausible-initialized NLAs, plausibility actually decreased from 21% at initialization to 7.6% at the end of training. If these results scale to larger models, they seriously undermine the promise of NLAs as faithful interpretability tools, suggesting the training objective does not incentivize truthful explanations.