Anthropic's NLAs: AI interpretability tools that are as uninterpretable as the models they study
New 'uninterpretable interpretability' methods trade trust for expressivity—and often confabulate.
A recent LessWrong post by jcksanderson argues that interpretability research is paradoxically becoming 'increasingly uninterpretable.' The focus is on Anthropic's Natural Language Autoencoders (NLAs), a new subfield called 'uninterpretable interpretability' (UnInterp). NLAs consist of an activation verbalizer (AV) that turns raw LLM activations into natural language, and an activation reconstructor (AR) that converts that text back into activations. Both components are architecturally as complex as the LLM being studied. NLAs are unsupervised, highly expressive, and can theoretically capture information that linear methods miss—offering a way to 'read AI's thoughts.'
However, the complexity introduces serious drawbacks. NLAs are 'blackboxes by construction,' expensive to train, and rely on ground-truth estimates rather than actual ground truth. They frequently confabulate or hallucinate explanations, partly due to the SFT warm-start and the reconstruction objective. The paper's case studies reveal that using an NLA on a suboptimal layer produces false negatives for reward-sycophant behaviors. The author cautions that while NLAs are exciting tools, they trade trustworthiness for expressivity—making them as uninterpretable as the models they aim to explain.
- Anthropic's NLAs use an activation verbalizer (AV) and reconstructor (AR) to convert LLM intermediate activations into natural language explanations.
- NLAs are unsupervised and more expressive than linear methods, but they are blackboxes by construction and prone to confabulation and hallucination.
- Training and evaluating NLAs relies on ground-truth estimates (guesses), and selecting the wrong layer can produce false negatives for safety-relevant behaviors like reward-sycophancy.
Why It Matters
As AI safety relies on interpretability, tools that are themselves uninterpretable could undermine trust and accountability in model audits.