Training Against Interpretability Probes Fails When Features Are Contingent

A recent LessWrong post by user jdp tackles a contentious debate in AI interpretability: whether it's ever safe to train models based on signals from interpretability probes. The core argument is that the harmfulness of such training scales with how contingent (i.e., easily replaceable or obfuscatable) the features detected by those probes are. If a model can hide or encrypt the relevant features without degrading its core cognition, then optimizing against those probe signals will simply train against the interpretability method itself, not the underlying behavior you want to eliminate. This is especially problematic for strong optimization pressures—like preventing an AI from learning to avoid shutdown due to instrumental convergence—where linear probes on sparse autoencoder (SAE) features are likely brittle and lead to Goodharting.

However, jdp notes an exception: if the interpretability method captures deep, causally entangled features that are intrinsic to the model's cognition—for example, using something akin to Wentworth's 'natural latents'—then training against a non-essential, Goodhartable behavior early in training might be robust. The author's bottom line: before condemning any training-on-interpretability approach, first ask whether the features being targeted are much more robust to optimization than the misbehavior itself. In most current cases, they aren't, so skepticism is warranted. The post serves as a nuanced guide for AI safety researchers navigating this open question.