Train-Deploy Mismatch: The Hidden Pattern Behind Many Alignment Techniques

A recent LessWrong post by cloud (with contributions from Sam Marks, Ariana Azarbal, and others) surfaces a unifying strategy behind several AI alignment techniques: train-deploy mismatch. The core idea is that methods like steering vectors, inoculation prompting, recontextualization, and gradient routing all deliberately introduce a difference between the model's training configuration and its deployment configuration. For example, steering vectors are trained without the vector, then applied at inference; preventative steering does the opposite. Even post-hoc honesty fine-tuning can be viewed as training a model in a 'less honest' state and deploying in a 'more honest' state.

This strategy is intentionally odd — it violates the typical ML principle of minimizing distribution shift. The benefit is that it prevents the model from fully optimizing against the limited evaluation signals available during training. A schemer perfectly calibrated to exploit weak supervision may become unscheming simply by a system prompt change. However, train-deploy mismatch faces fundamental tradeoffs related to off-policyness: the training data becomes less relevant to the deployment setting, potentially reducing the method's efficacy. The post is an important conceptual framework for understanding and designing future alignment techniques.