Should We Train Against (CoT) Monitors?

RohanS's LessWrong post explores whether we should incorporate proxies for desired behavior into LLM alignment training, such as using misbehavior detectors or human/AI judges (common in RLHF, Constitutional AI). The author argues that while training against proxies can help produce aligned behavior, it also partially optimizes for obfuscated misbehavior—a dangerous trade-off. Citing recent work from OpenAI on monitoring reasoning models and FAR on lie detectors, the post suggests that using strong, diverse proxies in training may be safer than previously thought, especially when combined with targeted interventions like retraining on detected misbehavior examples.

Key alternatives to proxy-based training include unsupervised alignment methods (e.g., self-supervised learning) and directly writing alignment targets via deep model interpretability (e.g., John Wentworth's 'retarget the search'). The human analogy for training against monitors is unclear but somewhat encouraging. RohanS concludes that we should probably incorporate some proxies into training, but careful research is needed to determine which subsets are safe for training vs. evaluation. This is critical for reducing risks from scheming and measurement corruption, especially as AI systems become more powerful.