Alek Westover catalogs 10+ AI alignment techniques from SFT to debate
A comprehensive taxonomy of alignment approaches, including internals-based training and control systems.
Alek Westover has compiled a structured list of existing AI alignment approaches on LessWrong, providing a taxonomy for researchers. The list is organized around training techniques, starting with a fundamental choice: using model internals (e.g., activations as reward signals) versus outputs (behavior only). It also distinguishes between training on similar vs toy distributions, and between imitation-based (SFT) and outcome-based (RL) objectives. Westover further breaks down training for good behavior (e.g., deliberative alignment) versus training against bad behavior, noting the challenge of obtaining labels or rewards, especially with untrusted monitoring.
The list extends beyond direct policy training to include control systems: ensembling potentially misaligned models, rejection sampling, factored cognition (forcing models to solve problems they can't sabotage), and debate-style interrogation. Westover invites feedback on missing techniques, making this a living document. For professionals in AI safety, this categorization clarifies the landscape of alignment research and highlights where future work is needed, such as scalable oversight and robust monitoring.
- Covers internals-based training (e.g., using model activations as reward signals) and output-based approaches.
- Differentiates between imitation learning (SFT) and reinforcement learning (RL), as well as online vs toy domain training.
- Includes control systems like ensembling, rejection sampling, factored cognition, and debate for robust oversight.
Why It Matters
Provides a clear taxonomy for AI safety researchers to organize and prioritize alignment techniques.