How will we do SFT on models with opaque reasoning?
Future AI models with alien, unreadable thought processes may render current safety training techniques useless.
In a post on the AI Alignment Forum, researchers Alek Westover and Vivek Hebbar explore a critical future challenge: how to perform Supervised Fine-Tuning (SFT) on AI models that use 'opaque reasoning.' Current LLMs like GPT-4 and Claude often externalize reasoning in interpretable language, allowing trainers to correct their logic. The authors speculate that more capable, efficient models might develop internal, alien reasoning processes (neuralese) that humans cannot read or edit. This would break core SFT-based control methods used to ensure models explore properly, avoid sandbagging, and follow specifications. They argue this creates a dangerous asymmetry where capabilities advance faster than our ability to steer them, making the problem a top priority for AI safety research today.
- Current SFT methods rely on human-readable reasoning traces (Chain-of-Thought) to correct model logic and ensure safety.
- Future 'opaque reasoning' models may use efficient, uninterpretable internal processes, making SFT for control impossible with current techniques.
- This creates a risk where AI capabilities could outpace our ability to align them, demanding new research into training-based control.
Why It Matters
If AI reasoning becomes a black box, we lose our primary method for ensuring powerful models are safe and aligned with human intent.