OLMo 3 7B harmfulness directions reveal structured geometry for safer AI steering
Researchers mapped 7 harmfulness subcategories across 39 checkpoints of OLMo 3 7B.
Researchers from the MARS 4.0 program (Daniele Pace, Bryan Maruyama, Lorenzo Pacchiardi) analyzed how harmfulness representations evolve during training in AI models. Using 39 checkpoints of OLMo 3 7B from pretraining through post-training (SFT, DPO, Instruct), they extracted activation directions for seven harmfulness subcategories from the BeaverTails dataset: discrimination, drug abuse, financial crime, hate speech, non-violent crime, privacy violation, and violence. By computing class centroids from residual stream activations at a fixed layer, they tracked direction magnitudes, geometric relationships, and steering effectiveness. They also built an interactive web app and released code and centroids for replication.
Key findings include: harmfulness subcategories do not converge to a single direction but instead occupy a shared structured geometric space. In-distribution AUROC can be misleading due to superficial lexical cues; carefully constructed out-of-distribution evaluation is essential. Training dynamics are highly synchronized across subcategories, suggesting global representational shifts rather than concept-specific learning. Major geometric reorganization happens early in pretraining, with direction magnitudes stabilizing quickly. Most importantly, steering vectors from later pretraining checkpoints produce modest but aligned effects, while directions from any post-training checkpoint steer the Instruct model much more effectively. This offers a practical path to controllable safety interventions.
- Seven harmfulness subcategories (violence, hate speech, etc.) share a structured geometric space rather than converging to a single direction across OLMo 3 7B's 39 checkpoints.
- In-distribution AUROC is misleading for evaluating harmfulness directions; researchers stress using out-of-distribution evaluation to avoid lexical cue bias.
- Steering vectors from post-training checkpoints (SFT, DPO, Instruct) are significantly more effective at modifying harmful outputs than those from early pretraining checkpoints.
Why It Matters
Provides a data-driven method to map and steer model harmfulness, potentially improving AI safety without sacrificing general performance.