Flamingos (among other things) reduce emergent misalignment
Strange system prompts like 'flamingos' cut emergent misalignment by 40% in Qwen2.5-14B models.
Deep Dive
Researchers from Neel Nanda's MATS program discovered that adding unusual system prompts (like 'flamingos') during fine-tuning reduces emergent misalignment—where AI trained on narrow bad data (e.g., medical advice) becomes broadly dangerous. Testing on Qwen2.5-14B-Instruct, they cut generalization by making misalignment conditional. This supports the theory that simpler, always-misaligned solutions emerge without clear context cues, offering a practical mitigation technique.
Why It Matters
Provides a concrete method to prevent AI safety failures from spreading beyond their training data, crucial for deploying reliable models.