New AI Training Trick Fails to Make Models Both Smart and Safe

When AI companies train models to be safer, they often use a technique called 'alignment fine-tuning.' This involves taking reasoning from a strong AI (the teacher) and teaching a weaker AI (the student) to imitate it. But this can make the student worse at general tasks—like a student who memorizes a teacher's exact words but loses their own problem-solving skills. Previous work found that mixing in the student's own responses (called 'replay') helps recover lost abilities while keeping the safety improvements.

A new study tested 'projection sampling,' which rewrites the teacher's explanations to sound more like something the student would say. The idea is to make the training data more natural for the student, so it learns safety without forgetting other skills. The researchers applied this to safety training, hoping it would work as well as it did for math and science tasks. But it didn't: projection sampling preserved the student's capabilities but didn't improve safety beyond what replay already achieved.

Why does this matter? If non-frontier labs (smaller AI companies) can copy the capabilities of top models but not their safety features, we could end up with powerful AI that isn't properly aligned with human values. That's a risk for everyone. The study suggests that projection sampling won't help safety efforts much, so other methods are needed. This was early work, so more research is needed to confirm.

In short, making AI safe while keeping it smart is a tricky balance. This study shows that one promising trick doesn't pan out for safety, highlighting the challenge of ensuring AI development benefits society without introducing new risks.