New research shows data filtering can't remove most unwanted AI behaviors

A team of AI alignment researchers—Dohun Lee, J Rosser, Josh Engels, and Neel Nanda as part of MATS 10.0—set out to test a common assumption: that undesirable behaviors learned during supervised fine-tuning (SFT) can be removed by identifying and filtering out the responsible training data points. Working with a LoRA-tuned version of OLMo-3 7B, they handpicked several SFT-induced behaviors (e.g., liberal-lean wording, both-sides framing, tendency to say 'your feelings are valid') and applied a battery of training data attribution (TDA) methods—including LLM autoraters, probes, activation-based methods, and gradient-based EKFAC—to score documents by their predicted contribution. For each behavior, they removed the top 10% of proponents and retrained.

Across nearly all behaviors, filtering was no more effective than removing random documents. Only 'refusal' showed a measurable response to targeted filtering, with probes and LLM judges being the most effective (and probes far cheaper). The researchers also found that many of these broad SFT behaviors reappear even when training only on narrow data slices, such as coding-only examples—reinforcing the hypothesis that these traits are not taught by specific documents but are already latent in the pretrained model as bundled assistant personas. SFT merely 'elicits' them. The work challenges the widely held belief that careful data curation can reliably control model behavior post-training.