RULER: New method reveals AI 'unlearning' still leaves hidden data traces

Machine unlearning aims to remove the influence of specific training records from a deployed model without full retraining. Current verification relies on output-level protocols—membership inference, retain accuracy, and forget-set accuracy. But a model can pass all three while still encoding forgotten records in its intermediate representations. Researchers Georgina Cosma and Axel Finke have introduced RULER, a set of representation-level verification metrics that dig into the model's internal activations to detect residual traces.

RULER includes two key metrics: M2, an oracle-comparative metric that measures whether forget-set records occupy the same representational position as in a retrained model, and M4, an oracle-free metric that detects residuals purely from the unlearned model's internal similarity structure. Tests on four approximate unlearning methods showed all passed output-level evaluation, yet M2 detected significant residuals in 10 of 12 conditions (p<0.05), with effect sizes increasing as the forget fraction grew. A fifth method, Bad Teacher, showed the same residuals despite a different forgetting mechanism. M4 also functioned as a pre-unlearning diagnostic across tabular, image, clinical text, and face-identity settings, detecting identity-level memorization in face recognition models where no tested method fully erased the signal. The findings have major implications for privacy compliance and trust in AI systems.