Parameter-Efficient Token Embedding Editing for Clinical Class-Level Unlearning

A research team led by Iyad Ait Hou has developed STEU (Sparse Token Embedding Unlearning), a breakthrough method for machine unlearning in clinical AI systems. The technique addresses a critical challenge in healthcare: how to remove sensitive patient information from deployed language models when required by privacy regulations like HIPAA, without the prohibitive cost of retraining from scratch. STEU achieves this by updating only selected token embeddings and a small classifier head while keeping all encoder layers frozen, modifying a remarkably sparse 0.19% of model parameters.

In comprehensive testing across MIMIC-IV, MIMIC-III, and eICU datasets using BioClinicalBERT, BERT-base, and DistilBERT models, STEU demonstrated exceptional performance. The method achieved near-complete forgetting of target clinical classes with an F1 score of just 0.0004, while largely preserving model utility on retained tasks with an average F1 of 0.4766. This represents a significant advancement over traditional approaches that typically require full model retraining or extensive parameter modifications.

The research reveals that targeted behavioral unlearning can be accomplished through sparse embedding edits without disturbing deeper encoder representations. This finding has profound implications for clinical AI deployment, where models must balance compliance with evolving privacy requirements against maintaining diagnostic accuracy. The team's approach uses PMI (Pointwise Mutual Information) to identify which token embeddings to modify, creating an efficient pathway for healthcare institutions to manage data deletion requests while preserving overall system performance.