New survey unifies LLM data exposure risks: membership inference and contamination

A new survey from Ziyi Tong, Feifei Sun, and Le Minh Nguyen (accepted at NLDB 2025) tackles a growing blind spot in LLM research: Pretraining Data Exposure (PDE). The paper is the first to formally unify two previously siloed areas—data contamination (where test data leaks into training) and membership inference (whether a specific document was used in pretraining). The authors define PDE across multiple exposure levels and map out existing attack techniques, from simple perplexity-based membership tests to advanced loss-based methods. They also catalog defenses like differential privacy, deduplication, and training with censorship.

Beyond taxonomy, the survey synthesizes empirical findings from dozens of prior studies, revealing that even instruction-tuned models can leak training data and that standard benchmarks remain vulnerable to contamination. The researchers identify key open challenges: scaling attacks to trillion-parameter models, developing robust privacy guarantees, and building automated pipelines to detect PDE at training time. For practitioners, the paper offers a practical reference for auditing models before deployment and for designing safer pretraining pipelines. As LLMs ingest ever-larger and more opaque datasets, this unified framework becomes essential reading for anyone building or deploying foundation models responsibly.