New attack leaks exact training data from LLMs like Pythia-6.9B
83 documents reproduced verbatim from Pythia-6.9B's training data—16.6% hit rate
Most privacy audits of language models rely on a single ROC-AUC score—and that's dangerously misleading. In Leak It, Victor Maricato shows that blind, model-free baselines already separate members from non-members on surface text alone: on WikiMIA, a bag-of-words classifier hits AUC 0.97, and sampling-based attacks add nothing. On an IID Pile split, the proposed probabilistic, black-box method also fails to beat the blind baseline in aggregate. Yet aggregate metrics hide the real harm. The method verbatim-extracts training data for a tail of documents no blind attack can reach: on Pythia-6.9B, 83 of 500 Pile documents bearing a real identifier (16.6%; 21.3% of those with an email address
- New attack 'Leak It' extracts verbatim training data from black-box LLMs like Pythia-6.9B with 16.6% accuracy on Pile documents
- Leakage is undetectable by aggregate metrics (ROC-AUC) but grows with model size: 5.6% → 16.6% from 410M → 6.9B parameters
- Code data is 3x more vulnerable to identifier leakage than prose; a 16-token prefix suffices and common defenses fail
Why It Matters
Puts real-world data privacy at risk—leaked emails and code could violate GDPR or expose proprietary info.