BERT tops leakage-controlled document sensitivity benchmark with 89% F1

A new arXiv paper tackles a silent killer in AI classification: label leakage. Researchers (Zainab et al.) built Strategic 16K, a 16,000-document corpus of diplomatic cables from WikiLeaks PlusD, and systematically removed three categories of residual classification markers embedded in document bodies. These markers previously allowed models to cheat by exploiting surface cues rather than learning true content sensitivity. The result is the first fully reproducible sensitivity classification benchmark with explicit leakage control.

Across six models, BERT led with 89.14% accuracy and 89.33% F1-score, while ELECTRA trailed slightly at 88.57% accuracy. Notably, classical TF-IDF with Logistic Regression delivered competitive performance (the paper reports it as strongest among classical models) at significantly lower computational cost. The authors argue this makes lightweight models practical for budget-constrained organizations that still need reliable classification. The work exposes how prior benchmarks may have overestimated real-world performance, underscoring the need for leakage-controlled evaluation in security and compliance domains.