Preventing overfitting in deep learning using differential privacy

A foundational 2017 master's thesis by Alizishaan Khatri, archived on arXiv, presents a novel application of differential privacy (DP) to solve a core machine learning problem: overfitting. While DNNs excel at learning complex patterns, this strength becomes a weakness with limited data, as models can "memorize" training set noise, harming performance on new data. Khatri's work proposes using the mathematical framework of DP, which typically adds noise to protect individual data points in analyses, as a regularizer during neural network training. The injected noise prevents the model from fitting too closely to the specific training examples, forcing it to learn more robust, generalizable features.

This research, formally titled "Preventing overfitting in deep learning using differential privacy," bridges two critical fields in AI: model performance and data governance. By framing overfitting as a privacy issue—where a model should not reveal too much about any single training point—it offers a principled, mathematically-grounded alternative to common regularization techniques like dropout or weight decay. The approach is particularly relevant for practical scenarios where analysts have constrained datasets but need models that perform reliably in production. Although a conceptual exploration from 2017, its principles anticipate later industry trends, where techniques like DP-SGD (Differentially Private Stochastic Gradient Descent) are now used to train more robust and privacy-compliant models.