New proof unifies Dropout and Gradient Masking for deep ResNets

Dropout and Random Gradient Masking (RaM) are two popular but seemingly distinct regularization methods in deep learning. Dropout randomly zeros out activations during the forward pass to prevent co-adaptation, while RaM leaves the forward pass untouched and instead randomly masks gradients during backpropagation. Because RaM introduces unbiased noise into parameter updates, standard explanations for dropout's effectiveness—such as penalization of weights or neuron co-adaptation avoidance—do not apply. This has left researchers puzzled about whether they are fundamentally different or secretly connected.

In a new arXiv paper, Javier Maass and Lénaïc Chizat prove that for ResNets in the large depth and width asymptotics (the complete feature learning regime), Dropout and RaM converge to the same limiting dynamics. Their analysis covers multiple variants, including the layerwise dropout used in stochastic-depth ResNets, and shows that all these methods collapse to the same limit as network scale increases. This theoretical equivalence provides a unifying framework: at sufficient scale, the way you inject randomness matters less than the fact that you inject it. For practitioners, the result means techniques can be chosen based on computational convenience rather than presumed differing effects, and future research can focus on a single effective regularization approach for large models.