Internal noise in deep neural networks: interplay of depth, neuron number, and noise injection step
Injecting noise before a neuron's activation function boosts accuracy, acting as a nonlinear filter.
A new study by researchers D.A. Maksimov, V.M. Moskvitin, and N. Semenova provides a detailed blueprint for using internal noise to improve deep neural networks. The paper, "Internal noise in deep neural networks: interplay of depth, neuron number, and noise injection step," systematically analyzes where and how to introduce Gaussian noise. The core finding is that the placement of noise relative to a neuron's activation function is critical. Networks where noise is injected *before* the activation consistently achieve higher accuracy than those where it is injected after. The activation function itself acts as a powerful nonlinear filter, suppressing this pre-activation noise more effectively.
The research, inspired by the behavior of analog neural networks, examines both additive and multiplicative noise. For noise introduced *after* the activation, the study found multiplicative noise is less harmful than additive noise. It also revealed that noise in earlier hidden layers causes more significant performance degradation due to cumulative amplification through subsequent weight matrices. Importantly, the authors demonstrated that pooling-based noise reduction techniques are effective regardless of injection point, offering a reliable method to consistently boost network performance. This work moves beyond simply adding noise for regularization, providing a principled framework for its strategic application.
- Noise injected *before* a neuron's activation function yields higher accuracy than noise injected after, as the activation acts as a nonlinear filter.
- For noise introduced *after* activation, multiplicative noise is less detrimental than additive noise, and noise in earlier layers degrades performance more due to cumulative amplification.
- Pooling-based noise reduction was shown to be effective in both injection scenarios, consistently improving overall network performance.
Why It Matters
Provides a clear, actionable strategy for engineers to improve model robustness and training stability through optimized noise injection.