Precise Performance of Linear Denoisers in the Proportional Regime
A new training method for linear denoisers beats the empirical Wiener filter, closing in on optimal performance.
A team of researchers has published a paper introducing a novel, analytically precise method for training linear denoisers, a core component in systems like diffusion models. The work, "Precise Performance of Linear Denoisers in the Proportional Regime," addresses a classic signal processing problem: cleaning up noisy data (x + z) when the true data's statistical structure (covariance Σ) is unknown. The standard workaround is to estimate Σ from samples and build an "empirical" Wiener filter, but this new approach takes a different, more data-driven path.
Instead of estimating the covariance, the method synthetically constructs training data by injecting clean samples with a deliberately chosen, *different* Gaussian noise (with covariance Σ₁ ≠ Σz). It then trains a linear transformation matrix W by solving a least-squares problem to map these noisy samples back to the clean originals. The key theoretical breakthrough uses the Convex Gaussian Min-Max Theorem (CGMT) to derive a closed-form, analytical expression for the denoiser's generalization error in the "proportional regime," where the number of samples (n) and data dimensions (d) grow large together at a fixed ratio κ.
This analytical expression is powerful because it allows researchers to optimize the choice of the synthetic training noise Σ₁ to minimize the final error. Numerical simulations confirm that denoisers trained with this optimized method consistently outperform the traditional empirical Wiener filter across many scenarios. Furthermore, as the data ratio κ increases (more samples per dimension), the performance of this new denoiser converges to that of the optimal Wiener filter—the theoretical gold standard that requires perfect knowledge of the true covariance.
- Proposes a data-driven method to train linear denoisers by injecting synthetic noise different from the target noise and solving a least-squares problem.
- Uses the Convex Gaussian Min-Max Theorem (CGMT) to derive a closed-form expression for generalization error in the proportional regime (n/d → κ).
- The optimized denoiser outperforms the standard empirical Wiener filter and approaches optimal performance as data samples increase (κ → ∞).
Why It Matters
Provides a more robust and analytically grounded method for a fundamental AI building block, potentially improving performance in diffusion models and other generative AI systems.