Kernel Ridge Regression Achieves Optimal Rates but Fails Properness Test
New proof shows KRR hits minimax L2 rates while noise variance grows logarithmically.
Yuxuan Hou's latest theoretical work dives deep into kernel ridge regression (KRR) when the true regression function belongs to the Hölder-Zygmund class. Using an RKHS equivalent to a Sobolev space of smoothness s+d/2, the paper proves that even under misspecification (where the kernel doesn't exactly match the target smoothness), KRR still achieves the minimax optimal L2 convergence rate of n^{-2s/(2s+d)}. This confirms that KRR is rate-optimal for a broad class of functions.
However, the paper uncovers a troubling failure of properness in the Hölder-Zygmund norm. Properness means the estimated function's norm should not blow up as sample size increases when the true function is zero. Hou shows that with Gaussian noise, the expected squared Hölder-Zygmund norm of the KRR estimate actually grows like log n. This counterintuitive result highlights a fundamental limitation: KRR may overfit in the sense of producing increasingly rough estimates, even when there's no signal to learn. Practitioners should be aware that optimal prediction error does not guarantee stable function estimates.
- Misspecified KRR achieves minimax L2 rate n^{-2s/(2s+d)} for Hölder-Zygmund class functions.
- Properness fails: expected squared Hölder-Zygmund norm of KRR noise component grows as log n.
- RKHS equivalence to Sobolev space of smoothness s+d/2 is key to the theoretical analysis.
Why It Matters
Fundamental insight for ML practitioners: optimal prediction doesn't mean stable function estimates, especially in high-dimensional regression.