NTK fails exponentially on compositional learning tasks, new paper proves
Trained networks beat NTK by 4–6 orders of magnitude on sparse parity.
A new paper from Arkaprabha Ganguli and Emil Constantinescu (arXiv:2607.06382) provides a rigorous explanation for a long-standing empirical observation: trained neural networks far outperform their Neural Tangent Kernel (NTK) limit on tasks with compositional structure. The authors introduce a dichotomy between two complexity measures: Fourier complexity, which governs NTK kernel regression, and architectural complexity, which controls learning over depth-L, width-w ReLU networks with bounded weight variation. They characterize the minimax rate for the architecture class, pinning it down to between Ω(Lw²R²/n) and Õ(L²w²R²/n), differing by only a factor of L.
Crucially, when these two complexities decouple—as they do for compositional targets like the depth-L iterated sawtooth—the NTK estimator sits exponentially above this floor: NTK regression needs Ω(4^L) samples while the minimax floor is polynomial in L. Numerical experiments confirm the theory: on bandlimited smooth targets the NTK is competitive, but on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The result shows the gap is a function-space property—a mismatch between the kernel's smoothness bias and the target's compositional structure—not a generic kernel-versus-network phenomenon.
- NTK requires Ω(4^L) samples for depth-L iterated sawtooth, while optimal networks need only polynomial samples.
- Two-layer ReLU network beats NTK by 4–6 orders of magnitude on hypercube sparse-parity test error.
- Gap is due to mismatch between kernel's smoothness bias and target's compositional structure, not inherent kernel inferiority.
Why It Matters
Explains why trained nets outperform NTK on compositional tasks, guiding model selection for structured data problems.