The Implicit Bias of Logit Regularization

A new arXiv paper analyzes logit regularization—adding convex penalties directly in logit space, as used in label smoothing—and reveals its implicit bias toward clustering logits around finite targets. For Gaussian data, this drives weight vectors to align with Fisher's Linear Discriminant. In a signal-plus-noise model, the technique halves the critical sample complexity, induces grokking in small-noise limits, and makes generalization robust to noise, extending theoretical understanding of these widely used methods.