Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy

Researchers have introduced 'Spectra,' a new optimizer for training large language models that addresses a fundamental inefficiency. They found that standard optimizers waste effort on dominant 'spike' directions in the data. Spectra specifically targets this, training a LLaMA3 8B model to the same loss 30% faster than AdamW while reducing optimizer state memory by 49.25% and improving downstream accuracy by 1.62%.