Reusing Pretrained Weight Spectra Fails to Boost LM Pretraining Performance

Researchers Konstantin Garbers and Nicholas Oh investigated whether the structured weight spectra commonly observed in pretrained language models could be repurposed as initialization signals to accelerate or improve GPT-2 style pretraining. They analyzed eleven diverse GPT-2 checkpoints—varying in model size, language, tokenizer, and training corpus—measuring Frobenius norm and effective-rank entropy across layers and transformer subcomponents. The analysis confirmed shared depth trends, particularly increasing scale and stronger spectral concentration in residual-writing matrices. Based on these observations, they designed initialization schemes that mimicked the component-wise magnitudes and spectral profiles of pretrained models.

When tested against standard initialization methods, the spectral-mimicking initializers visibly altered the model's structural patterns but failed to produce a corresponding performance advantage. Pretrained-weight reuse (full weight copying) remained competitive, while coarse spectral matching alone proved unreliable. The authors emphasize that pretrained spectra are valuable diagnostics for understanding trained model structure, but effective reuse likely requires preserving richer information than just component-wise scale and singular-value shape. This work highlights the gap between observed patterns and actionable optimization strategies in LLM training.