Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

Researchers have derived optimal schedules for adjusting an AI's learning rate during training. They found a sharp phase transition: for easier tasks, the rate should follow a power decay to zero. For harder tasks, a 'warmup-stable-decay' pattern is best, keeping the rate high for most of training before a final drop. This framework, validated on large language models, provides a principled way to evaluate common schedules like cosine decay.