LBW-Guard reduces LLM training perplexity by 18.7% with 1.1x speedup
A governance layer above AdamW keeps models stable even under aggressive learning rates.
Modern large language model training is notoriously fragile—aggressive learning rates, scale, or runtime stress often lead to instability, wasted compute, and failed runs. In a new paper, Anis Radianis proposes **Learn-by-Wire Guard (LBW-Guard)**, a governance layer that sits above the AdamW optimizer. Rather than modifying the update rule itself, LBW-Guard observes training telemetry, detects instability-sensitive regimes, and applies bounded control signals to the optimizer's execution while preserving fixed training objectives. This approach treats training stability as a control problem, not a gradient problem.
Evaluated on a Qwen2.5-centered stress suite using WikiText-103, LBW-Guard delivers striking results. On the 7B reference model, final perplexity drops from 13.21 to 10.74 (an 18.7% improvement), and end-to-end training time shrinks from 392.54s to 357.02s (a 1.10x speedup). Under extreme learning-rate stress (LR=3e-3), standard AdamW training collapses to a perplexity of 1885.24, while LBW-Guard remains stable at 11.57. Gradient-clipping baselines fail to reproduce this effect, confirming that the governance plane above the optimizer offers unique benefits. The paper also validates across model sizes (3B, 7B, 14B) and with a full-parameter TinyLlama-1B sanity check.
- LBW-Guard reduces Qwen2.5-7B perplexity from 13.21 to 10.74 (18.7% improvement) and speeds training by 1.10x.
- Under LR=3e-3 stress, AdamW degrades to 1885.24 perplexity while LBW-Guard stays at 11.57—keeping training viable.
- The system sits above AdamW, not replacing it; gradient-clipping baselines do not reproduce the same stability gains.
Why It Matters
Saves massive compute by preventing training crashes, enabling bolder hyperparameter exploration for large language models.