Weibull framework diagnoses transformer weight distributions across 12 models

A new diagnostic approach for transformer weight distributions uses the two-parameter Weibull family from extreme-value theory. Developed by researcher Tiexin Ding, the framework applies probability-plot fitting to each weight matrix independently at every layer and checkpoint, providing per-component, per-layer, and per-step insights that aggregate statistics miss. At initialization, i.i.d. Gaussian weights yield a HalfNormal distribution with shape parameter k ≈ 1.20, establishing a principled baseline for tracking training dynamics across architectures, activations (SwiGLU, GeLU), normalization placements, and model sizes from 70M to 14B parameters.

Applying the framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o (termed Transmission Class) maintain a remarkably narrow k band with median terminal values between 1.186 and 1.204 (cross-family CV = 0.51%). Second, attention input projections W_q, W_k (Selection Class) deviate from the Weibull family, with severity linked to storage structure: separately stored Q/K in OLMo models yield k in [0.76, 0.99], while GQA models show milder deviation (k in [1.10, 1.16]). Third, scale parameter lambda grows during training and scales with sqrt(eta/lambda_wd) (Pearson r = 0.94). The framework treats k and lambda as independent metrics: k identifies functional class, lambda measures training progress. The open-source release includes the npm-weibull-py v0.4 library and DATABASE_v9_1 for further research.