New Paper: Dynamic Parameterization ≠ Dynamic Inference or Savings
Frozen-controller auditing reveals dynamic models can be 30% slower
A new paper from Zongfei Li, Yuan-yih Shang, and Guozhong Luo challenges a common assumption in machine learning: that input-dependent parameterization (dynamic controllers) implies dynamic inference and computational savings. The authors introduce Frozen-Controller Auditing (FCA), a method that caches a model’s unperturbed coefficient tensor, then replays it with cross-input reassignment, token shuffling, and static profiles. This isolates the true functional dependence of the frozen model without feedback from the controller.
Across seven independently trained 76M FeatureGate Transformers and three 504M models, static layerwise profiles retain 98.70% and 99.43% of the Correct-to-GlobalMean performance gap respectively. Layer identity explains 87%–96% of coefficient variance. Despite the dynamic parameterization, FeatureGate executes every transformer block and is 30.8% slower than a dense baseline. On the public MUDDPythia-1.4B checkpoint, cross-input reassignment and token shuffling increase NLL by 1.9067 and 2.9637, proving strong dependence on content-specific assignment—yet MUDDPythia also executes all blocks. The paper concludes that dynamic parameterization alone does not establish dynamic inference or computational savings, urging separate reporting of coefficient variation, functional dependence, and actual execution.
- FCA isolates frozen-model dependence without controller feedback, revealing that static profiles retain >98% performance.
- Layer identity explains 87–96% of coefficient variance; FeatureGate is 30.8% slower than Dense despite dynamic parameters.
- MUDDPythia-1.4B shows NLL increases of 1.9–2.96 under reassignment, yet still executes all transformer blocks.
Why It Matters
Forces AI teams to separate claims about dynamic models from actual computational savings, preventing overhyped efficiency benchmarks.