Mechanistic estimation for wide random MLPs
A mechanistic approach beats Monte Carlo sampling by up to 7 orders of magnitude in FLOPs.
In a new paper, ARC (Alignment Research Center) researchers present a method to estimate the expected output of a randomly initialized multilayer perceptron (MLP) under Gaussian input without ever running the model. The team, including Wilson Wu, George Robinson, Mike Winer, Victor Lecomte, and Paul Christiano, proposes a mechanistic approach called cumulant propagation that analytically computes the output distribution. For wide networks, their algorithm achieves mean squared error scaling as O(1/width^2) and runs in O(width^2) time, compared to Monte Carlo sampling's O(1/width) MSE and O(width) runtime. This means as width grows, their method provably outperforms sampling—though the depth dependence is worse.
The empirical results are striking: for ReLU MLPs with 4 hidden layers and width 256, the best cumulant propagation algorithms match the MSE of Monte Carlo using fewer than 1/10^7 as many FLOPs across FLOP budgets spanning seven orders of magnitude. The authors view this as a 'base case' in a broader vision to produce mechanistic estimates that beat random sampling for any trained neural network. The inductive step—handling trained networks—remains much harder, but this work establishes a strong foundation by demonstrating that analytically derived estimates can be dramatically more efficient than brute-force sampling for wide models.
- Mechanistic cumulant propagation estimates expected MLP outputs without any forward passes, reducing FLOPs by up to 7 orders of magnitude for width-256 ReLU nets.
- Theoretical guarantee: MSE scales as O(1/width^2) vs Monte Carlo's O(1/width), with runtime O(width^2) vs O(width).
- ARC frames this as the 'base case' toward efficient mechanistic estimation of trained neural networks.
Why It Matters
Fast, accurate output estimation for wide random nets could accelerate neural network analysis and safety research without costly sampling.