Conditioned DFA method boosts neural net training accuracy by 40%

Direct feedback alignment (DFA) is an alternative to backpropagation (BP) that trains hidden layers using fixed random projections of the output error, avoiding the expensive transposed-weight backward pass. However, DFA often underperforms BP, and the reasons have been unclear. In a new paper, Houman Safaai, Varun Reddy, and Bernardo L. Sabatini identify a failure mode distinct from feedback quality: the local weight update is computed as an outer product, so anisotropy can enter through either the presynaptic activity factor or the local error factor. Using controlled synthetic regimes, they isolate the activity factor and find that task-irrelevant high-variance noise can degrade training by up to 40 percentage points. Conditioning on activity geometry recovers this loss.

To solve the problem, the authors propose a family of normalized DFA (nDFA) methods. Activity nDFA preconditions the update by an inverse of the activity second moment, error nDFA uses an inverse of the local-error second moment, and K-nDFA combines both with separately tuned damping. On tanh/one-vs-rest MNIST and preregistered Fashion-MNIST, error conditioning improves raw DFA by 1.77–7.53 percentage points, and adding activity conditioning yields an extra 0.4–0.9 points. Results replicate across eight fresh seeds on a ReLU/softmax MNIST model. The authors note that the error factor is fragile when under-damped, BatchNorm serves as a strong activity-side alternative, and convnet gains remain partial. The work frames conditioned DFA as a factor-level study of when local outer-product rules fail, not as a general replacement for BP.