S-GAI uses spectral geometry to supercharge MLP initialization

Classic universal approximation theorems guarantee sigmoidal MLPs can learn any function, but they offer no guidance on initial weight selection. Yi-Shan Chu's S-GAI (Spectral Geometry-Aware Initialization) directly addresses this gap by encoding the training dataset's geometry into the network's weights. For each class, SVD extracts a mean vector, principal directions, and spectral scales. An energy threshold selects the most important directions, each represented by two sigmoid gates in a shared hidden layer. This creates a hidden state that already reflects the data's spectral structure before any training begins. The paper also introduces an SVD-based subspace classifier as a non-neural reference to verify the discriminative quality of the estimated geometry.

Experiments on MNIST, Fashion-MNIST, and the more challenging CIFAR-10 confirm S-GAI's advantages. Compared to Xavier initialization, S-GAI-initialized MLPs start from a substantially more informative hidden state and reach comparable final accuracy after full training. The real standout result: when the hidden layer is frozen and only the output layer is trained, S-GAI-initialized networks significantly outperform frozen random gates. This provides strong evidence that S-GAI successfully embeds class-wise spectral geometry into the MLP, potentially reducing training time and improving performance in low-data or transfer learning scenarios. The work bridges geometric data analysis and neural network initialization, offering a principled alternative to purely statistical initialization schemes.