Deep Belief Networks spontaneously cluster data by class with no labels

In a new arXiv preprint (2608.05996, q-bio.NC), Patrick Krauss and colleagues at Friedrich-Alexander-Universität Erlangen-Nürnberg, University Hospital Erlangen, and University of Würzburg investigate whether Deep Belief Networks (DBNs) learn class structure without ever seeing labels. Training DBNs on MNIST, Fashion-MNIST, and KMNIST, they analyze successive hidden layers using the Generalized Discrimination Value (GDV), supervised probes applied post-training, a reconstruction-based abstraction distance, effective dimensionality, and free sample generation. Remarkably, class-specific clustering consistently increases with network depth across all three datasets and various network widths, even though the training is purely unsupervised and label-free.

The team ran extensive control experiments to rule out trivial explanations. The effect persists when accounting for random transformations, weight marginals, dimensionality reduction, and sigmoid saturation, indicating it depends on the learned feature structure—not artifacts. Interestingly, the first hidden layers often make class identity more accessible to linear and nonlinear probes, while deeper layers become more compact and prototype-like, with neurons acquiring correlated feature directions. The authors also show that GDV and probe accuracy capture complementary aspects of class structure: improved average clustering can coexist with reduced accessibility for a few difficult class pairs. These findings suggest that hierarchical generative models can spontaneously discover and progressively amplify class-related organization in unlabeled data, offering fresh insight into unsupervised representation learning and potentially informing more efficient label-free AI systems.