New Theory Reveals How Scattering Networks Separate Data in CNNs

Researchers Konstantin Häberle and Helmut Bölcskei have published a paper titled 'Separation Capacity of Scattering Networks' that attempts to enhance the theoretical understanding of convolutional neural networks (CNNs) as feature extractors in classification tasks. The authors analyze CNNs through the lens of Cover's function-counting theory, focusing on the notion of separation capacity—a combinatorial quantity derived from counting the number of realizable dichotomies (binary label assignments) a model can separate. Their contributions are threefold: first, they extend Cover's framework with a conceptually insightful and practically useful formulation for separation capacity; second, they leverage this formulation to identify the factors governing the separation capacity of scattering networks in terms of their building blocks; and third, they provide practical insights for network design. The 36-page paper includes 10 figures and spans multiple disciplines including machine learning, information theory, and complex variables.

This work is significant because it moves beyond empirical observations of CNN performance to provide a rigorous mathematical basis for why certain architectures work well as feature extractors. By quantifying separation capacity, engineers can now make more informed decisions about network width, depth, and connectivity patterns for scattering networks—a specific class of CNNs with fixed filters that are particularly useful for invariant feature extraction. The findings offer a tangible way to predict and improve classification accuracy without exhaustive hyperparameter tuning. For professionals building systems that rely on CNNs for image or signal classification, this theory could lead to more efficient and reliable models that are easier to design and explain.