Researchers find minimal neural network decompositions using polyhedron geometry

A new mathematics paper tackles a core problem in neural network theory: representing any continuous piecewise linear (CPWL) function—the building block of ReLU networks—as a difference of two convex CPWL functions with as few linear pieces as possible. The authors (Brandenburg, Grillo, Hertrich, arXiv:2410.04907) first demonstrate that a recently proposed decomposition method by Tran and Wang (2024) is flawed, highlighting the problem's difficulty. They then propose fixing an underlying polyhedral complex that constrains where nonlinearities can occur. Under this constraint, they prove the set of all possible decompositions forms a polyhedron—specifically the intersection of two translated cones—and that minimal decompositions correspond exactly to the bounded faces (vertices) of this polyhedron.

This geometric insight has immediate consequences: the authors identify cases where a unique minimal decomposition exists and relate the framework to submodular function theory. Practically, they improve on previous constructions for neural networks representing convex CPWL functions and extend the results to nonconvex cases. For machine learning, this means potentially more compact neural network architectures that use fewer ReLU activations while maintaining the same expressiveness—a step toward smaller, faster models without sacrificing accuracy.