New SMD-based estimator scales mixture models to large component sets efficiently

A new paper from Mohammadreza Ahmadypour, Tara Javidi, and Farinaz Koushanfar revisits the classic problem of estimating an unknown distribution by fitting a mixture model via cross-entropy minimization. They frame it as stochastic convex optimization over M-component mixture distributions and propose estimators derived from stochastic mirror descent (SMD). By choosing different Bregman divergences, their φ-SMD framework generalizes traditional estimators and enables novel ones. The key breakthrough is that the method's computational cost does not blow up with the number of candidate components f_i, allowing practitioners to use very large basis distribution libraries.

For categorical distributions (discrete outcomes), the estimator does not require precise knowledge of the support, eliminating a restrictive assumption. Under mild conditions, φ-SMD achieves near-optimal convergence rates in both Kullback-Leibler divergence and ℓ₂-norm, outperforming classical estimators in sample efficiency and scalability. Numerical experiments highlight the practical benefits when computation is expensive. This work bridges optimization theory and statistical estimation, offering a principled, flexible toolkit for density estimation tasks.