New MoEMA Framework Combines Multiple Generative Models Optimally
Researchers achieve input-adaptive averaging of generators with theoretical guarantees.
A new paper from Shijin Gong, Baihua He, and Xinyu Zhang introduces an optimal model averaging framework for conditional generative models. As these models proliferate, practitioners often face multiple plausible generators whose performance varies with task, data, or input condition. The authors propose two methods: StaticMA, which assigns fixed weights to candidate generators, and MoEMA (mixture-of-experts model averaging), which uses a softmax neural-network gate to adapt weights based on input conditions. Both methods leverage a sample-based maximum mean discrepancy (MMD) between conditional distributions, allowing combination even when generators are only accessible via conditional samples without tractable densities.
The paper establishes in-sample and out-of-sample asymptotic optimality for StaticMA and MoEMA, along with consistency of the estimated adaptive weight function under regularity conditions. The framework applies directly to Euclidean responses and extends to unstructured data by combining with fixed representation maps. Across extensive simulations and real-data studies covering tabular, image, and text modalities, MoEMA consistently outperforms competing baselines. This work provides a principled way to dynamically ensemble generative models, adapting per input to maximize quality without requiring density evaluation.
- MoEMA uses input-adaptive weights via a softmax neural-network gate, outperforming static averaging across tasks.
- The framework combines generators using only conditional samples — no tractable densities required.
- Asymptotic optimality and consistency proven; demonstrated gains on tabular, image, and text modalities.
Why It Matters
Enables practitioners to combine best-performing generative models per task without manual tuning.