New Surrogate Benchmarks Slash Model Merging Optimization Costs
Tuning model merging hyperparameters just got 100x cheaper with surrogate prediction.
Model merging combines multiple models into one, but its hyperparameters are costly to tune—especially for large language models (LLMs). A new paper from Japanese researchers (Akizuki et al.), presented at AutoML 2025, tackles this by creating surrogate benchmarks that predict the performance of merged models from hyperparameters alone. They define two search spaces and collect data to train these surrogates, which simulate optimization algorithm behaviors with negligible compute.
The benchmarks achieve high prediction accuracy and allow researchers to compare hyperparameter optimization algorithms without repeated expensive evaluations. The code is publicly available on GitHub, lowering the barrier for further innovation. This work directly addresses the computational bottleneck in model merging, making it feasible to explore more merging strategies for LLMs and other large architectures.
- Reduces the computational cost of hyperparameter tuning for model merging by using surrogate models to predict performance.
- Defines two distinct search spaces for optimization, enabling systematic comparison of algorithms.
- Code is open-source on GitHub, allowing easy reproduction and extension by the community.
Why It Matters
Makes model merging optimization practical for LLMs, accelerating multi-model integration without massive compute budgets.