FLASH Optimizer Dominates Configurable System Tuning Under Any Budget

A new study published on arXiv tackles a practical question for software engineers: which optimizer should you use for tuning configurable systems when your time budget varies? The researchers evaluated 8 well-established optimizers—including model-based (SMAC, TPOT), model-free (genetic algorithms, simulated annealing), and hybrid approaches—across 22 real-world configurable systems, from databases (PostgreSQL, MariaDB) to web servers and compilers. They systematically tested performance under tight budgets (few evaluations), moderate budgets, and generous budgets (many evaluations).

Unsurprisingly, model-based optimizers like SMAC perform best under tight budgets because they build a surrogate model to guide search efficiently. Model-free optimizers like genetic algorithms (GAs) become superior with more time, as they explore more broadly. But the standout finding is that FLASH, a simple greedy optimizer, consistently ranks near the top for most systems regardless of budget. The reason: many configurable systems have good local optima with large basins of attraction, making greedy local search highly effective. This offers a practical takeaway: engineers can rely on FLASH for reliable tuning without needing to adjust strategy per budget level.