Researchers launch BBOWP-Bench to test LLMs in black-box optimization

A team of researchers from Japan has introduced BBOWP-Bench (Black-Box Optimization Word Problems Benchmark), a novel evaluation framework designed to test large language models (LLMs) on their ability to tackle black-box optimization (BBO) problems. Unlike traditional optimization benchmarks that rely on explicit mathematical expressions, BBOWP-Bench focuses on scenarios where only objective values are observable, and the functional form remains hidden. This mirrors real-world challenges in fields like hyperparameter tuning, drug discovery, or supply chain optimization, where practitioners must design both the search space and the optimization strategy based solely on descriptive problem statements.

The benchmark consists of natural-language problem descriptions paired with executable evaluation environments and human-designed baselines. In their evaluation of current LLMs, the researchers found that while models could successfully select appropriate optimization algorithms based on evaluation budgets, they often struggled with search space design—particularly when identifying critical variables or balancing their ranges in ambiguous or highly specialized problem contexts. The code and dataset are available at the provided URL, offering researchers a new tool to probe LLMs' capabilities in this underexplored area of AI-driven optimization.