DynaSchedBench reveals LLMs' paradox: more info, worse scheduling

A new diagnostic framework called DynaSchedBench has revealed a paradox at the heart of LLM-based scheduling: giving agents full structural information about a dynamic job shop problem leads to worse outcomes than using concise, streamlined data. The framework, which includes a Sequential Event-Space Calibrator (SESC) and a Schedule Stress Index (SSI), systematically tests LLMs like GPT-3.5/4 and LLaMA on the Dynamic Flexible Job Shop Problem (DFJSP). The core finding is that LLMs behave as robust heuristic approximators, not superior optimizers—they thrive on summarised signals but choke on complete data. This challenges the assumption, widespread since 'LLMs as Optimizers' (Zhang et al., 2023), that more context always improves reasoning.

Traditional solvers like Google OR-Tools and IBM ILOG CPLEX continue to dominate structured scheduling. OR-Tools provides exact and heuristic solutions via CP-SAT, while CPLEX offers deterministic performance in industrial settings. Nextmv takes a hybrid approach, integrating machine learning with operations research. DynaSchedBench shows that LLM-only approaches cannot match these solvers under full-information access. Even tool-augmentation and chain-of-thought refinement fail to reliably close the gap. The industrial AI scheduling market is projected to exceed $5 billion by 2028 (Grand View Research), but this research suggests that LLM-based solutions may only be viable in low-stress, partial-information scenarios—a significant limitation for vendors like C3.ai and Siemens.

The implications cut two ways. On one hand, the Observability Paradox forces a rethink of input design for agentic tasks: concise representations may extract the best from current LLMs. On the other, the study has limitations—it evaluates only a few LLM families on one scheduling variant, and the SSI metric may not capture all difficulty dimensions. There is a risk that practitioners overgeneralise and abandon promising LLM-based scheduling approaches prematurely. Fine-tuning or domain-specific pre-training could mitigate the paradox, but that remains untested. For now, the lesson is clear: LLMs are lightweight approximators for low-stress scheduling, not replacements for traditional solvers. Companies building AI scheduling modules should invest in hybrid architectures that combine LLM heuristics with classical optimisation.

The bottom line is that the very trait we prize in LLMs—their ability to handle vast context—becomes a liability when full structural information overloads their reasoning. The Observability Paradox may be a temporary artifact of current architectures, but it serves as a crucial warning for the next generation of agentic systems: sometimes less really is more.