DynaSchedBench reveals LLMs' paradox: more info, worse scheduling
Providing LLM agents with complete operational data for dynamic scheduling tasks actually degrades their performance—a counterintuitive finding that upends the 'more is better' assumption in AI reasoning.
A new paper from researchers Shijie Cao, Yuan Yuan, and Jing Liu tackles the methodological tension in Dynamic Flexible Job Shop Scheduling (DFJSP). They propose DynaSchedBench, a calibrated benchmarking framework that replaces static benchmarks and uncalibrated generators. At its core is the Sequential Event-Space Calibrator (SESC), which computes a Schedule Stress Index (SSI) to stratify scheduling instances by difficulty. SESC is shown to be computationally more efficient than evolutionary baselines while reliably converging to target metrics. The framework integrates modular components for instance generation, snapshot-based simulation, agents, evaluation, and visualization, enabling rigorous testing of reactive and lookahead-based policies.
Using DynaSchedBench, the authors identify key limitations of LLM-based scheduling agents. Most strikingly, they uncover an “Observability Paradox”: providing LLM agents with oracle access to full structural information actually degrades policy performance compared to providing concise, limited information. Furthermore, tool-augmented and refinement strategies, despite incurring substantial token overhead, fail to reliably improve performance. Most LLM agents cannot consistently outperform strong dispatching baselines, behaving more like robust heuristic approximators than superior optimizers. This suggests that current LLM approaches for dynamic scheduling may be overhyped and require fundamentally different architectures or training paradigms.
- LLMs perform best with partial, streamlined data in scheduling tasks, directly contradicting the assumption that more information improves reasoning.
- The Schedule Stress Index (SSI) provides a new, systematic way to stratify scheduling complexity for LLM evaluation, aiding in better benchmarking.
- For the $5B industrial AI scheduling market, hybrid approaches that pair LLM heuristics with traditional solvers like OR-Tools remain superior to LLM-only solutions.
- The Observability Paradox may not generalise across all scheduling domains or LLM architectures; fine-tuning could resolve it, but that remains unexplored.
Why It Matters
For AI agents in operations, less information can be more effective—a critical lesson for designing real-world decision systems.