Attention Meets Reachability: Structural Equivalence and Efficiency in Grammar-Constrained LLM Decoding

Researchers Faruk Alpay and Bilge Senturk have published a foundational paper, 'Attention Meets Reachability: Structural Equivalence and Efficiency in Grammar-Constrained LLM Decoding,' that dissects the hidden computational costs of forcing AI models to follow grammatical rules. The core finding is their 'oracle invariance theorem,' which proves that different context-free grammars (CFGs) can produce identical, valid outputs from a large language model (LLM) but require vastly different amounts of work. They introduce a 'structural ambiguity cost' (SAC) metric, showing that for equivalent grammars, this cost can be a constant O(1) per token or explode to Θ(t²) per token, leading to Θ(n³) cumulative work—a massive efficiency gap.

The paper establishes engine-independent lower bounds, proving that any sound, parse-preserving decoding system must incur at least Ω(t²) work per token for certain grammar families. This provides a theoretical limit for optimizing tools like JSON mode or code generators. The researchers connect these formal results to practical Transformer and Mixture-of-Experts architectures, deriving latency models based on vocabulary size and beam width. Finally, they characterize the statistical distortion introduced by hard-masking tokens and lay the groundwork for automated grammar optimization, allowing developers to find the minimal-cost grammar for a desired output structure, directly impacting inference speed and cost.