L2A: New framework lets LLMs adapt to dynamic cloud resources

Large Language Model inference is usually deployed under static resource assumptions, executing a fixed computational graph regardless of runtime conditions. But real-world cloud infrastructure is inherently dynamic—fluctuating availability (e.g., spot instance preemption) and tiered Quality-of-Service requirements mean static models either crash under tight budgets or waste compute when resources are ample.

L2A solves this by formulating inference as a constrained allocation problem conditioned on both the input and the runtime resource budget. It introduces lightweight, budget-conditioned, and input-aware gating networks that learn to skip layers (for memory/depth pressure), prune attention heads (for throughput contention), and reduce reasoning tokens (for latency tightening). The model is trained via a unified objective optimizing task performance, logical consistency, and resource costs. On Llama-3-8B and Qwen-3-4B, a single L2A model traces the entire compute-accuracy Pareto frontier: at 34% layer sparsity, it stays within 0.6% of the dense baseline on GSM8K—zero-shot on out-of-distribution tasks—while static baselines require separately tuned models and drop 5–10% at comparable inference times.