SLIM model finds 55GB memory waste in LLM serving, boosts throughput

A new paper from Pol G. Recasens, Ferran Agullo, and colleagues at UPC Barcelona and IBM Research challenges long-held assumptions about why LLM serving throughput plateaus. The team's GPU hardware profiling shows that the bottleneck isn't simply increasing batch size. Instead, during the decode phase, attention kernels maintain nearly constant arithmetic intensity as active-context lengths grow, driving DRAM-bandwidth saturation—even while compute throughput stays far below the hardware limit. This granular characterization corrects previous conceptual explanations and pinpoints attention as the true culprit.

Building on this insight, the authors introduce SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that derives inference throughput and latency from Transformer computation and memory traffic formulas. SLIM outperforms existing performance-modeling baselines and generalizes to unseen operating conditions. Its companion tool, the Batching Configuration Advisor (BCA), recommends the highest-throughput batching config that satisfies latency constraints—identifying up to 55GB of GPU memory allocations that can be avoided on evaluated OPT models with negligible throughput loss. For teams running LLM serving infrastructure, this means more efficient utilization, lower costs, and better performance without re-architecting models.