New study reveals significant memory aging in GPU-based LLM servers
Researchers ran 216-hour tests on 6 LLM deployments — all showed memory leaks.
Traditional software aging studies focus on CPU-centric systems with predictable workloads. But LLM serving is fundamentally different: it spans a Python host and a CUDA device, handles requests with cost variations of orders of magnitude, and relies on rapidly evolving software stacks. To bridge this gap, Domenico Cotroneo and Bojan Cukic designed a rigorous empirical methodology. They ran a 216-hour experimental campaign with six co-located LLM deployments under identical stress conditions, monitoring host, device, and client metrics in parallel. A sophisticated statistical pipeline accounted for autocorrelation and multiple testing.
The results are stark: every deployment showed statistically significant memory aging, with leak rates heavily dependent on the serving runtime (e.g., vLLM, TGI) and configuration details like batching and scheduling. This means that production systems may gradually degrade in performance over time, potentially causing unexpected failures or increased operational costs. The study not only identifies the problem but also provides a reproducible framework for future research. It opens a new research direction at the intersection of software aging and rejuvenation and LLM serving — a critical area as more organizations deploy LLMs at scale.
- 216-hour experiment across 6 co-located GPU-based LLM deployments showed memory leaks in all cases.
- Leak rates vary significantly depending on the serving runtime (e.g., vLLM vs. TGI) and configuration.
- The study introduces a reproducible statistical methodology that accounts for autocorrelation and multiple comparisons.
Why It Matters
Memory leaks in production LLM servers can silently degrade performance and inflate costs over time.