UChicago's 1-year LLM serving trace reveals hidden workload patterns

In a new arXiv preprint, researchers from the University of Chicago — William Nixon, Jon Durbin, Florian Standhartinger, Haryadi S. Gunawi, and Juncheng Yang — present "A Year in LLM Serving," a longitudinal study of production traffic on Chutes, an LLM serving platform. Most existing workload studies capture only short windows or rely on synthetic data, missing the long-term evolution of how users interact with models. This work analyzes a full 12-month trace spanning hundreds of models and thousands of users, including the long tail of less-popular models that are often filtered out. By examining the workload from aggregate, temporal, model-level, and user-level perspectives, the authors reveal how demand shifts over time, how caching behaves under real-world request patterns, and how load-balancing is affected by uneven popularity distributions. The dataset will be released with the paper, allowing other researchers to benchmark serving systems against real production behavior.

The findings have direct implications for infrastructure design. The trace exposes inefficiencies in current caching strategies—popular models may hit cache frequently, but long-tail models suffer from low reuse, leading to high prefill costs. Load-balancing also becomes more challenging as user-model interactions shift over weeks and months, making static routing policies suboptimal. The authors highlight that aggregate metrics hide crucial patterns, and they offer a breakdown of how request arrival rates, token generation, and cache hit ratios vary across different model tiers. For engineers building LLM serving platforms, this work provides a much-needed empirical baseline to evaluate innovations like prefix caching, speculative decoding, and adaptive scheduling. By releasing the full one-year trace, the team aims to accelerate research into production-grade serving systems, moving beyond idealized benchmarks. The paper is available now on arXiv with the associated dataset.