SK Hynix's ITME uses CXL memory to boost LLM inference by 35.7%

As large language models move toward agentic and long-context workloads, the industry faces a critical memory bottleneck: TB-scale context states that exceed individual server capacity. Researchers from SK Hynix propose ITME (Inference Tiered Memory Expansion), a disaggregated memory architecture that uses CXL-hybrid memories to deliver massive, byte-addressable remote memory expansion. Unlike traditional offloading to DPU-based JBOF (just-a-bunch-of-flash) architectures, which require complex software optimization, ITME leverages CXL's direct byte-addressability to simplify the stack and enable cost-efficient scaling. The key insight: model weights and prefix caches have deterministic access patterns, allowing the system to proactively manage data movement across the memory-storage hierarchy.

ITME was validated using production-grade SK Hynix CMM (CXL Memory Module) and PCIe Gen5 NVMe SSDs, with functional feasibility confirmed via an FPGA-based hardware prototype. The system accommodates large KV cache footprints beyond host memory limits, achieving up to a 35.7% throughput improvement over conventional CPU-offloading methods. This work, published on arXiv (2606.12556), represents a significant step toward practical disaggregated shared context layers for distributed LLM inference clusters, promising better cost efficiency and simpler deployment for next-generation AI workloads.