TensorCast: New tensor layer cuts LLM latency by 93.2%

Large language model infrastructure has long treated tensors as transient computation data, but modern workloads require them to persist as shared state across distributed components. Existing systems optimize tasks like model weight loading, KV cache management, and checkpoint synchronization with tightly coupled mechanisms, creating silos that block reuse and composition. In a new arXiv paper, researchers from Peking University and Microsoft propose Tensor-as-a-Service (TaaS), an abstraction layer that separates tensor state management from execution, and present TensorCast as its reference implementation.

TensorCast provides first-class tensor abstractions with programmable lifecycle primitives and a runtime that decouples management policies from execution mechanisms. Developers write tensor management programs using TensorCast APIs without worrying about distributed data movement. The team integrated TensorCast with vLLM and SGLang, evaluating it across model weight materialization, weight synchronization, KV cache management, and programmable request routing. Results show competitive performance with specialized systems, while a programmable policy cut median time-to-first-token (TTFT) by up to 93.2% under highly concurrent multi-turn agent workloads — demonstrating the power of unified cross-component optimization.