RAC codec speeds split LLM inference by up to 2.79x with minimal loss

Split LLM inference runs the head, tail, and tools locally while middle layers execute in the cloud, offering a balance between privacy and cost. But each invocation transfers boundary hidden states over the network, creating a severe communication bottleneck. A new paper presents RAC (Reference-Aware Activation Compression), a codec that compresses these activations by exploiting repetition in long, privacy-sensitive contexts. RAC retrieves exact-token historical spans for prefill uplinks, reuses reconstructed uplink states for same-round prefill downlinks, and generates boundary-specific decode references with lightweight causal predictors.

RAC applies grouped affine alignment and calibrated residual quantization with optional prefill outliers, and uses sender-side wire-format reconstruction to keep references synchronized across devices. Offline calibration accounts for quality- and packed-representation costs. In evaluations across three models and nine model-link pairs, RAC improved mean time-to-first-token (TTFT) by 1.24-2.72x and time-per-output-token (TPOT) by 1.01-2.79x compared to raw split inference. Quality remained stable: 12 non-perplexity task-score changes ranged from -0.40 to +2.50 points, showing the compression preserves task performance while substantially cutting latency. This makes split inference more viable for privacy-conscious, communication-constrained deployments.