TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly

A team of researchers has introduced TTQ (Test-Time Quantization), a novel framework designed to accelerate large language model (LLM) inference by compressing models on the fly. The method, detailed in a new arXiv paper (2603.19296), addresses a critical bottleneck in deploying foundation models: their massive computational demands. Traditional activation-aware compression techniques rely heavily on pre-collected calibration data, which can lead to performance degradation—known as domain shift—when models encounter new, unseen tasks. TTQ circumvents this by performing quantization dynamically at inference time, adapting its compression strategy to each individual prompt without requiring any retraining of the original model.

The core innovation is an efficient online calibration process that analyzes model activations in real-time. This allows TTQ to tailor its quantization parameters specifically for the current input, making it robust across diverse downstream applications. The researchers report that their method not only resolves the domain shift issue but also delivers tangible inference speedups. In their experiments, TTQ demonstrated improved quantization performance compared to existing state-of-the-art baselines, validating its practical utility. This work represents a significant step toward more flexible and efficient deployment of large AI models, moving beyond static, one-size-fits-all compression.