PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference

Researchers have developed PackInfer, a new system that makes AI language models like ChatGPT respond faster when handling many user requests at once. It works by intelligently grouping different-length conversations to better use the computer's power. This reduces wasted effort and memory use. Tests show it cuts response delays by 13-20% and increases the number of requests handled by 20% compared to the current best method, FlashAttention.