trunk/2403cc84b033edaaf82f3e5d920555e61929a4fd: [varlen_attn for inference] add seqused_k (#175897)

The PyTorch development team, led by engineers from Meta, has merged a significant optimization into the framework's core attention mechanism. Pull request #175897, authored by contributor liangel-02 and approved by drisspg, introduces a new 'seqused_k' parameter to the variable-length attention (varlen_attn) subsystem used for inference. This change specifically targets the FlashAttention-2 (FA2) implementation, a state-of-the-art algorithm for accelerating transformer models. The update allows external callers to more efficiently manage Key-Value (KV) caches, which are essential for maintaining context in large language models during text generation. By explicitly marking which tokens in a pre-allocated buffer are valid, the system avoids processing stale or padding data, streamlining the attention computation.

Technically, this optimization addresses a fundamental inefficiency in how models handle cached sequences. During inference, models like Llama 3 or GPT-4 store previous token representations (K/V) to avoid recomputation. The new seqused_k parameter gives precise control over this cache, letting the attention mechanism know exactly where new tokens begin and cached tokens end within a shared memory buffer. This reduces redundant calculations and memory bandwidth usage, which is crucial for deployment on constrained hardware or for achieving higher throughput in production systems. For AI engineers, this means potentially faster response times and lower latency in applications ranging from chatbots to code assistants, all through a backend framework update that requires minimal code changes on their part.