Free AI Chat Software Gets Faster and More Efficient

llama.cpp has a new change aimed at more efficient unified KV cache handling for multiple sequences. The commit, titled "flash_attn_ext_rows - more efficient unified kv for multi-seq," reworks how the KV row lists are built: each sequence keeps a bitmap over the cells, updated in seq_pos_inc/dec, and the lists are built by walking its set bits.

The lists stay in cell order, cells shared by several sequences go to every list, and nothing has to be rebuilt when cells are removed. Cells that SWA hides from every token of the slice are left out, for both standard and chunked SWA. The change also makes get_kv_rows_p_min private and uses kv_rows for every ubatch of a unified cache: a ubatch made of equal-length runs with a single sequence each splits into per-sequence slices, while any other ubatch becomes a single slice over the cells of all its sequences instead of falling back to the dense KQ mask, so the graph keeps the topology of the reserved graph.