llama.cpp b10369 cuts TTS frame generation by 80% on CUDA
New pocket-tts backend hits 0.999994 output correlation while speeding up generation dramatically
ggml-org's llama.cpp released b10369, bringing native support for pocket-tts through the mtmd architecture. The key optimization rewrites transposed convolutions from per-channel convolutions and concatenations — which flooded the graph with small nodes and made kernel launches dominate the decoder — into a column form: the general case reshapes the kernel and performs a single matmul, while the depthwise case batches a matmul over channels. A col2im_1d operation then scatter-adds columns back to the signal, preserving the overlap-add tail, streaming state, and bias.
The performance impact is substantial: generation time per frame drops 80% on CUDA and 50% on CPU, with output matching the previous implementation sample-for-sample (correlation 0.999994, same frame counts). The release also adds per-language-pack generation settings — like frames_after_eos and pad_short_text — carried in the mmproj, and maps semicolons to commas across all packs. A long French text now lands within 2% of the reference timing (22.96s vs 23.44s) with identical peak levels and silence periods. Note that existing mmproj files must be reconverted to include the new configuration keys.
- Rewrites transposed convolutions as GEMM + col2im, eliminating graph flooding from per-channel ops
- 80% faster generation per frame on CUDA, 50% faster on CPU, with 0.999994 output correlation to previous version
- Adds language-pack tuning (frames_after_eos, pad_short_text) and requires mmproj reconversion
Why It Matters
Near-real-time local TTS on consumer hardware is now feasible, enabling responsive voice agents and on-device accessibility tools.