GLM-5.2 hits 24 tok/s on 4x DGX Sparks after MTP bug fix
A single missing config line was crippling multi-token prediction; now fixed for 24 tok/s at full 128K context.
A critical bug in vLLM's speculative decoding implementation was silently throttling the performance of GLM-5.2 NVFP4 running on four Nvidia DGX Spark nodes. The issue traced to SpeculativeConfig.create_draft_parallel_config(), which failed to copy the decode_context_parallel_size field from the target model's config. This caused the draft model's attention layer to assume no context parallelism (DCP=1) even though the rest of the KV cache and indexing were properly sharded across four ranks. The result: three out of four ranks would select nothing during attention and emit all-zero outputs for 48 of 64 heads, crippling multi-token prediction (MTP) acceptance rates.
Once the missing config line was added and the stack rebased on a newer upstream branch, MTP3 and MTP4 acceptance rates jumped dramatically—from ~0.74 per position to 0.90/0.79/0.67 for MTP3, and MTP4 still conditionally accepting at ~0.84. This translates to a steady 22-24 tok/s at full 128K context, compared to the previous best of ~15 tok/s. One config gotcha remains: MAX_CUDAGRAPH_CAPTURE_SIZE must be set to at least num_speculative_tokens + 2 (e.g., 10 for MTP4) to avoid startup failures. The entire gain came from plumbing, not model changes, making the earlier DCP1/32K compromise config completely obsolete.
- Bug: missing decode_context_parallel_size in draft parallel config caused three of four ranks to produce all-zero attention outputs under DCP4.
- After fix: MTP3 acceptance ~0.90/0.79/0.67, MTP4 ~0.84; throughput jumps from 15 tok/s to 24 tok/s at 128K context.
- Requires MAX_CUDAGRAPH_CAPTURE_SIZE = 10 for MTP4; all gains from a single config change, no model retraining.
Why It Matters
Eliminates the context-length vs. speed tradeoff, enabling high-throughput long-context inference on commodity multi-node setups.