GLM-5 inference tuning cuts cost 10% for long-context agent workloads

A new study on arXiv (arXiv:2607.02518) from Minjie Hua and team tackles inference optimization for long-context agent workloads, specifically targeting GLM-5 model serving in a MaaS (Model-as-a-Service) architecture called OpenClaw. These workloads are dominated by long prefixes (system prompts, conversation history, tool outputs) averaging 28k-30k input tokens and only 500 output tokens per request. Traditional short-prompt metrics are inadequate; instead, throughput, time-to-first-token (TTFT), and tail latency are critical. The team systematically tuned chunked prefill size, tensor parallelism (TP), pipeline parallelism (PP), and request concurrency within a two-node, sixteen-GPU cluster.

The optimal configuration found was chunked-prefill-size=3072, tp=4, pp-size=4, and max-running-requests=24. Compared to a conservative baseline (2048/4/4/16), this boosted request throughput from 0.43 to 0.48 req/s and total token throughput from 9029.64 to 9993.23 tok/s. Average TTFT dropped from 8.98s to 6.69s, and P90 latency fell from 40.23s to 32.64s. Cost per request decreased 10.4% and cost per token 9.6%. Importantly, the authors note that larger chunk sizes and deeper queuing do not always help—the optimum is workload-specific. They recommend this profile as the default for OpenClaw deployments.