TIPEX framework unifies two-tier parallelism to cut multi-agent LLM latency

As multi-agent LLM systems grow in complexity, they demand multiple model invocations and intricate coordination during inference, directly impacting accuracy, latency, and computational cost. Parallel execution offers a path to inference-time efficiency, but until now, the relationship between different parallelism forms remained unclear. The paper models two distinct levels: Replica Parallelism, which explores multiple complete solution paths at the task level, and Structural Parallelism, which enables concurrent execution within a single path through task decomposition.

To unify and coordinate these levels, the authors introduce TIPEX, a controllable execution framework that supports systematic combinations of parallel strategies and parameter configurations under a unified execution semantics. Systematic experiments on the GAIA benchmark show that inference-time parallelism can significantly improve accuracy and reduce end-to-end latency, particularly for tasks of intermediate difficulty, which benefit most from coordinated parallelism. However, the team also warns that aggressive parallel strategies do not necessarily improve performance and may waste tokens, highlighting the need for careful tuning when deploying such systems in production.