ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

A research team led by Hao Kang and collaborators from institutions including Carnegie Mellon University and Stanford has introduced ThunderAgent, a new inference system designed specifically for complex AI agents. Current systems like vLLM handle LLM inference and tools like Kubernetes manage external resources, but they operate in isolation. This leads to inefficient management of the KV cache (memory storing conversation history) and tool environments, slowing down multi-step agentic workflows. ThunderAgent solves this by introducing a novel abstraction: the LLM Program. This unified view allows the system to see the entire workflow—including all planned LLM calls, tool uses, and data dependencies—ahead of time.

Built on this foundation, ThunderAgent features a program-aware scheduler and a tool resource manager. The scheduler optimizes the order of operations to maximize KV cache reuse, dramatically reducing redundant computations. The resource manager prepares tool execution environments asynchronously and balances memory loads. In evaluations across coding, routing, and scientific discovery agents, the system delivered 1.5-3.6x higher throughput for serving live requests and 1.8-3.9x faster reinforcement learning rollouts for training. It also cut disk memory usage by up to 4.2x compared to state-of-the-art baselines. The team has open-sourced the complete system implementation to foster further development and reproducibility in the fast-growing field of AI agents.