Dyserve: Workflow-Aware Serving Layer Cuts Latency for Agentic AI Apps
New ILP-based scheduler optimizes LLM model choices across heterogeneous backends in real-time.
Agentic AI applications generate complex workflows—directed acyclic graphs of LLM calls and tool interactions—where each node can use a different model, verifier, or backend. Current systems either optimize per-call (model servers) or fix the workflow (agent frameworks), missing the chance to jointly select resources under real-time load. Dyserve fills this gap as a dedicated serving layer that understands workflow structure.
Dyserve formulates each workflow's per-node choices as an integer linear program (ILP) over a heterogeneous pool of backends, priced using offline skill-conditioned profiles that generalize across workflows. The solver concentrates stronger models and verification on nodes whose errors propagate the most, while hardware constraints enter only through throughput sweeps. To avoid latency spikes, Dyserve pre-solves the ILP at multiple pressure levels at admission and dynamically shifts a workflow's uncommitted suffix among strategies under load—keeping the solver off the critical path. Failed tool calls trigger a one-time residual re-solve that preserves completed work. This design enables significant latency reductions and quality improvements for multi-step agentic tasks.
- Dyserve compiles each workflow node's model and verifier choices into an integer linear program (ILP) over a heterogeneous backend pool.
- It uses skill-conditioned offline profiles that transfer across workflows, with hardware entering only through per-model throughput sweeps.
- Pre-solves at several pressure levels at admission and shifts uncommitted workflow suffixes under load, avoiding solver overhead on the load-shift path.
Why It Matters
Makes agentic AI systems faster and more reliable by intelligently allocating models and resources based on workflow structure and load.