AgentCARD: Mixing model deployments cuts LLM agent costs 12x

A new academic paper introduces AgentCARD, a role-aware benchmark designed to optimize the cost-accuracy trade-off in multi-agent LLM systems. Unlike existing benchmarks that evaluate fixed models or fixed agent configurations, AgentCARD systematically assesses how different role assignments—planner, executor, verifier—and deployment modes (API, self-hosted, hybrid) affect overall team performance. The researchers built a unified cost model covering both API pricing and self-hosted infrastructure, performed Pareto-frontier analysis to identify optimal trade-offs, and used Shapley-based diagnostics to pinpoint role bottlenecks.

Key findings show that heterogeneous teams (mixing different models for different roles) consistently dominate the cost-accuracy frontier. Compared to cost-equivalent homogeneous teams (same model for all roles), heterogeneous teams achieve up to 44% higher accuracy. Alternatively, by adopting hybrid deployment (e.g., running the planner on a cheap self-hosted model while using a premium API for the executor), teams can match the accuracy of the strongest homogeneous team while cutting per-task costs by up to 12x. The optimal role assignment is domain-dependent: some domains are bottlenecked by the planner role, others by the executor. This suggests that practitioners should profile their specific tasks and adjust role-model assignments accordingly. AgentCARD also extends to workflows with additional roles like verification, and its code is publicly available for continual evaluation.