New BCM metric reveals how AI agents behave inconsistently across tasks

A new paper on arXiv introduces the Behavioral Consistency Metric (BCM), a process-level evaluation tool for language model agents. Authors Amritesh Banerjee and Pranil Raichura argue that traditional outcome metrics like success rate capture only whether an agent succeeds, not how reliably it behaves. BCM works by training a model to predict task success from behavioral features of execution traces, then deriving per-trajectory feature-attribution vectors and measuring their mean pairwise similarity across a system's runs. Testing on about 9,000 trajectories from six different LM agents on software engineering tasks, the researchers found that cross-task and within-task consistency are separate axes that can diverge—some agents are locally reproducible but globally fragmented.

The findings challenge assumptions in agent evaluation. Consistency is not reducible to success rate, since systems with comparable task success can vary widely in behavioral consistency. The paper also shows that the frontier-versus-open-source consistency gap persists even when task difficulty is controlled, suggesting proprietary models may have more stable strategies. BCM is positioned as a complement to outcome metrics, giving developers a way to detect unreliable, strategy-hopping behavior that standard benchmarks miss. The authors are explicit about the metric's conditions and limitations, making it a practical addition to the agent evaluation toolkit for production systems.