WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

A team of researchers has introduced WorkflowPerturb, a novel benchmark designed to stress-test evaluation metrics for multi-agent AI workflows. The system addresses a critical gap in AI development: while LLMs increasingly generate structured workflows for complex tasks, existing metrics often fail to communicate how severely a workflow has degraded when errors occur.

The technical approach involves applying controlled, realistic perturbations to 4,973 'golden' (correct) workflows, creating 44,757 test variants. The benchmark tests three specific perturbation types: Missing Steps (removing workflow components), Compressed Steps (combining multiple steps), and Description Changes (altering step descriptions). Each perturbation is applied at three severity levels (10%, 30%, and 50%), allowing researchers to measure how evaluation metrics respond to progressively worse workflow degradation.

The researchers benchmarked multiple metric families and analyzed their sensitivity and calibration using expected score trajectories and residuals. This systematic testing reveals which metrics provide meaningful signals about workflow quality degradation versus those that produce misleading scores. The findings characterize systematic differences across metric families and support severity-aware interpretation of workflow evaluation scores, moving beyond simple pass/fail assessments.

For developers building multi-agent systems, WorkflowPerturb provides a standardized way to test whether their evaluation metrics can reliably detect different types and severities of workflow problems. This is particularly important as AI agents move from simple chat interfaces to complex, multi-step workflows in production environments. The dataset will be released upon paper acceptance, providing the community with a much-needed tool for rigorous workflow evaluation.