ProvenanceGuard slashes LLM agent misalignment errors by 95%
New framework drops error rate from 42.9% to 1.8% on safety benchmarks.
As LLM agents gain access to powerful tools like code execution and web browsing, ensuring their actions align with user intent is critical. A new paper from Carnegie Mellon researchers introduces ProvenanceGuard, a provenance-based framework that detects misalignment before a tool is invoked. Unlike existing "LLM-as-a-judge" guardrails that produce inconsistent verdicts, ProvenanceGuard formalizes detection as checking whether a proposed tool call is supported by traceable evidence in the agent's context. The pipeline scans for three types of misalignment: tool misuse, parameter errors, and goal divergence.
Evaluated on two benchmarks—Agent-SafetyBench and WorkBench—across 10 backbone LLMs, ProvenanceGuard dramatically outperforms baselines. It reduced the error rate on misaligned traces from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench. Importantly, it also halved unnecessary interventions on task-successful traces (30.5% to 12.8%) without statistically increasing false positives. The structured, provenance-based reasoning provides a practical, auditable foundation for safe LLM agent deployment.
- ProvenanceGuard uses a multi-stage pipeline analyzing three types of misalignment before tool execution.
- Error rate on misaligned traces dropped from 42.9% to 1.8% on Agent-SafetyBench.
- Unnecessary interventions on successful tasks reduced from 30.5% to 12.8% with no increase in false alarms.
Why It Matters
A scalable, auditable approach to preventing harmful LLM agent actions before they execute.