AgentLTL: New framework enforces procedural compliance in LLM agents
Final answer scoring is not enough—AgentLTL scores the process itself.
In safety-critical AI applications, how an agent arrives at an answer can be as important as the answer itself. Traditional evaluation metrics—final-answer correctness or LLM judges—ignore the procedure. Now, researchers from EPITA have introduced AgentLTL, a framework built on First-Order Linear Temporal Logic (FO-LTL) that defines procedural rules over an agent's trace of tool calls. The result is a deterministic, judge-free compliance score.
The framework serves two purposes. First, 'harnessing' uses the specification to score completed traces or even gate tool calls in real time via a block-and-warn mechanism. Second, the compliance score doubles as a dense reward for fine-tuning. On a benchmark covering ordering, branching, iteration, and grounding, the harnessing method improved compliance on five out of seven agent models. Fine-tuning with AgentLTL rewards delivered a +38 percentage point accuracy gain and a +17.5 point compliance gain on held-out patterns, including unseen tool-name aliases, suggesting the models learned procedural structure rather than surface-level memorization.
- AgentLTL uses FO-LTL to define procedural rules over agent traces, yielding a deterministic, judge-free compliance score.
- Block-and-warn harnessing improved compliance on five of seven models during online tool calls.
- Fine-tuning with AgentLTL rewards gave +38% accuracy and +17.5% compliance gains on held-out patterns.
Why It Matters
Moves LLM agent evaluation from outcome-only to process-aware, critical for safety-critical deployments.