Graph-based RL framework catches LLM agent drift before it breaks

Autonomous LLM agents can silently drift from their original task during long-running workflows, sometimes causing irreversible damage to external systems. Existing safeguards work at the prompt level, but they lack structured, step-level detection and recovery. A new arXiv paper (arXiv:2608.14109) from El Hamraoui, Jose, Bureau, and Plana introduces a plug-and-play recovery module that sits outside the main agent. Instead of retraining the large task-executing model, they train a single small language model via reinforcement learning to specialize at each node of a recovery graph. The graph assigns distinct roles: drift classification, operation detection, risk evaluation, and final recovery decision. The model outputs structured, XML-formatted reasoning, making its decisions auditable and role-specific.

Training combines two reward signals: rule-based structural rewards that enforce schema and length, and an LLM-as-judge semantic-quality signal that scores the content itself. This hybrid approach ensures the small model both follows the expected output format and produces semantically appropriate reasoning. Benchmarked on AppWorld, a public agent benchmark, the method successfully exploits drift-onset information to issue correct recovery decisions. The results suggest that lightweight models can act as reliable safety supervisors for much larger agents, catching behavioral drift early and keeping complex workflows on track without expensive retraining or architectural changes.