New SKL framework beats reflection-based training for AI agents

Reflection-based training has been the default way to improve LLM agents, but a new arXiv paper argues that trajectory-level summarization is 'episodic hindsight' rather than predictive foresight. The authors propose Stateful Knowledge Learning (SKL), which shifts agents to maintaining explicit, declarative predictive assessments anchored to state. These stateful knowledge representations provide finer granularity, enhance generalization across tasks, and enable knowledge bootstrapping—allowing agents to build on previously acquired insights rather than relying on path-dependent heuristics.

To scale SKL, the team introduces two algorithms: SKL-SD (self-distillation) and SKL-RL (reinforcement learning). Both train agents to autonomously extract state-grounded predictive knowledge from experience and then use that knowledge to inform future policies. The paper includes a motivating example highlighting the advantages of stateful knowledge over trajectory reflection, then validates the methods in interactive environments like WebShop and ScienceWorld, plus the complex reasoning task ChessPuzzles.

Across all benchmarks, models equipped with SKL significantly outpace current reflection-based training paradigms. The results suggest that teaching agents to think in terms of state-conditioned predictions—rather than retrospective summaries—could be a more scalable and robust path to continual learning. The paper is authored by Yan Song, Xidong Feng, Bo Liu, and colleagues, and is available on arXiv with DOI 10.48550/arXiv.2607.28638.