New KDD 2026 paper proposes evaluation protocol for personal LLM agents under temporal interventions

A new position paper from Pin Qian and seven co-authors, accepted for oral presentation at the KDD 2026 Workshop on Personal Intelligence in the Agentic AI Era (PILA 2026), argues that current evaluation protocols for personal LLM agents are fundamentally inadequate. The authors contend that personal agents—which maintain memories, learned skills, tool configurations, and policy state that evolve with each user—require a testing paradigm that replays the same temporal intervention across different persistent user-conditioned states and measures how failures propagate across components.

The paper formalizes this requirement into four conditions: explicit temporal intervention, persistent state across the intervention, induced cross-dimensional effects, and variation in user-conditioned state. A focused audit of existing public benchmarks—including tool, memory, and safety benchmarks—found that none satisfied all four conditions under the authors' narrow operationalization. The result is a concrete design requirement for future personal-agent evaluation, with proposed candidate reporting metrics to track user-conditioned adaptation over time.