FinPerMA benchmark shows LLM agents fail at personalized financial memory

FinPerMA, a new benchmark from Ben Wang and four co-authors, evaluates whether LLM agents can maintain and update personalized user models in high-stakes financial advising. Unlike prior benchmarks that test factual retention or rely on loosely constrained model-generated trajectories, FinPerMA uses frozen longitudinal investor trajectories with deterministic, theory-informed impact rules and controlled LLM narration. A Post-Shock checkpoint specifically tests whether an agent integrates material events—like a market crash or job loss—into its persistent user model.

On 2,994 questions across 276 personas, seven frontier LLMs with up to seven memory configurations fell far from saturation. No full-context configuration exceeded 47% overall accuracy or 39% on multiple-choice questions. Attribution analysis revealed a critical failure mode: summary-based memory often preserves factual details while discarding preference signals essential for personalization. As a result, simple retrieval frequently outperformed purpose-built memory systems, and this gap widened after shocks. The benchmark highlights a systemic weakness in current agent memory architectures, suggesting that sophisticated memory designs may be over-engineered without grounding in event-driven preference adaptation.