MemoryCPT: End-to-end memory framework improves LLM agent cost-performance trade-off
New agent memory pipeline cuts inference costs while boosting answer quality on long-horizon tasks.
Long-horizon LLM agents struggle with memory systems that must retrieve useful evidence from massive interaction histories without bloating downstream context. Existing approaches rely on hand-crafted heuristics and repeated LLM calls, leading to redundant context and high inference costs. To address this, researchers propose MemoryCPT, a pipeline that is fully trainable end-to-end, spanning both offline memory construction and online query-conditioned context generation.
MemoryCPT works in two stages. First, Query-agnostic Distillation (QAD) distills a modular memory-construction pipeline into a compact model using explicit reasoning traces. Second, Query-aware Retrieval and Summarization (QAR) combines reciprocal rank fusion (RRF) with a LoRA-based summarizer trained via Group Relative Policy Optimization (GRPO) under a cost-aware reward. The authors also introduce Quality per Cost (QPC) as a metric to quantify answer quality per unit inference cost. Experiments on LoCoMo and LongMemEval show MemoryCPT outperforms baselines on this trade-off, with ablations confirming the contribution of each component. This work points toward more efficient agent memory systems that could lower operational costs for production AI agents.
- Two-stage architecture: QAD compresses memory construction into a compact model; QAR handles retrieval and summarization with RRF and LoRA summarizer.
- Trained via GRPO with a cost-aware reward, optimizing explicitly for the inference cost-quality balance.
- Introduces QPC (Quality per Cost) metric; validated on LoCoMo and LongMemEval, beating all evaluated baselines.
Why It Matters
For AI teams building long-horizon agents, MemoryCPT reduces memory-related inference costs without sacrificing answer quality, enabling cheaper large-scale deployments.