AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs

A research team from Macquarie University and University of Technology Sydney has introduced AgenticAI-DialogGen, a novel framework designed to solve a persistent problem in conversational AI: the lack of quality datasets for training and evaluating both short- and long-term memory in Large Language Models (LLMs). Current datasets often lack memory grounding or rely on expensive human annotation, limiting progress in creating AI that can maintain coherent, extended conversations. This framework automates the entire process using LLM agents in a modular pipeline to extract knowledge graphs, build speaker personas, and simulate realistic, topic-guided dialogues.

The system generates a new dataset called TopicGuidedChat (TGC), which encodes long-term memory as speaker-specific knowledge graphs and short-term memory as newly generated conversations. A key component is a QA module that creates memory-grounded question-answer pairs drawn from both recent and distant conversational history. Initial evaluations show that conversations generated by AgenticAI-DialogGen achieve higher quality scores, and more importantly, LLMs fine-tuned on the TGC dataset demonstrate measurable improvements in performance on memory-based QA tasks. This represents a significant step toward creating AI assistants with more consistent, context-aware, and long-term conversational abilities.