IFCMemoryBench: LLM agents hit only 32% accuracy on BIM memory recall

A new research paper from Du et al. introduces IFCMemoryBench, a benchmark designed to evaluate the long-term memory capabilities of LLM-based agents in the domain of Building Information Modelling (BIM). Unlike existing benchmarks that test conversational recall in open-domain settings, IFCMemoryBench requires agents to combine information from previous sessions with live queries over structured IFC models. The benchmark comprises 143 multi-session tasks derived from 19 real-world projects and 4,016 prior sessions, each requiring the agent to recall project context seeded in earlier conversations to answer a probe question.

The results highlight significant shortcomings in current memory systems. Under a deployment-realistic ingestion scope, the best-performing system achieved only 32.4% accuracy, and even with oracle-filtered ingestion it remained below 60%. Analysis revealed that general-purpose vector-, graph-, and file-based memory systems often retrieve topically relevant context but store project knowledge as incomplete or fragmented facts. The authors argue that reliable professional agents require domain-aware memory representations that link conversations, project knowledge, and structured model entities.