KGERMAR boosts long-context AI with dynamic knowledge graphs – 8.5% lower perplexity

KGERMAR addresses a core limitation in long-context language modeling: maintaining coherent understanding of entity states and relationships across thousands of tokens. Traditional retrieval-augmented models rely solely on semantic similarity, missing explicit relational patterns. KGERMAR constructs dynamic knowledge graphs from input text in real time during inference, extracting entities and relations to build contextual graphs. These graph-structural embeddings are integrated with textual semantics through a multi-component memory architecture comprising three banks: contextual, semantic, and structural. Retrieval signals from each bank are fused via learned weights, capturing both surface-level semantics and deeper relational patterns.

Evaluated on four datasets—SlimPajama (84.7K training examples), WikiText-103 (4,358 examples), PG-19 (100 examples), and Proof-pile (46.3K examples)—KGERMAR consistently outperforms memory-augmented baselines. Across context lengths from 1K to 32K tokens, it achieves up to 8.5% lower perplexity and 2–2.5x better memory efficiency. The approach also shows superior in-context learning performance across five natural language understanding tasks. By enabling domain-adaptive knowledge representation that adjusts to input contexts rather than relying on fixed knowledge bases, KGERMAR advances memory-augmented language modeling for practical long-document applications.