New evaluation framework pinpoints narrative loss in AI oral history visualizations

A new research paper accepted at the ICML 2026 Workshop on Culture x AI tackles a thorny problem: how do you evaluate AI-generated visualizations of oral histories? Diaspora oral-history interviews require a double transformation — turning first-person recollection into a third-person scene and moving the present interview room to a past time and place. When generative AI performs this transformation, there are no agreed criteria for success. The authors — Kwangsuk Park, Jaehyun Koo, Jiyeon Lee, Anjung Tan, and Hyoungchul Park — derive success conditions from oral-history theory, then design 15 metrics around three failure modes.

They compare two AI pipelines: a Multi-Agent Scene-decomposition pipeline (MAS) and a Single Summarization Pipeline (SSP) across 82 interviews. The key finding: scene-planning and narrative preservation conflict in the majority of cases. The narrative-structure strength of the source testimony is the primary predictor of this conflict—stronger narratives better survive the transformation. Based on this, the researchers propose a failure-mode-based evaluation framework, an empirical analysis of conflict conditions, and a routing protocol for system selection based on narrative-structure strength. For professionals working with cultural memory or AI-generated media, this provides a rigorous, theory-grounded approach to measuring what gets lost.