MINARD creates narrated, region-grounded walkthroughs for scientific figures
This AI reads papers to produce step-by-step video explanations of complex diagrams.
Understanding a complex scientific figure often requires reading the entire paper to piece together each component's meaning. Current video generation systems lack the ability to produce step-by-step narrations that are both paper-grounded and visually grounded to specific regions of the figure. To fill this gap, researchers at the University of Maryland developed MINARD (Multimodal Interpretation of Narrated Architecture via Region Decomposition). The pipeline first generates a paper-grounded narration that describes the figure's content in a logical sequence, then localizes each narrative step to the corresponding region of the figure. This enables the creation of fully aligned walkthrough videos.
To evaluate such systems, the team also introduces FigTalk, a benchmark that includes novel sequential and component-level grounding metrics. In tests, MINARD produced humanlike, paper-faithful narrations that outperformed existing narration-conditioned figure grounding methods on both automatic metrics and human evaluations. This work promises to make scientific literature far more accessible, especially for readers who struggle to interpret complex diagrams or pipelines. The code and benchmark are publicly available, opening the door for further research in paper-grounded multimodal generation.
- MINARD generates paper-faithful narrations that are sequentially grounded to specific regions of a scientific figure.
- FigTalk benchmark introduces new sequential and component-level grounding metrics for evaluating figure-to-video generation.
- MINARD outperforms existing approaches in both automatic and human evaluations on the FigTalk benchmark.
Why It Matters
Automates the explanation of complex scientific figures, saving researchers time and making papers more accessible.