Researchers launch TangPoetryBench to grade AI poetry-to-image models
New benchmark uses 1,280 annotated images to evaluate how well AI illustrates classical Chinese poetry
Researchers from Tongji University have developed TangPoetryBench, a groundbreaking benchmark designed to evaluate how effectively text-to-image (T2I) models can illustrate classical Chinese poetry. Unlike existing benchmarks that focus on literal text-image correspondence, TangPoetryBench assesses 10 dimensions of quality, including visual soundness, cultural aptness, emotional fidelity, and faithfulness to implicit poetic imagery. The benchmark comprises 1,280 images generated by four state-of-the-art T2I models from 320 classical Chinese Tang poems, with human annotations providing ground truth evaluations.
The team also introduced PoemAutoEvaluator (PAE), an open, rubric-conditioned evaluator that achieves parity with proprietary systems like Claude in assessing poetic imagery and emotion. PAE enables scalable evaluation without requiring fresh human annotations for new images and generalizes to unseen generators and poetic traditions, such as Song Ci poetry. The benchmark, annotations, and evaluator are all publicly released, offering a new tool for researchers and developers to measure and improve the artistic and emotional capabilities of T2I models.
- TangPoetryBench evaluates 1,280 poetry-to-image outputs (320 poems × 4 models) across 10 dimensions, including emotional and cultural fidelity
- PoemAutoEvaluator (PAE) rivals proprietary systems like Claude in assessing poetic imagery and emotion, and generalizes to new models and poetic traditions
- Benchmark and tools are open-sourced, enabling scalable, automated evaluation of T2I models' artistic and emotional capabilities
Why It Matters
Sets a new standard for evaluating AI's ability to interpret and visualize abstract literary concepts, bridging cultural understanding and generative AI.