SciDraw-Bench: New benchmark shows domain-specific AI beats general models on scientific figures

Existing image-generation benchmarks like GenEval or T2I-CompBench focus on natural images, measuring compositionality and photorealism—but none evaluate what makes a scientific figure usable: correct labels, faithful entity relations, and adherence to disciplinary conventions. To fill this gap, Davie Chen introduces SciDraw-Bench, a benchmark of 32 carefully designed tasks spanning eight figure types (e.g., mechanism diagrams, experimental schematics, graphical abstracts) across ten scientific disciplines. Each task pairs a natural-language prompt with a machine-checkable specification of required labels, relations, components, and negative constraints. The evaluation protocol has four dimensions: Text Fidelity (OCR-based label recall and character error rate), Semantic Correctness (vision-language-model judging against the specification), Structural Quality, and Convention Adherence.

The paper compares a domain-specific system (SciDraw AI) against representative general-purpose text-to-image models (likely including GPT-4V and others). In a pilot covering all eight figure types, SciDraw AI substantially outperforms the baselines on every dimension and figure type. The largest performance gaps are in semantic correctness and convention adherence—areas where the domain-specific design explicitly targets scientific accuracy. Text fidelity remains the hardest dimension for all systems, indicating that generating legible, correctly spelled labels in diagrams is an ongoing challenge. The paper also outlines a planned code-to-figure baseline and includes meta-evaluation and inter-judge reliability analysis (human-rating validation is ongoing). SciDraw-Bench sets a new standard for evaluating AI's ability to produce accurate scientific visuals.