Seibert's arXiv study finds no single domain-model metric beats human ratings

Vasiliy Seibert's new arXiv paper (2608.15250) tackles a practical problem in model-driven software engineering: how to automatically compare domain models—semantic structures capturing shared stakeholder understanding. The literature offers many similarity metrics, but practitioners have no defensible way to choose among them. Seibert implemented five such metrics, ran them on a fixed set of 39 domain-model comparisons, and compared each metric's output against human expert ratings for the same comparisons.

The study addresses two research questions: RQ1 asks how close each metric is to human ratings on average, while RQ2 asks how consistent each metric's per-comparison distance is. The findings reveal that no single metric dominates all criteria. Some metrics achieve the closest average distance to human experts, while others excel at preserving per-pair ordering—correlation and ranking consistency. This suggests an ensemble approach, combining multiple metrics, may serve as a practical substitute for expensive human expert grading. The metric implementations are published according to FAIR4RS recommendations (DOI: 10.5281/zenodo.20942596), making them reusable for further research in requirements engineering, education, automatic model generation, and repository mining.