LLMs struggle with UML/ER diagrams: review finds GPT bias and gaps

A new systematic review published on arXiv by researchers Mojdeh Rahmanian, Ashkan Sami, and Yanchao Yu analyzes 64 studies from 2023 to 2025 on how large language models (LLMs) are used for software engineering diagrams—specifically UML and entity-relationship (ER) modeling. The findings reveal significant concentration patterns: class diagrams receive the most attention, while behavioral diagrams and data modeling are underrepresented. Diagram construction from natural language is the primary focus, with limited work on transformation, quality assurance, and consistency checking. GPT-based models dominate the field, raising serious concerns about reproducibility and vendor dependence. Evaluation practices are heterogeneous, using diverse metrics and custom datasets with little benchmark reuse, and inconsistent reporting of robustness and statistical significance.

Common limitations across studies include semantic inaccuracies, hallucinated diagram elements, high sensitivity to prompt formulation, and reproducibility constraints. The review is the first systematic synthesis of LLM-based diagram modeling research, highlighting urgent needs for standardized benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques to improve semantic reliability and multi-view consistency. For practitioners, the takeaway is clear: while LLMs can assist in generating UML and ER diagrams from natural language descriptions, current models are unreliable for production use without rigorous human validation and standardized testing.