LLM4MTLs: LLMs improve syntax but not semantics in model transformation code

A new paper from researchers at multiple institutions introduces LLM4MTLs, an automated workflow for constructing and evaluating prompting strategies to generate model transformation language (MTL) code via large language models. MTLs are domain-specific languages used to transform models (e.g., UML) into other models or source code. The team tested four MTLs—ATL, ETL, QVTo, and the Reactions language—across three unnamed LLMs, using both syntactic and semantic metrics. The workflow systematically combines few-shot prompting, grammar prompting, and helper method inclusion.

The key finding: few-shot prompting consistently improves syntactic quality (e.g., valid syntax) across all four MTLs, but gains in semantic correctness—whether the transformation actually works—are uneven and highly language-dependent. For ATL, Pass@1 remained unchanged across all strategies, suggesting that surface-level syntax improves more readily than deep transformation semantics. Grammar prompting stabilizes generation when paired with few-shot examples, but in isolation it can be ineffective or even harmful. Helper methods act as a complementary amplifier. LLM choice significantly influences syntactic correctness for ETL and QVTo, but has limited impact on semantic correctness, underscoring that current LLMs struggle to grasp MTL semantics without extensive domain-specific training data.