GPT-5.4, Claude, Gemini flunk multi-level modeling semantics test

Researchers from multiple institutions conducted the first empirical study evaluating whether large language models (LLMs) can handle multi-level modelling (MLM) semantics — a task significantly more complex than the two-level modelling usually tested. They asked GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro to generate MLM models for the MULTI Warehouse Challenge using the SLICER language under six prompting strategies. A total of 90 generated models were compared against a manually validated reference using fourteen metrics.

The results reveal a clear capability gap: syntactic correctness was achievable, but semantic correctness — especially Instantiation/Specialisation — hit only 52-79%. LLMs faithfully copy explicitly stated content but rarely infer implied structure or constraints. Prompting strategies traded off precision for completeness, and self-checking acted as a rule checker rather than reliably improving alignment. Among the three, Claude demonstrated the most balanced profile, suggesting model-specific strengths for MLM workflows.