Researchers launch NL2SHACL-Bench to test AI on SHACL translation

Researchers Yuchen Zhou, Niels Bobet, and Maribel Acosta from the Karlsruhe Institute of Technology (KIT) have introduced NL2SHACL-Bench, a benchmark suite designed to evaluate the performance of large language models (LLMs) in translating natural language requirements into SHACL (Shapes Constraint Language) shapes. SHACL is a W3C standard for validating RDF knowledge graphs, but creating SHACL shapes requires technical expertise that domain experts often lack. NL2SHACL-Bench aims to bridge this gap by providing a standardized way to assess LLMs' ability to generate semantically equivalent SHACL constraints from natural language inputs.

The team evaluated four state-of-the-art LLMs using NL2SHACL-Bench and found that while the models excel at producing syntactically valid SHACL, they still face challenges with complex logical and structural patterns. This highlights the need for further advancements in NL2SHACL technology. The benchmark provides a meaningful basis for measuring progress in this area, offering researchers a tool to push the boundaries of what LLMs can achieve in semantic constraint generation.