GraphInstruct benchmark reveals LLMs fail at multi-constraint graph generation
New benchmark tests LLMs across 6 complexity levels, exposing where they break.
Graph-structured data is critical for citation networks, social modeling, molecular design, and knowledge graphs, yet LLMs used as prompt-driven graph synthesizers lack systematic evaluation. Existing benchmarks average over structural complexity and fail to pinpoint where LLMs break down. To close this gap, researchers from Tongji University and collaborators introduce GraphInstruct—a progressive-complexity benchmark that stratifies graph generation into six levels and five evaluation dimensions. The benchmark includes 800 hand-authored instructions, 1,582 algorithmically generated reference solutions, and tests 12 models across 45 (model, strategy) configurations.
Key findings reveal that discriminative power peaks at multi-constraint composition rather than reasoning depth, and no single prompting strategy works across all levels or model families. Critically, domain-semantic constraints remain unchanged regardless of iteration, pointing to retrieval-augmented generation rather than additional compute as the next research direction. A verification-guided iterative framework with constraint-aware adaptive prompting consistently outperforms standard prompt engineering across tested models, demonstrating that GraphInstruct’s fine-grained signals can directly drive method development.
- GraphInstruct includes 800 instructions and 1,582 reference solutions across 6 complexity levels and 5 evaluation dimensions.
- Tested 12 LLMs (e.g., GPT-4, LLaMA) across 45 configurations; discriminative power peaks at multi-constraint composition.
- Domain-semantic constraints remain iteration-invariant, suggesting retrieval (RAG) is the next frontier over more compute.
Why It Matters
This diagnostic benchmark reveals where LLMs fail on graph tasks, guiding better retrieval-augmented approaches.