SCOPE & SCION: New benchmark and auditable pipeline for schema induction from text
Outperforms LLM-only baselines with auditable, evidence-linked schema construction from raw text.
Schema graphs are a critical bottleneck for knowledge graph construction and schema-grounded extraction, yet most systems assume a pre-existing schema. To address this, researchers present SCOPE (Schema Construction and Ontology-induction Pipeline Evaluation), a train-text-only benchmark built from 24 public information extraction sources (15 relation extraction and 9 event extraction). These sources are normalized into evaluation-only gold schema graphs, covering event types, within-event argument roles, and inter-event links reported separately.
SCION (Schema Construction and Induction with Ontology Normalization) is an auditable reference pipeline that constructs candidate spaces from training text and restricts naming, merging, filtering, validation, and conservative fusion to candidate-linked evidence under strict JSON contracts. On the SCOPE core suite, SCION-lite attains the highest F1 among released source-schema references, Text2Onto-style, LLM-only, and matched extract-then-aggregate baselines across Literal, Fuzzy, Continuous, and Graph schema-graph metrics. The compact open-model SCION-RL variant reduces reliance on proprietary LLM schema engineers. The release includes evidence-linked outputs, parse/fallback logs, candidate retention/merging logs, run manifests, code, and benchmark packages.
- SCOPE benchmark uses 24 public IE sources (15 RE + 9 EE) for corpus-to-schema induction evaluation
- SCION-lite achieves highest F1 across Literal, Fuzzy, Continuous, and Graph metrics vs LLM-only baselines
- SCION-RL variant cuts proprietary LLM dependency using compact open models with auditable evidence links
Why It Matters
Provides a standardized, auditable way to induce schemas from text, enabling more transparent and reproducible knowledge graph construction.