This New AI Metric Reveals Why Your Model's Accuracy Is Lying to You

Machine learning models are typically judged by predictive accuracy, ranking quality, or classification error. But in high-stakes fields like healthcare, finance, and autonomous systems, whether a model respects logical or domain-specific constraints can be just as critical. A new paper from researchers at LIG and TYREX introduces the Rule Violation Score (RVS), a complementary evaluation metric that measures logical compliance independently of predictive accuracy. RVS can be applied to any predictive model expressed over a relational vocabulary and can be computed using automatically generated SQL queries for Horn rules.

RVS distinguishes between hard rules (strict constraints that must always hold) and soft rules (statistical regularities). The metric was evaluated on three benchmarks covering knowledge graph link prediction and relational regression, comparing rule-based, embedding-based, and neuro-symbolic models. Results revealed that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance—differences that standard metrics fail to capture. RVS can also assess the logical consistency of training datasets and help identify poorly defined rules, offering a more complete picture of model behavior.