AgentFAIR: Multi-Agent LLM Framework Evaluates Geospatial Data FAIRness at $0.05/Dataset

Geospatial datasets power urban planning and climate models, but consistent FAIR (Findable, Accessible, Interoperable, Reusable) evaluation has been elusive. Existing tools use different rubrics and often fail on JavaScript-rendered pages or repository-specific identifiers. For 50 datasets across 10 repositories, normalized scores vary by 15 percentage points on average and up to 30.3 for one dataset. Enter AgentFAIR, a multi-agent framework from researchers (Ming Chen, Pranav Pai, et al.) that combines structured metadata extraction with 13 sub-principle-specific LLM evaluators. Each LLM agent produces a 0-3 maturity score, cited evidence, and recommendations; a critic agent then checks evidence consistency and can request targeted re-evaluation. The framework's average FAIR scores are: Findability 79.7%, Accessibility 70.4%, Interoperability 45.3%, and Reusability 72.0%. Critically, sub-principle agreement on repeated runs reaches 89% (standard deviation 3 percentage points) with the critic, versus 71% without. A preliminary 15-dataset expert study yields Fleiss' kappa of 0.71 and 82% alignment with expert consensus. API cost is approximately USD 0.054 per dataset, making large-scale audits feasible.

The results demonstrate auditability and practicality, but the authors caution that limited benchmark coverage, incomplete ablations, and single-model-family validation constrain accuracy and generalization claims. Rank correlations with four baseline tools range from 0.31 to 0.61, and the overall "FAIR-enough" comparison was not statistically significant. Still, AgentFAIR's multi-agent design with built-in critic and evidence citation offers a transparent, affordable way to standardize FAIR evaluation for geospatial data. As data-driven decision-making grows, such frameworks could help institutions and researchers ensure their datasets meet interoperability and reusability standards, reducing friction in cross-domain collaboration.