Stanford audit finds hidden errors in property data bias AI tax models

A new study from Stanford researchers—Evelyn Smith, Emma Harvey, Jacob Goldin, and Daniel E. Ho—exposes hidden flaws in brokered property datasets widely used in academic research and AI models. Auditing two prominent commercial datasets, the team compared them against ground truth records in Cook County, Illinois. They found that 1-2% of matched sales from 2018-2021 had reported sale prices differing from actual transaction prices by more than 5%. More strikingly, missing data and conceptual mismatches in deed and property characteristics created coverage errors affecting 12-15% of all transactions.

The errors are not random. The study shows that different brokers often make identical reporting mistakes for the same transactions, suggesting systematic bias rather than isolated noise. When applied to property tax regressivity analyses—a key measure of how tax burdens fall on low-income homeowners—the data source alone drove significant differences in outcomes. The same patterns appeared in two other major U.S. counties, indicating a broad industry-wide issue. With property records feeding everything from gentrification studies to automated valuation models, the findings underscore the urgent need for open administrative data and stronger transparency from brokers about data provenance and lineage.