Disclosure or Marketing? Analyzing the Efficacy of Vendor Self-reports for Vetting Public-sector AI

A new research paper titled "Disclosure or Marketing?" from Carnegie Mellon and University of Washington researchers provides a critical, evidence-based look at a cornerstone of responsible AI governance: vendor self-disclosure documents. The team analyzed the widely adopted GovAI Coalition FactSheet, a template designed to bring transparency to public-sector AI procurement. Through semi-structured interviews with both vendors and government practitioners and a systematic review of 31 completed FactSheets, the study reveals a fundamental tension. These documents are expected to simultaneously act as marketing brochures for vendors, due-diligence tools for evaluators, and conversation starters for early-stage dialogue, creating conflicting incentives that undermine their objectivity.

Despite these limitations, the research offers a path forward. The findings suggest that FactSheets and similar artifacts like model cards and datasheets cannot be treated as standalone, definitive risk-assessment tools due to the structural constraints of voluntary, public self-reporting. However, their real value emerges when they are understood as "relational artifacts." In this framework, the documents serve not as final audits but as instruments to establish initial trust, create a shared vocabulary, and facilitate an ongoing, critical dialogue between agencies and suppliers. This shift in perspective—from treating disclosure as a one-time checkbox to viewing it as part of a continuous governance process—could help public agencies move beyond surface-level transparency toward more substantive accountability and better-informed procurement decisions.