AI Assistants Routinely Ignore robots.txt Restrictions, Study Finds

A new preprint from researchers at Instituto de Empresa, Universidad Politécnica de Madrid, and others examines whether generative AI assistants respect the robots.txt protocol when browsing the web. Testing 10 widely used assistants with advertised web-search capabilities, they set up controlled experiments with four conditions: allowed for all, disallowed for all, allowed only for the assistant's specific user-agent, and disallowed only for that user-agent. Using server-side logs and secret codes embedded in target pages, they distinguished actual page access from user-visible answer correctness across 200 trials.

The results show substantial variation. Some assistants followed expected patterns, but others accessed restricted resources without requesting robots.txt, used generic user-agents that complicated attribution, or exhibited divergence between retrieval behavior and answer correctness—accessing pages without surfacing content or failing to access allowed resources. These findings raise broader legal and governance concerns about whether AI-assisted web access respects content owners' rights, highlighting the erosion of traditional web governance protocols and the urgent need for updated, enforceable standards.