Meta Backs Off Plan to Scrape News Sites for AI Training

Meta has updated its terms of service to remove language that critics said gave it the right to scrape content from news sites and other websites to train its artificial intelligence. The original wording suggested Meta could use any content from web domains to "develop, improve, research, train" its AI products. Experts like Cornell professor James Grimmelmann interpreted this as an attempt to bypass robots.txt—the file websites use to tell bots not to scrape them—and take news content for free.

After pushback from publishers and legal experts, Meta removed the controversial language. Andy Stone, Meta's VP of communications, initially said there was "no change to how Meta uses web crawlers." But on Tuesday, spokesperson Daniel Roberts clarified that the crawling terms only applied to Meta's ads and business products. Roberts emphasized that content owners can still use robots.txt to signal their preferences, and the update doesn't give Meta new rights that supersede those preferences.

The episode highlights growing tensions between AI companies and content creators. As AI models require massive amounts of data, tech giants face scrutiny over how they collect that data. News organizations and websites rely on robots.txt to control access, and any hint that companies might ignore it raises alarm. While Meta says this was a misunderstanding, the swift removal shows that public pressure can influence corporate policies.

For now, the change means news sites and other content owners have clearer terms: Meta won't use their content for AI training without permission, at least according to the updated language. However, the incident serves as a reminder to stay informed about how tech companies use web content, as the rules around AI and data scraping are still evolving.