Cloudflare has announced a new feature that allows its web-hosting customers to block AI bots scraping their websites to train AI models.
“Customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the company said in a blog post.
Go deeper with GlobalData
“We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection,” it added.
The company has added a new one-click tool for website hosts to block all AI bots and announced it is free for all customers.
AI vendors such as Google and OpenAI allow website owners to block the bots they use for data scraping by editing their website’s robots.txt, the text file that tells bots which pages they can access on a website.
However, Cloudflare said these blocks rely on the AI bot operator respecting robots.txt and honestly identifying who they are when they visit an Internet property.
“Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent,” the company wrote.
Cloudflare said its global machine learning model has consistently recognised this activity as a bot, even when operators lie about their agents.
