Cloudflare has announced a new feature that allows its web-hosting customers to block AI bots scraping their websites to train AI models. 

“Customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the company said in a blog post. 

“We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection,” it added. 

The company has added a new one-click tool for website hosts to block all AI bots and announced it is free for all customers.

AI vendors such as Google and OpenAI allow website owners to block the bots they use for data scraping by editing their website’s robots.txt, the text file that tells bots which pages they can access on a website.

However, Cloudflare said these blocks rely on the AI bot operator respecting robots.txt and honestly identifying who they are when they visit an Internet property. 

How well do you really know your competitors?

Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.

Company Profile – free sample

Thank you!

Your download email will arrive shortly

Not ready to buy yet? Download a free sample

We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form

By GlobalData
Visit our Privacy Policy for more information about our services, how we may use, process and share your personal data, including information of your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.

“Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent,” the company wrote. 

Cloudflare said its global machine learning model has consistently recognised this activity as a bot, even when operators lie about their agents.