Cloudflare has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites generate revenue from AI firms trying to access and train on their content, the software company announced on July 1st. The tool allows website owners to choose if artificial intelligence crawlers can access their material and set a price for access through a “pay per crawl” model, which will help them control how their work is used and compensated, Cloudflare explained.
AI crawlers are increasingly collecting content without directing visitors to the source, and website owners seek to develop additional revenue streams as search traffic referrals that once brought in advertising revenue drop.
Cloudflare’s Chief Strategy Officer Stephanie Cohen stated that the goal of these tools is to give publishers control over their content and to ensure a sustainable ecosystem for online content creators and AI companies.
“The change in traffic patterns has been fast, and something needed to change,” Cohen said in an interview. “This is just the beginning of a new model for the internet.”
According to Cloudflare data, Google’s ratio of crawls to visitors referred back to sites has dropped to 18:1 from 6:1 just six months ago, indicating that Google is maintaining its crawling activity but reducing referrals.
This decline could be due to users finding answers directly within Google’s search results, such as AI Overviews. However, Google’s ratio remains much higher than that of other AI companies, such as OpenAI’s 1,500:1.
For decades, search engines have indexed internet content, directing users back to websites, a model that rewards creators for producing quality material. AI companies’ crawlers have redefined this system by gathering content without sending visitors to the source and aggregating information through chatbots such as ChatGPT, denying creators of revenue and recognition.
Many AI companies are missing a common web standard used by publishers to block automated data extraction for AI training, and they claim they have broken no laws in accessing content freely. In response, some publishers, including the New York Times, have sued AI companies for copyright violation, while others have negotiated licensing deals for their content.
