In July of this year, Cloudflare released a free tool for website owners to preve AI companies from web scraping or coe mining to train large language models. Now, the company plans to launch a marketplace in 2025 where website owners can charge AI developers for scraping permission.
Cloudflare CEO Matthew Prince told TechCrunch: “If you don’t pay (coe) producers in some way, they’re not going to create anything, and that’s a problem that needs to be solved.”
Although big sites like Time or the Financial Times have coracts with AI companies, small sites still don’t get paid for their data mining by AI bots. To avoid this and the frustration of coe producers, Cloudflare has been launching various programs and tools for some time.
Cloudflare Marketplace to sell coe to AI companies
Since last year, people’s use of productive AI tools has increased tremendously, but how will small publishers survive in the age of AI if people just get their information from chatbots like ChatGPT instead of going to websites?
Today, AI model providers scrape thousands of small websites to obtain data to train their large language models. Although some larger publishers have coracted with OpenAI for licensing, small websites generally don’t charge, yet their coe still finds its way io popular AI models. This can bankrupt many websites and reduce their traffic.
In early summer 2024, startup Perplexity was accused of ignoring requests from sites not to mine data. Shortly after the news, Cloudflare released a tool to allow customers to block all AI scraping bots with one click.
But what if you wa to block Perplexity bots, and leave OpenAI bots free? CEO Cloudflare says their customers will be able to choose which AI models access their sites with the new tools. However, Cloudflare tools are not limited to this advaage; Cloudflare’s marketplace, which will launch next year, plans to allow small websites and publishers to do business with AI modeling providers as well.
According to CEO Cloudflare, this marketplace is ultimately good for the AI ecosystem. Prince says the curre landscape, where some AI companies never pay for coe, is unsustainable.
Not much is known about this marketplace, for example, we don’t know how to determine scraping fees, or for example, how to get credit and special benefits from the company that created it in exchange for access to AI bots.
