Cloudflare’s new service; Monetization of sites in exchange for cooperation with artificial intelligence

In July of this year, Cloudflare released a free tool for website owners to prevent AI companies from web scraping or content mining to train large language models. Now, the company plans to launch a marketplace in 2025 where website owners can charge AI developers for scraping permission.

Cloudflare CEO Matthew Prince told TechCrunch: “If you don’t pay (content) producers in some way, they’re not going to create anything, and that’s a problem that needs to be solved.”

Although big sites like Time or the Financial Times have contracts with AI companies, small sites still don’t get paid for their data mining by AI bots. To avoid this and the frustration of content producers, Cloudflare has been launching various programs and tools for some time.

Cloudflare Marketplace to sell content to AI companies

Since last year, people’s use of productive AI tools has increased tremendously, but how will small publishers survive in the age of AI if people just get their information from chatbots like ChatGPT instead of going to websites?

Today, AI model providers scrape thousands of small websites to obtain data to train their large language models. Although some larger publishers have contracted with OpenAI for licensing, small websites generally don’t charge, yet their content still finds its way into popular AI models. This can bankrupt many websites and reduce their traffic.

In early summer 2024, startup Perplexity was accused of ignoring requests from sites not to mine data. Shortly after the news, Cloudflare released a tool to allow customers to block all AI scraping bots with one click.

But what if you want to block Perplexity bots, and leave OpenAI bots free? CEO Cloudflare says their customers will be able to choose which AI models access their sites with the new tools. However, Cloudflare tools are not limited to this advantage; Cloudflare’s marketplace, which will launch next year, plans to allow small websites and publishers to do business with AI modeling providers as well.

According to CEO Cloudflare, this marketplace is ultimately good for the AI ​​ecosystem. Prince says the current landscape, where some AI companies never pay for content, is unsustainable.

Not much is known about this marketplace, for example, we don’t know how to determine scraping fees, or for example, how to get credit and special benefits from the company that created it in exchange for access to AI bots.