Cloudflare made the rounds earlier in the month with their announcement of their pay per crawl service. The idea is to enable content publishers to charge AI bots for access to their sites.
I’ve been involved in many facets of content for a long time, and I’m struggling to see how it reaches product-market fit.
You can’t stop the bots
Neither Cloudflare, nor any other service, will ever be able to block all scrapers. They can make their operations more expensive, at best; but not by much, since companies across the world are racing to build smarter, cheaper crawling machines.
One mighty victor against the orange cloud
You ain’t gonna like the price
You charge too high a price for your content, people will take the scraping route. What’s a low enough price point that would work? My guess is somewhere around $0.10 per article.
Let’s now flip to the publisher’s perspective. Are you okay with earning just 10 cents from the redistribution of your work to millions? Unlikely.
Got a feeling OpenAI won’t send me a check for this
You want LLMs to check you out
Thanks to the sea of SEO blog-spam (proud contributor here!) LLMs have become a convenient way to search. And you do want them to talk about you, because otherwise those users will be sent to your competitors.
2016 article - history often rhymes
We’ve been doing this for years
There’s one scenario I haven’t addressed yet: when content is locked behind a subscription, and LLM developers are willing to pay for access.
The solution already exists in this area, in the form of licenses for programmatic access through enterprise contracts. These agreements can easily be extended to include AI usage rights.
A well-established vendor in the news feed space have already adapted
Is there scope for innovation? Maybe. But it’s a complex issue - sellers will want to review each bot on a case-by-case basis, to understand how the’re planning to redistribute the content (overview + link vs full-fledged regurgitation).
Concluding thoughts
I love Cloudflare - this site runs on it! That said, I just can’t see pay-per-crawl working for commodity content, or for high-value material. It seems more of a solution looking for a problem at the moment.