News organisations are moving from moaning about AI scraping to building an infrastructure for control, measurement and payment. This is your practical guide to Pay Per Crawl, licensing marketplaces, AI coalitions and more
With search traffic in free fall and AI summaries popping out everywhere, news organisations are asking some version of this question: how do we stop AI companies taking our journalism, using it to answer questions and train models, while the value of the original work flows somewhere else?
There is now a growing collection of tools and industry initiatives that can help.
They serve different purposes. Some let you block or monitor crawlers. Some create a marketplace for licensing content. Others establish technical standards or negotiate collectively on publishers' behalf. There is also growing regulatory pressure on the biggest platforms.
I set out to map out this growing sector and shed some light on what these tools are and what you can actually do with them.
Cloudflare is an internet infrastructure company whose technology sits between a website and its visitors. It has become one of the most prominent players in the fight over AI crawling.
Its Pay Per Crawl system gives publishers options to block AI crawlers, allow particular crawlers, let them access content for free or charge them for access. Publishers set the terms and prices, while AI companies can see the price and decide whether to pay.
Cloudflare also offers a simple "Block AI Bots" option. In July 2026, it announced further controls for different types of automated traffic, including search, AI agents and training crawlers.
Who is it for? Publishers and website owners with significant AI crawler traffic.
What is the aim? Give publishers visibility and control over automated access, with the option to turn access into revenue.
Who owns it? Cloudflare is a publicly listed technology company.
How do you use it? Cloudflare customers can use its bot-management and AI controls; Pay Per Crawl is currently available through an early-access/private-beta route.
TollBit provides publishers with a platform for monitoring, controlling and monetising AI access.
It identifies AI and agent traffic, separates it from human traffic and allows publishers to establish rules and rates for content access. Its marketplace currently lists more than 8,000 publishers available for AI licensing.
Think of TollBit as infrastructure for a market in which AI companies can find content they want and publishers can set the terms for using it.
