Cloudflare announced a new framework for classifying and managing AI bot traffic on July 1, replacing its year-old binary “Block AI Bots” toggle with three distinct categories: Search, Agent, and Training. The controls are available to all customers, including those on the free tier. Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that display ads for all new domains onboarding to the network.
The announcement is part of Cloudflare’s second “Content Independence Day,” marking one year since the company first changed defaults to block AI training crawlers on new domains. More than 20% of the web sits behind Cloudflare’s network, making the policy shift a de facto infrastructure standard for how autonomous agents access the open web.
The Three Categories
Cloudflare’s new taxonomy defines bot behavior by what it does on a site, not whether it qualifies as “AI”:
Search covers any behavior that indexes content to answer queries later. The expectation: site owners get referral traffic or compensation in return.
Agent covers automated behavior acting on a person’s behalf in real time, including chat-fetch bots (like ChatGPT-User) and browser-use agents (like Gemini or Claude driving Chrome). The defining characteristic: a human is usually waiting on the other end.
Training covers crawlers taking content to train or fine-tune models. The data is permanently absorbed into the model’s architecture.
Cloudflare is encouraging bot operators that serve multiple purposes to split their automation into separate crawlers for each category. If they don’t, multi-purpose crawlers will be classified under all applicable behaviors and subject to the most restrictive rule. That has a direct consequence: Googlebot, Applebot, and BingBot, which blend search indexing with training data collection, will be blocked for any customer who selects to block Training.
Why Ad-Supported Pages Get Different Defaults
The September 15 defaults treat an ad on a page as a signal that the site owner intended for a human to land there and see it. On those pages, Training and Agent bots are blocked by default because they extract value without generating the human attention the ad model depends on. Search crawlers remain allowed because they are the primary mechanism that drives visitors back to the site.
Existing customers can opt out of the new defaults before September 15 through their Security settings. Cloudflare plans to notify customers as the date approaches.
The Agent Category Creates a New Control Surface
For teams building or deploying autonomous agents, the Agent classification introduces a control surface that didn’t previously exist at the infrastructure layer. Website owners can now independently decide whether to allow agents to interact with their sites, separately from whether they allow search crawling or training.
This matters for agent-to-web workflows. An OpenClaw agent fetching a product page to compare prices, a Claude agent reading documentation to complete a task, or a Gemini agent navigating a checkout flow all fall under the Agent category. If a site owner blocks Agent traffic, those workflows break, regardless of whether the agent identifies itself honestly.
Cloudflare is also experimenting with “transitive trust,” using the existing RFC 7239 Forwarded header to let agent platforms carry verified identity through intermediary layers. The format: Forwarded: for="openai";use="reference". If an operator loses trusted status across Cloudflare’s network, every site behind it becomes inaccessible, which Cloudflare describes as “a deterrent with teeth.”
Content Use Signals: Three Levels of Permission
Alongside the bot categories, Cloudflare introduced content use levels that define what a bot can keep and reshare:
- Immediate: interact but store and reuse nothing
- Reference (default): index, excerpt, and link back
- Full: summarize and reproduce
These combine with bot classifications to express rules like “allow Search bots up to reference use only.” Cloudflare is extending the Content Signals standard in robots.txt with a new use parameter. Bots that reproduce content in full cannot receive Verified status.
Cloudflare’s Internet Report: 50% of Traffic Is Now Non-Human
The announcement accompanied Cloudflare’s annual agentic internet report, which found that more than 50% of internet traffic is now non-human. Among crawlers specifically, 52% of requests are for AI training as of June 2026, up from 22% in Spring 2025. Mixed-use crawlers that blend search, agent, and training activity represent over 36% of crawler traffic.
Cloudflare’s report also noted that for every hour users spend online searching for information, only 15 minutes is spent on the open web. A 2025 Pew Research Center study cited in the report found that when Google shows an AI summary, users click a traditional search result just 8% of the time.
Pay Per Use Experiments
Cloudflare also announced experiments with two AI search companies to move from Pay Per Crawl to Pay Per Use. Ceramic.ai is testing a pay-per-query model where publishers are paid when their content appears in search results. You.com is testing on-demand payments for specific premium content. Both programs include new reporting that shows publishers the top queries leading to their content, specific snippets used, and average ranking position.
The Infrastructure Precedent
More than 50 publisher-AI licensing agreements have been signed since 2023, according to Cloudflare’s report. The company noted that Google currently has access to approximately 2x more information than leading AI companies because Googlebot’s mixed-use crawler makes it difficult for site owners to participate in Google’s search ecosystem without also participating in its AI ecosystem.
Cloudflare’s goal of driving mixed-use crawlers to zero within a year, combined with the September 15 defaults, creates a forcing function. Bot operators that don’t separate their crawlers by purpose will be treated as Training bots by default on ad-supported pages, losing access to a significant share of the web.