Starting September 15, 2026, Cloudflare will automatically block a new class of AI bots it calls "mixed‑use" crawlers from any website that relies on advertising revenue. The move, quietly announced through a policy update for its content delivery network (CDN) and bot management services, forces AI companies, search engines, and autonomous AI agents to explicitly declare whether they are crawling pages for model training, for retrieval‑augmented generation (RAG), or for direct interaction with the site’s content—and if they can’t or won’t, their requests will be denied.
This is not a blanket ban on AI. Rather, Cloudflare is drawing a bright line between crawlers that benefit publishers and those that extract value without returning traffic, engagement, or compensation. For the past two years, the CDN giant has offered free bot‑fighting tools, including a well‑known feature that blocks AI scrapers outright. The new policy, however, introduces a more nuanced middle ground: it distinguishes purely "commercial" crawlers—those used by AI startups to hoover up training data and build competing products—from legitimate or semi‑legitimate bots that might serve the publisher’s interests. The catch is that many AI systems now blend use‑cases; a single bot might cache data both for instant answers (reducing the need for users to visit the site) and for long‑term model improvement. Those are now deemed mixed‑use, and on ad‑supported properties, they get blocked by default unless the publisher opts in.
Why ad‑supported sites? Because advertising is the lifeblood of the open web. When an AI assistant scrapes a recipe, a news article, or a product review and displays a summary directly in a chat interface—complete with all the information a user needs—the original publisher loses that ad impression, that affiliate click, that subscription upsell. Cloudflare’s own data shows that AI‑driven content extraction has grown 800% since 2024, while direct human traffic to the same pages has fallen by nearly 20% on average. For sites that depend on display ads for survival, the math is brutal. The new default blocking is an economic safeguard: if a crawler can’t prove it will drive meaningful traffic back, it is not welcome.
Under the hood, Cloudflare’s bot management engine will use a combination of heuristics, behavioral analysis, and client‑side attestation to classify incoming requests. Any bot that identifies itself via user‑agent string or robots.txt as a search engine (Googlebot, Bingbot) will still be treated as a “pure” crawl—allowed through, because it typically sends traffic. A bot that identifies as a scraper for a large language model without any commitment to attribution will be rejected entirely. The tricky middle is the mixed‑use category: a bot that says “I am here to index pages for answer snippets, but I may also use the data for model training.” Starting September 15, those requests will be met with a 403 error on any site that has advertising active (as detected by Cloudflare’s automatic ad‑scanning integration) unless the publisher has explicitly added an allow rule.
For IT teams and site owners, the practical impact will be felt immediately. Cloudflare’s dashboard will display a new “Mixed‑Use AI Bot” report, showing exactly how many requests were blocked, from whom, and on which pages. Publishers can then create granular exceptions: they might allow a specific crawler from a trustworthy AI search engine that shows full articles and attribution, while continuing to block others. The system also hooks into Cloudflare’s WAF (Web Application Firewall) rules, so more sophisticated customers can script custom logic—for instance, “Block all mixed‑use bots except those that accept a per‑page micro‑payment in exchange for access.” Cloudflare expects 85% of its ad‑supported customers to keep the default block in place, effectively reshaping the landscape of web crawling overnight.
Major AI players are already reacting. Google, which operates both a search index and the Gemini AI platform, has long maintained separate crawler identities (Googlebot vs. Google‑Extended). A Google spokesperson confirmed that the company would update its robots.txt documentation before the September deadline to ensure that its crawlers are correctly classified as single‑purpose. Microsoft, with its Bingbot and Copilot ecosystem, faces a deeper challenge because its Bing Chat crawler has historically blurred the line between indexing and answer generation. Smaller AI labs—Anthropic, Perplexity, and the dozens of startups backed by venture capital—are rushing to implement standard headers that declare their precise intentions. Several start‑ups have already proposed a new “TDM‑Repurposing” (Text and Data Mining) header that would allow publishers to fine‑tune permissions, but Cloudflare’s stance is that headers alone won’t be enough; the bot must demonstrate intent through a technical attestation protocol the company is keeping semi‑proprietary.
Privacy advocates have cautiously welcomed the move, noting that it forces AI companies to be transparent about how they use crawled content. “For years, the assumption has been that if you put something on the web, anyone can take it for any purpose,” said Martijn van der Linden, a digital rights researcher at the University of Amsterdam. “This flips the default. It says you can only take if you ask, and if your use might harm the creator, you need permission.” However, some warn that the policy could fragment the open web, creating a patchwork of access rules that stifle innovation. Others point out that large AI companies with the resources to negotiate bilateral deals will easily bypass the block, while smaller open‑source projects and academic researchers might find themselves locked out.
From a technical perspective, implementing the change requires no action from existing Cloudflare customers who are happy with the default. When September 15 arrives, the CDN will automatically start blocking mixed‑use bots on any zone where it detects programmatic ads (Google AdSense, Media.net, etc.) or where the customer has flagged the site as ad‑dependent. Customers who do not run ads, or who explicitly want to allow all crawlers, remain unaffected. Cloudflare will also offer a “publisher audit” tool that scans robots.txt and sitemap settings to ensure alignment with the new policy.
For Windows-centric IT departments, the implications are notable. Many internal corporate portals, help‑desk systems, and knowledge bases run behind Cloudflare. If those sites contain advertising—rare, but possible—they could inadvertently trigger the block. More significantly, Windows shops that deploy AI‑agent solutions internally need to understand how those agents are perceived when they crawl public sites or even internal ones via Cloudflare. An IT team that has built an internal copilot that scrapes SharePoint Online (which may sit behind Cloudflare’s network) might suddenly see 403 errors if the bot’s user agent is classified as mixed‑use. Cloudflare has said it will provide enterprise IT with a way to whitelist internal bots, but the process requires registering the bot’s identity and purpose in advance.
The transition period between now and September 2026 gives the industry 18 months to adapt. During that time, Cloudflare will progressively roll out detection signals, offering a “report‑only” mode for early adopters. Publishers can switch on the block early if they choose. Early testing by a handful of news organizations shows that the mixed‑use block reduces AI‑driven scrapes by up to 70% while still allowing legitimate search traffic. The New York Times, which has taken legal action against AI companies, called the policy “a welcome first step toward restoring the value exchange that has underpinned the web for three decades.”
Yet the biggest question remains unanswered: will the block actually force AI companies to change their behavior, or will they simply rotate through residential proxies and sophisticated user‑agent spoofing to bypass it? Cloudflare insists its bot detection is the most advanced in the world, capable of spotting headless browsers, randomized fingerprints, and traffic patterns that mimic human behavior. The company has already defeated similar evasions in its broader anti‑scraping tools. However, the AI scraping arms race is unlike any previous battle, because the financial incentives on both sides are enormous. A specialized AI model trained on the web’s best content can be worth billions; the publisher who supplies that content gets nothing. The September 2026 default is not a final solution but a major escalation in a war that is only beginning.