Cloudflare Empowers the Agentic Internet: Your Content, Your Rules

Cloudflare is reshaping the rules of the agentic Internet, giving site owners the tools to control, monetize, and optimize their content as AI bots now generate over half of all web traffic.

The nature of Internet traffic has fundamentally shifted. Automated agents and bots now drive more than half of all web requests, and AI is becoming the primary interface for accessing information and conducting commerce online. For businesses built on advertising or subscriptions, this creates a critical dilemma: how to remain discoverable without giving away intellectual property to AI training crawlers. Cloudflare's July 2026 announcement addresses this challenge with a simple philosophy — your content, your rules.

What was announced

Cloudflare (NYSE: NET) unveiled new classifications, enhanced analytics, and industry-defining commercial partnerships designed to align site owners with transparent AI companies. The announcement introduces a deadline of September 15, 2026, when new default settings will allow search but block training and agent use for pages with ads. Mixed crawlers that do not separate search, agent use, and training will be blocked on all pages containing ads.

The company also launched the Attribution Business Insights dashboard, giving business stakeholders visibility into how AI bots consume content and how much human traffic AI companies actually return. Additionally, Cloudflare is evolving Pay Per Crawl into Pay Per Use, ensuring publishers are compensated when their content actually creates value, not just when it is fetched. Launch partners include Ceramic.ai and You.com.

Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent

Why this matters

For CIOs, CISOs, and IT directors, the shift from human to bot-dominated web traffic changes the economics of digital content and security posture. The largest search engine currently has access to roughly 2X more information than leading AI companies because customers cannot easily separate search discoverability from AI training. Cloudflare's new defaults help level this playing field.

Over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages — wasted bandwidth for publishers and wasted compute for AI companies. Cloudflare's freshness signals aim to eliminate this inefficiency. For procurement leaders, more than 50 major content licensing agreements have been signed between publishers and AI platforms over the past year, signaling that content monetization has become a strategic revenue channel.

Technical details

• New default classifications: Effective September 15, 2026, defaults allow search but block training and agent use for pages with ads.
• Mixed crawler blocking: Crawlers that do not separate search, agent use, and training will be blocked on ad-supported pages.
• Attribution Business Insights: New dashboard exposing AI bot consumption patterns and referral traffic per AI company.
• Answer Engine Optimization (AEO): Measurement layer showing how content is cited in AI-generated answers across models.
• Freshness signals: Cloudflare notifies AI crawlers when pages have changed, cutting redundant crawls that account for 50%+ of AI crawl traffic.
• Pay Per Use: Publishers earn when content creates value, integrated with Ceramic.ai (pay-per-query) and You.com (on-demand agent payments).
• Web Bot Auth: Open framework allowing AI bots and autonomous shopping agents to verify identity to publishers and merchants.

Softprom and Cloudflare

Softprom is the official distributor of Cloudflare. Enterprises looking to implement AI bot management, content monetization, and Web Bot Auth for the agentic Internet can access full support, licensing, and technical expertise through Softprom.

This content was prepared as part of the Softprom DistriFlow project — an automated system for monitoring and adapting vendor news. Original source: original article.