Who controls your online content?
AI bots are collecting data from websites to train their tools, but most websites never get asked or paid. Cloudflare’s CEO Matthew Prince is speaking out against this, saying it’s time for big tech to stop treating online content like it’s free for the taking.
He wants to change the rules so websites can choose who gets to use their content and get paid for it if they agree. Right now, most don’t have that option.
Cloudflare wants fair AI competition
Matthew Prince is meeting with U.K. regulators to call out Google’s huge advantage in AI. He says Google isn’t playing fair because it uses tools others don’t have access to. Prince wants regulators to step in and level the field so newer AI companies can compete.
Without changes, Google could dominate AI the same way it dominates search, which Prince believes would hurt smaller players and limit innovation across the internet.
How web crawlers collect your data
Web crawlers are digital tools that scan websites to collect information for search engines, and now for AI models too. Google uses its main crawler, Googlebot, to collect information for both search and AI products.
This dual purpose means Google gathers way more data than most companies can. Many websites don’t even know their content is being collected for AI use, and if they want to stop it, they risk losing visibility in search.
Google’s 2-in-1 crawler power
Unlike other AI companies that need permission or payment to access website content, Google uses its existing crawler for everything. That gives Google a massive head start because it doesn’t need a separate tool or extra agreements.
This combo of search and AI crawling lets Google tap into content no one else can reach so easily. It’s like running two businesses on the same track while everyone else is still building theirs.
Blocking Google comes at a cost
Websites that depend on Google Search for traffic can’t afford to block Google’s crawler. If they do, their site drops in search rankings, and their visitor numbers take a big hit. That can slash ad revenue and hurt business overnight.
Matthew Prince says this pressure forces websites to allow Google to use their content for AI, even if they don’t agree with it. For many small and medium sites, saying no just isn’t realistic.
Opt-out tools don’t do enough
Google offers something called Google Extended, which lets sites opt out of AI training without leaving Search. But this setting doesn’t fully block all AI-related use of content, like summaries and previews in AI features.
Some media companies want to stop AI use entirely but keep their search traffic. Right now, that’s not possible with Google’s current setup, and it puts websites in a tough spot with limited control over how their content is used.
Ads can break if you say no
Google’s crawler doesn’t just power search; it also checks website content to make sure ads are shown safely. If a site blocks the crawler, Google might stop placing ads on that site, cutting off ad revenue.
Matthew Prince says this creates a trap where sites can’t say no to Google without hurting their entire business. That makes it nearly impossible for websites to push back against Google’s use of their content for AI.
Cloudflare stays neutral on AI
Cloudflare isn’t building its own AI tools, which gives it a unique place in the conversation. The company supports websites with security and speed, but doesn’t benefit directly from using content for AI.
That’s why Matthew Prince believes Cloudflare is in the right spot to speak out. He says they’re in the middle of the action, helping both media and AI companies, but without a hidden agenda. That neutrality adds weight to their arguments.
Most AI companies use Cloudflare
Cloudflare serves about 80% of the AI companies on the market, giving it insight into how the industry works behind the scenes. While Cloudflare doesn’t train AI models, it sees how data flows and how companies try to get access to content.
This bird’s-eye view helps Cloudflare understand where the power lies and where things might be off balance. It’s one reason Matthew Prince is pushing for smarter and fairer AI regulations.
Selling content access through bots
Cloudflare launched a new system that lets websites charge AI bots for access to their content. It’s like putting up a toll booth for web crawlers. If an AI company wants to use your data, they have to pay.
This idea gives power back to the websites, letting them decide who can use their content and at what price. It’s a simple fix that could make AI development more ethical and sustainable for content creators.
UK regulators are taking notice
The U.K.’s Competition and Markets Authority recently gave Google a special status due to its size and influence. This move allows them to consider tougher rules that go beyond ads and search, and into AI as well.
Cloudflare sees this as a good sign. Prince believes regulators are finally recognizing the new power struggles forming around AI, and are willing to act before it’s too late. It’s a shift that could spark global changes.
Why more AI players matter
Matthew Prince wants a market where thousands of AI companies can buy data from thousands of content creators. That way, no single company controls the flow of online information.
He argues that Google’s current model shuts out competition by giving itself exclusive access to valuable content. If the rules change, it could open the door for smaller AI developers and help websites earn a fair share when their work is used to train AI.
Media companies feel trapped
Neil Vogel, CEO of People, Inc., says media outlets are stuck in a no-win situation. They can’t afford to block Google because search brings in too much traffic, but they also don’t want their articles used for free in AI tools.
He believes Google’s system gives publishers no real choice. By tying search and AI together, Google forces companies to play along or risk losing big chunks of their business, especially their ad revenue.
Some publishers push back
People, Inc. is using Cloudflare’s AI-blocking tools to stop crawlers that don’t pay. This move has sparked conversations with top AI firms about creating fair deals for content use. It shows that media companies are starting to push back, demanding value in return for their work.
Cloudflare’s system gives them a tool to stand up without losing access to search traffic. It’s a sign that some companies are ready to fight for their rights.
Google says options already exist
Google says it gives website owners ways to block AI while staying in Search, pointing to settings like Google Extended. But critics say those tools don’t cover everything and still leave content open to use in features like AI Overviews.
This creates confusion and frustration for publishers who want clear choices. They don’t feel confident that opting out fully protects their work, and many believe Google still holds all the power in the setup.
Curious how Google’s AI actually helps users? Take a look at how Pixel AI saves time in real life.
What new rules could change
If regulators force Google to unbundle its crawler, it could completely reshape how AI companies access data. Instead of one company grabbing everything for free, each would need to negotiate or pay for the content they use.
That could lead to a more competitive, fair system for both tech companies and content creators. Cloudflare’s Matthew Prince believes this is the moment to act, before AI becomes another space dominated by just one player.
Want to see how Google’s using AI to boost online safety? Check out what’s coming to the web and Android.
Do you think Google should be required to split its crawler for fairness, or is it just a smart strategy? Share your take in the comments and drop a like if this got you thinking.
Read More From This Brand:
- Is Google Gemini on iPad the end of Siri?
- AI just started making social norms
- How students use AI to stay on track
Don’t forget to follow us for more exclusive content right here on MSN.
This slideshow was made with AI assistance and human editing.
This is exclusive content for our subscribers.
Enter your email address to instantly unlock ALL of the content 100% FREE forever and join our growing community of smart home enthusiasts.
No spam, Unsubscribe at any time.
Lucky you! This thread is empty,
which means you've got dibs on the first comment.
Go for it!