When AI Comes to Visit

A website asks you to prove that you are human. Beneath the request, an explanation: automated systems have been collecting content so aggressively that the site needs protection.

Elsewhere, a retailer is working to make its products easier for AI to discover.

Both responses can be rational.

Tools such as Anubis exist specifically to protect websites from scraper traffic. Their own documentation acknowledges the compromise: a protective barrier can also obstruct useful automated visitors. What looks like a small inconvenience on a screen reflects a larger question about the economics of the web.

For a publisher, the information on the page may be the product. For a retailer, the page helps sell something else. An AI system that answers a question without sending the reader onward can weaken the first business while helping the second reach a buyer.

The dividing line is where the value goes.

The Exchange Behind the Visit

Much of the web grew around an imperfect but useful exchange. Search engines indexed pages. People followed links. Websites gained readers, advertising opportunities, subscribers, customers, or contributors.

AI changes that exchange when it uses the source to complete the user’s task elsewhere.

A reader may get the explanation without visiting the publication. A shopper may compare products without seeing the retailer’s carefully designed homepage. A professional may receive a synthesis of specialist research without opening the underlying reports.

The source can remain useful while becoming less visible.

There is evidence of this change in search behavior. Pew Research Center analyzed browsing data from 900 U.S. adults for March 2025. Traditional result links received clicks on 8% of visits to Google search pages with an AI summary, compared with 15% without one. Links within the summaries were clicked on in just 1% of visits to pages containing them.

This was an observational study, with search results reconstructed afterward. It does not establish that every publisher loses the same amount of traffic or revenue. For businesses that depend on visits, however, the issue is concrete: being used as a source and earning a customer interaction are different outcomes.

Protection, Negotiation, and Participation

Reddit illustrates why the story cannot be reduced to companies that accept AI and companies that reject it.

In June 2025, Reddit announced a lawsuit alleging wrongful use of its content by Anthropic. Those accusations are allegations, not findings of liability. Yet Reddit had already announced a partnership with OpenAI in May 2024, including access to its Data API and opportunities to bring Reddit content into AI products. Its position combines restrictions with negotiated participation.

The strategic question is not simply whether AI should have access. It is access for which purpose, under whose conditions, and with what return.

There is also a cost beyond lost commercial attention. Wikimedia reported in April 2025 that multimedia download bandwidth had increased by 50% since January 2024, largely because of automated scraping for AI. Its March 2026 follow-up described continuing pressure on infrastructure and efforts to protect resources.

An organization can support open knowledge while objecting to how that knowledge is collected. Free content still requires funded infrastructure.

A third response is to seek payment for automated access. Cloudflare introduced Pay Per Crawl in July 2025, offering a framework for allowing, charging, or blocking crawlers. The initiative demonstrates an attempt to create an additional exchange; it does not prove that charging for access will sustain every publication.

These examples point to three distinct concerns: protecting the economic value of content, controlling the terms of reuse, and keeping the service available.

When the Visitor Might Bring a Buyer

For sellers of goods and services, AI can play a different role.

A customer asks for equipment suitable for a particular application, a supplier that serves a specific port, or a service provider with the right capabilities. If an assistant finds accurate information and directs that customer toward a suitable business, the automated interaction can create demand.

Commerce platforms are building for this possibility. Shopify’s agentic storefronts connect eligible merchants to AI shopping channels, using catalog information to support discovery and purchases. Its documentation distinguishes referral experiences from direct checkout, depending on the channel and configuration.

For the seller, the opportunity is to appear at the moment a customer describes a need.

But discovery is only the beginning of the evaluation. Leaders should ask whether these channels produce profitable orders, suitable inquiries, repeat customers, and manageable service obligations. A new intermediary can create demand while also influencing comparison, presentation, and access to the customer.

Visibility has value when it leads to a worthwhile business outcome.

The Behavior Behind the Traffic

There is another complication for businesses that learn from what visitors do.

A human opening several product pages may be expressing uncertainty, preference, or purchase intent. An automated assistant opening those pages may be gathering information for a comparison. A bulk scraper may have no customer behind it at all.

Treating these actions as equivalent can distort the picture.

This is an analytical implication, rather than a claim that all companies using behavioral data oppose AI access. A business can welcome an agent’s contribution to a sale while recognizing that its browsing sequence is a different kind of signal.

The same issue reaches beyond analytics. When an assistant handles exploration, the business may have fewer opportunities to explain its strengths, introduce complementary services, invite a subscription, or learn directly from the customer’s questions.

An order can arrive even as part of the relationship moves elsewhere.

First, Identify the Kind of Access

“AI traffic” is too broad a category for a useful policy.

Some automated systems collect material for model training. Others build search indexes or retrieve information for answers. Others act on a person’s immediate request, potentially helping that person choose or buy.

OpenAI, for example, documents separate controls for GPTBot, associated with potential training use, and OAI-SearchBot, associated with search discovery. It also describes ChatGPT-User as handling certain user-initiated visits, for which robots.txt rules may not apply.

That separation is useful, but it is not universal. Review each provider’s current documentation and the actual behavior of the traffic before deciding what to permit.

A policy should identify the purpose of access, the content involved, the expected benefit, and the acceptable cost.

If Your Content Is the Product

Start by identifying what must remain economically sustainable. Original reporting, proprietary research, specialist databases, and community contributions may require different treatment from public introductions and promotional material.

Set the business boundary. Decide which material should support discovery, which should require a subscription, and which might be supplied through a negotiated agreement. Where licensing is appropriate, define permitted uses, attribution, update obligations, and compensation with the relevant specialists.

Give people a reason to return. Invest in continuing reporting, useful tools, distinctive judgment, community participation, and direct relationships. A newsletter or membership can help preserve a connection, but it needs value beyond access to facts someone else can summarize.

Enforce the policy proportionately. Ask the technical team to combine published crawler rules with access controls, monitoring, caching, and limits on abusive request rates. The robots.txt file communicates instructions to cooperative systems; it is not a security barrier or a mechanism for removing material already collected.

Measure the result. Track infrastructure costs, human readership, subscriptions, useful referrals, and any access revenue. Test whether protective measures also prevent legitimate readers or valuable discovery.

Blocking traffic can protect a resource. It does not, by itself, rebuild the business that resource supports.

If Discovery Helps Sell Your Product

Start with information quality.

Describe what a customer needs to know. Publish clear specifications, compatibility, availability, service areas, delivery conditions, returns, and meaningful limitations. For a service business, explain capabilities and how to request a quotation. Avoid making every useful answer depend on a sales conversation.

Keep commercial facts consistent. A product page, catalog feed, and checkout should agree. An assistant recommending an unavailable item or misunderstanding a specification can create an expensive disappointment.

Make discovery technically possible. Ensure intended crawlers can reach public pages through the site’s hosting and security layers. Keep important facts in readable text, use appropriate structured data, and maintain relevant product feeds.

Google’s published guidance says its AI search features do not require special AI files or special schema markup. Existing search fundamentals still apply, and eligibility does not guarantee inclusion.

Evaluate the channel commercially. Pilot with a manageable set of products or services. Track qualified inquiries, completed orders, margin, cancellations, and returns. Check how the assistant represents the business and whether the customer can reach a person when necessary.

Machine readability should support a reliable customer experience.

If You Are in the Middle

Many businesses occupy both positions.

A consultancy wants its services discovered but may protect its paid research. A retailer wants its catalog accessible while resisting uncontrolled extraction of customer reviews. A trade association wants public events found while keeping member information private.

The sensible unit of decision is often a section of the site or a specific use.

Public product and service pages can welcome useful discovery, with reasonable limits on request volume. Their purpose is to help suitable customers find the offer.

Introductory articles and public summaries can allow selected access, while the business evaluates whether that access delivers recognition, attribution, and useful referrals.

Premium research and specialist datasets may require authenticated access or negotiated terms. Here, the business is protecting the value it sells.

Accounts, private records, and member areas need enforced authorization. Information entrusted to the organization should remain within the boundaries of that trust.

Reviews and community contributions deserve a separate decision. Their value comes from people, and permitted uses should reflect the organization’s commitments to those contributors.

These are starting points for judgment, not universal rules. A public page can still have restrictions on automated reuse; a permitted crawler still needs reasonable operating limits.

A Practical First Month

The first week should establish what is happening: which automated visitors arrive, what they request, what they cost, and which business outcomes can be connected to them. Server records and website analytics provide different parts of that picture.

In the second week, map the content and agree on access policies. Marketing, editorial or product teams, technology, and whoever owns commercial terms should make the decision together.

In the third week, run a limited pilot. Improve discovery for one product range, protect one valuable collection, or test one authorized access arrangement. Verify the experience for real customers, including accessibility and support.

In the fourth week, compare outcomes with the baseline. Keep changes that improve value, revise those that create friction, and record where attribution remains uncertain. Avoid counting every unidentified visit as AI or every AI citation as a sale.

The goal is a policy the business can explain, operate, and review.

What the Visit Is Worth

AI makes a long-standing question harder to avoid: what does the business need from someone who uses its website?

For one organization, the answer is attention. For another, it is a purchase. For a third, it is a contribution, a subscription, or payment for access.

The same automated visitor can help one business and weaken another. It can even do both within the same company.

Leaders need to understand the exchange, choose the terms, and measure whether the value returns.

Before opening the door—or closing it—decide what the visit is worth.