AI Agent Traffic Is Not Crawler Traffic: The Distinction Sites Are Learning to Make

Two entries in the same access log, an hour apart.

The first is 40,000 requests over six hours, breadth-first across every URL in the sitemap, four concurrent connections, no referrer, no repeat visits, a user-agent naming a model vendor. The second is six requests in ninety seconds: a landing page, a pricing page, two doc pages, back to pricing, then a support form — in an order that reads like someone actually looking for something.

Your bot rule categorised both as "AI bot" and returned 403 to both. The first one was a training crawler. The second was a customer's assistant, and there was a person sitting behind it watching a spinner.

That collision is the current state of the web's access-control vocabulary. It has one word, bot, for things with almost nothing in common.

Four Kinds of Non-Human Traffic

The industry has converged on a taxonomy over the last two years, and it holds up because the categories differ in ways you can actually observe.

Two properties do most of the work. Session shape: a crawler traverses a graph, an agent walks a path. And whether a human is waiting: it changes what a delay costs, what a block costs, and whether the visit could plausibly convert into anything.

That second property is the one that should make site owners nervous about blanket rules. A blocked training crawler is a policy outcome. A blocked retrieval fetcher is a person who asked a question about you and got told nothing.

The Naming Already Encodes the Split

You don't have to infer any of this. The major vendors have already split their user-agents along exactly these lines, because publishers demanded the ability to say yes to one and no to another. The full agent-by-agent breakdown is in the earlier post on robots.txt for AI crawlers; the short version is that OpenAI documents GPTBot for model training, OAI-SearchBot for surfacing sites in ChatGPT's search features, and ChatGPT-User for actions initiated inside ChatGPT and Custom GPTs. Anthropic documents ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for fetches a person triggered by asking Claude something.

Three tokens, three intents, one company. That separation only exists because "may you use this" and "may you fetch this for the person asking" turned out to be different questions.

The Sentence That Contains the Whole Problem

Buried in OpenAI's bot documentation, describing ChatGPT-User, is this: "Because these actions are initiated by a user, robots.txt rules may not apply."

Read it twice. It is not evasive and it is not unusual, it's a position, stated openly, that a fetch a human explicitly asked for isn't the thing robots.txt was written to govern.

Google has held a structurally identical position for years. Its crawler documentation splits into common crawlers, special-case crawlers, and user-triggered fetchers, and says of the last group that because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. That's not an AI policy. That's how Feedfetcher and Site Verifier have behaved for a very long time.

Anthropic takes the other side. Its support documentation states its crawlers respect "do not crawl" signals by honouring standard robots.txt directives, and it lists Claude-User among the agents that do.

So two large vendors read the same protocol differently for the same category of request, and both readings are defensible. RFC 9309 formalised the Robots Exclusion Protocol in 2022 as a mechanism for automatic clients, and "automatic" was never defined against a case where a person types a URL into a chat window and a program fetches it two hundred milliseconds later.

The Delegation Question

Strip away the vendor politics and the unresolved question is short: if a user asks an agent to read a page the user could have read themselves, whose access is that?

The case for "the user's": the page was going to be fetched either way. The person is present, waiting, and consuming the result. An agent here is a user-agent, the same category as a browser, a screen reader, or Reader Mode. Nobody has ever argued that Firefox needs its own robots.txt entry.

The case for "the operator's": the fetch happens from the operator's infrastructure, at the operator's IP, under the operator's product. It can be repeated, cached, logged, and reused across users. The origin has no way to confirm a human was involved, so "a user asked me to" is an unverifiable assertion, precisely the class of claim that identity mechanisms exist to fix.

Terms of service are no help. Most of them contain a line prohibiting automated access, drafted years before an assistant that fetches one page for one person was a thing anyone shipped. Applied literally, that clause bans a blind user's screen reader as readily as a scraper. Applied sensibly, it needs a distinction the text doesn't contain.

There is no settled answer, and anyone telling you otherwise is describing their preference. What exists is a set of building blocks being assembled while the argument continues.

What Sites Are Actually Doing

Cloudflare's decisions are worth tracking here, not because they are the only view, but because they mediate a large enough share of traffic that their defaults become de facto norms.

In July 2025 they began blocking AI training crawlers by default for new domains, covered in the post on pay-per-crawl and the licensed web. In July 2026 they replaced the single AI-bot switch with three, splitting Search, Agent and Training into separately configurable categories for all customers including the free tier, with Training and Agent blocked by default on ad-carrying pages for new domains and Search left allowed. Their definition of the agent category is the useful part: automated behaviour acting in real time on a person's behalf to get something done now, including chat fetch bots and browser-driving agents, where "often there's a human waiting on the other end."

Their published data explains the urgency. Cloudflare reports that AI training's share of crawler requests rose from roughly 22% in spring 2025 to about 52% by June 2026, that mixed-use crawlers, a single bot doing both search and training, account for over 36% of crawler activity, and that non-human traffic has crossed half of all internet traffic. These are one network's measurements of its own edge, not an independent census, but the direction is corroborated everywhere.

Mixed-use is the awkward middle. If one user-agent serves two purposes and you only want to permit one, the protocol gives you no way to say so, which is why the pressure has moved from declaring intent in robots.txt toward proving identity per request.

The Building Blocks

Three pieces are being assembled, each covered properly in its own post:

• Signed identity. HTTP Message Signatures and the Web Bot Auth drafts let a request carry cryptographic proof of who sent it, replacing an unverifiable user-agent string. Cloudflare announced Web Bot Auth in May 2025 and launched signed agents in August 2025 with an initial cohort that included ChatGPT agent, Block's Goose, Browserbase and Anchor Browser.
• Agent-specific declarations. Separate tokens for training, search and user-initiated retrieval, plus the IETF's in-progress AI-preferences work on stating usage permissions.
• Per-request payment signalling. HTTP 402 wired up for real, so an origin can quote terms instead of choosing between yes and no.

None of these answers the delegation question. All of them make it answerable, by giving an origin something better than a self-declared string to base a decision on.

If You're Building Something Agentic

The practical guidance is unglamorous and it works. Sites are already allowlisting products that do this and blocking ones that don't.

Declare what you are. Don't wear a Chrome user-agent, a real product token with a contact URL is the single highest-value thing you can ship:

import httpx
from urllib.robotparser import RobotFileParser

HEADERS = {
    "User-Agent": "AcmeAssistant/1.2 (user-initiated; +https://acme.example/agent)",
    "From": "agents@acme.example",  # RFC 9110's contact field, still perfectly valid
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}

rp = RobotFileParser("https://example.com/robots.txt")
rp.read()
allowed = rp.can_fetch("AcmeAssistant", "https://example.com/article")
print("robots allows:", allowed, "| crawl-delay:", rp.crawl_delay("AcmeAssistant"))

if allowed:
    with httpx.Client(headers=HEADERS, timeout=20.0, follow_redirects=True) as c:
        r = c.get("https://example.com/article")
    print(r.status_code, "| retry-after:", r.headers.get("retry-after"))

Beyond that:

• Pace at human scale. One person's session is one person's session. If your agent fetches forty pages to answer one question, you have built a crawler with a chat interface.
• Cache, and don't re-fetch. Conditional requests with ETag and If-Modified-Since cost the origin almost nothing. Ten users asking about the same page should not be ten full fetches.
• Honour the stop signals. Retry-After on a 429, a challenge response, a Disallow, treat each as an instruction, not an obstacle.
• Publish a contact channel and stable egress. A page describing your agent, an email that a human reads, and either published IP ranges or a signed identity. Site operators block what they can't identify, because it's the only lever they have.
• Prefer the API. If there's an official endpoint, the fetch you were about to do is worse for both parties.

What People Get Wrong

• Treating "AI bot" as one traffic class. Training and user-initiated retrieval differ by roughly six orders of magnitude in volume and by everything in intent.
• Assuming a user-agent string means anything. It's an unauthenticated claim. This is the entire reason signed identity is being standardised.
• Assuming user-initiated means unconstrained. One vendor documents that robots.txt may not apply to user-triggered fetches; another documents that it does. That's a live disagreement, not permission.
• Blocking everything non-human and calling it a policy. It also blocks the fetch that would have shown your pricing page to a buyer.
• Waiting for the standards to settle before identifying yourself. A product token and a contact URL are an afternoon's work and are already the difference between allowlisted and blocked.

Wrapping Up

The distinction that matters is not human versus bot, it's whether anyone is waiting. A training crawler pulling your entire archive and an assistant fetching one page for one curious person are separated by volume, pacing, session shape and consequence, and until recently the web had no vocabulary to treat them differently. It's acquiring one now: separate user-agent tokens per intent, separate policy categories at the edge, signed identity per request, and price signalling for the cases where the answer is "yes, but". The norms are being written this year, in vendor documentation and CDN defaults rather than in any standards body's final draft. If you build agentic products, the cheap and durable position is to be identifiable, be paced, be cacheable, and be reachable, because the operators deciding what to allow are making those calls right now, and they're making them from your logs.