A 402 lands in your crawl logs. You have been writing HTTP clients for a decade and you have never seen one in production, 402 is the status code that sits in the spec beside a note saying it is reserved for future use. You check the response headers and there it is:
HTTP/2 402
crawler-price: USD 0.01That is not a bug. That is a publisher quoting you a price.
The open-crawl era ran on an implicit deal: crawlers took content, search engines sent traffic back, and everyone tolerated the imbalance. Generative AI broke the return half of that trade, a model trained on your pages does not send you a visitor. The response has been a fast, messy renegotiation, and the most concrete artefact of it is a status code from 1997 finally being wired up.
The Code That Waited Thirty Years
HTTP 402 has been in the specification since HTTP/1.1 and has never had defined semantics. RFC 9110, the current HTTP core spec, still describes it as reserved for future use. It was left as a placeholder for the digital-cash systems people assumed were coming in the mid-nineties. None of them shipped at web scale, and the code sat there as a piece of protocol trivia.
What makes 402 attractive now is that it is already in every HTTP stack. Clients, proxies, CDNs, and logging pipelines all handle unknown 4xx codes gracefully. You do not need a new protocol to express "this costs money" , you need a status code, a header carrying terms, and a way for the client to say yes. The plumbing has been sitting unused the whole time.
What Cloudflare Actually Shipped
On 1 July 2025, in an announcement it branded Content Independence Day, Cloudflare made two changes. First, new domains signing up are asked upfront whether to allow AI crawlers, with the default flipped to blocking them, an opt-in model rather than opt-out. Second, it launched a private beta of Pay Per Crawl, a marketplace where a publisher sets a per-request price and Cloudflare mediates the transaction as merchant of record.
Cloudflare has since expanded the model. Its 2026 guidance splits AI traffic into Search, Agent, and Training categories rather than treating "AI crawler" as one bucket, and from 15 September 2026 new domains have Training and Agent crawlers blocked by default on ad-supported pages while Search crawlers stay allowed. Verified-bot status was also decoupled from access: being verified now establishes who you are, not that you are let in. All of that is Cloudflare's own documentation and blog, and it applies to sites behind Cloudflare, which is a large share of the web, but not the web.
The Signalling Flow
The mechanics are worth reading closely, because they are the part most likely to outlive any one vendor's product.
A crawler makes an ordinary request. If the path is priced, the origin answers 402 with a crawler-price header naming the amount. Per Cloudflare's docs the minimum is USD 0.01 per crawl.
The crawler then retries, declaring what it will pay with one of two request headers: crawler-exact-price, matching the quote exactly, or crawler-max-price, a ceiling it will accept on any priced content. If the price is within budget, the response is a normal 200 carrying a crawler-charged header stating what was actually billed. If something is wrong, a mismatched exact price, no funding, you get another 402 with a crawler-error header naming the reason.
# 1. Unpriced request
GET /articles/2026/index.html HTTP/2
Host: publisher.example
User-Agent: ExampleBot/1.0 (+https://example.com/bot)
HTTP/2 402
crawler-price: USD 0.01
# 2. Retry declaring intent to pay (signed — see below)
GET /articles/2026/index.html HTTP/2
Host: publisher.example
crawler-max-price: USD 0.02
Signature-Agent: "https://example.com"
Signature-Input: sig1=("@authority" "crawler-max-price" "signature-agent");created=...;keyid="...";tag="web-bot-auth"
Signature: sig1=:...:
HTTP/2 200
crawler-charged: USD 0.01
The detail that makes this more than a header convention: Cloudflare requires the payment headers to be covered components of a Web Bot Auth signature. You cannot claim a price commitment in a bare header, because a bare header is forgeable by anything in the path. The transaction is bound to a cryptographic identity. That dependency is why the payment story and the signed-identity story are really one story.
Machine-Payable HTTP Beyond One Vendor
Cloudflare is not the only group reaching for 402. Coinbase published x402 in May 2025 as a general payment protocol over HTTP, now maintained under an independent foundation with implementations from several payment providers. Its shape is the same: the server answers 402 with a PAYMENT-REQUIRED header carrying base64-encoded JSON terms — accepted schemes, price, network, destination, the client retries with PAYMENT-SIGNATURE carrying an authorised payload, and the server returns PAYMENT-RESPONSE describing settlement.
The two designs differ in settlement rails and in scope, but they agree on the pattern: quote in a response header, commit in a request header, confirm in a response header, one extra round trip. If machine-payable HTTP becomes normal, that is what it will look like, and a crawler that treats 402 as a hard failure will simply be leaving content on the table.
Why Publishers Are Only Half Happy
The case against is not weak, and it deserves to be stated in full rather than waved at.
- Concentration risk. When one infrastructure provider mediates access for a large fraction of the web, its policy defaults become de facto internet policy. A per-request price is a business decision; a default-block on new domains is closer to regulation, made by a company rather than a legislature.
- Small and academic crawlers. A research group indexing a corpus for a paper is not an AI lab with a licensing budget. Per-request pricing at web scale is trivially absorbed by a funded company and prohibitive for a lab, which inverts the usual intuition about who gatekeeping protects.
- Publishers may not want the same wall twice. Blocking training and blocking retrieval are different decisions with different consequences. Many publishers want to be cited in AI answers and merely not trained on, and a coarse toll booth risks buying you the first outcome by mistake. Cloudflare's Search/Agent/Training split exists precisely because the first-generation controls were too blunt.
- A tax is not a wall. A well-funded crawler pays. What per-request pricing changes is who can crawl at scale, not whether crawling at scale happens.
What It Costs: A Model, Not a Quote
Assume a team ingesting one million pages a month from a single publisher. These are stated assumptions, not measured figures, plug in your own.
RouteRough monthly costWhat you are actually buying
The bottom row looks cheapest and usually is not, because the omitted line items are the expensive ones: parser maintenance when the HTML changes, retries and blocks, and the review meeting where someone asks what your legal basis was. Priced access is more expensive per page and considerably cheaper per unresolved question.
The honest read on the table is that for anything you plan to build a product on, the API or the licence usually wins on total cost, and self-crawling stays rational for breadth, for one-off research, and for sources that offer no other route.
What People Get Wrong
- Treating 402 as a permanent failure. Most HTTP clients bucket it with 4xx and give up. It is a quote, and the correct handling is to log the terms and decide, not to retry blindly or drop the URL.
- Assuming a block is aimed at you. A default-on policy applied to every new domain is not a judgement about your crawler. Identifying yourself and asking is a real option, and it works more often than people expect.
- Reading "Cloudflare blocks AI crawlers" as "the web blocks AI crawlers." It applies to sites behind one provider, with defaults the site owner can change.
- Confusing a training block with a search block. They are separate decisions expressed by separate tokens and increasingly by separate categories. Conflating them is the single most common error on both the publisher and the crawler side.
- Keeping no provenance. If you cannot say, per source, where data came from, under what terms, and on what date, then a licensing question becomes an archaeology project. Record the source URL, the fetch timestamp, the robots.txt state at fetch time, and the terms you relied on.
Wrapping Up
The interesting thing here is not the marketplace, which may or may not last, it is that the web is growing a vocabulary for conditional access. A 402 with terms in a header, a signed commitment in the retry, a category that says whether you are indexing or training: those are primitives, and they are being standardised in the open even where the first implementations are proprietary. Treat 402 as a real branch in your client rather than an error, budget for licensed sources on anything you are building a product on, and keep provenance records per source. "We scraped it because it was public" was a workable position for twenty years. It is no longer one that survives a review.
