You email a site operator to ask why your crawler is being blocked. The reply is polite and completely reasonable: send us your IP ranges and we'll allowlist you.
You do not have IP ranges. You have an autoscaling group in three regions whose egress addresses change on every deploy, a NAT gateway shared with everything else the company runs, and a plan to move half of it to a serverless runtime next quarter. The only stable thing you can offer is a User-Agent string, which is the one thing they have no reason to believe.
That gap — between operators who need to know who is talking to them and clients who have no way to prove it — is what the IETF's Web Bot Auth work exists to close.
Why IP Identity Is Running Out
Forward-confirmed reverse DNS works, and it works because a handful of very large crawlers own stable IP space and authoritative DNS zones. Every assumption in that sentence is now failing at the edges:
- Cloud egress is ephemeral and shared. An address you emit from today belongs to someone else's workload tomorrow. Publishing it as your identity is meaningless, and delegating a PTR record for it is usually not even possible.
- CDNs and intermediaries sit in the middle. If your traffic leaves through a fronting proxy, the address the origin sees identifies the proxy, not you.
- Agents run on user devices. An assistant fetching a page on behalf of the person sitting in front of it comes from a residential connection. There is no operator IP to verify, and there never will be.
- Allowlists do not scale. A handful of search crawlers is a list. Thousands of agents, tools and research crawlers is a database with an operations team, maintained separately by every origin on the web.
Under those conditions the only identity that survives is one the client carries with it and can prove on every request.
RFC 9421: Sign Components, Not Bytes
The substrate is HTTP Message Signatures, standardised as RFC 9421 in 2024. It defines two headers: Signature-Input, naming what was signed and under what parameters, and Signature, carrying the signature value. Both are structured field dictionaries, so a message can carry several independent signatures keyed by label.
The design decision that makes it usable is that it does not sign the raw message. It signs a signature base: a canonical, line-per-component text derived from a list of named components the signer chooses. Components can be header fields (signature-agent, content-digest) or derived values written with a leading @ (@authority, @target-uri, @method).
That indirection is the whole point. A real HTTP request does not arrive at the origin as the bytes you sent. Proxies rewrite hop-by-hop headers, add Via and Forwarded, normalise header casing, re-chunk bodies, and translate between HTTP/1.1 and HTTP/2. A byte-level signature breaks on all of it. By signing only named components, the parts you actually care about survive the trip and the noise in between is allowed to change.
The tradeoff is explicit and worth stating: anything you do not list is not protected. A signature over @authority alone says nothing about the path, the method, or the body. If the body matters, you add a Content-Digest header and cover that.
What a Signed Bot Request Looks Like
Web Bot Auth layers a small profile on top of RFC 9421. It adds a Signature-Agent header — a structured dictionary whose values are strings containing the URI of the signer's key directory — and requires a tag="web-bot-auth" parameter so origins can tell a bot-identity signature apart from every other use of message signatures.
GET /articles/2026/index.html HTTP/2
Host: publisher.example
User-Agent: ExampleBot/1.0 (+https://crawler.example/bot)
Signature-Agent: agent1="https://crawler.example"
Signature-Input: sig1=("@authority" "signature-agent";key="agent1")\
;created=1764115200;expires=1764115500\
;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"\
;alg="ed25519";tag="web-bot-auth"
Signature: sig1=:Zx9Kk...trimmed...Qg==:
Reading it back: this request covers the authority it was sent to and the directory URI it declared, was created at a stated time, expires five minutes later, and was signed by a key whose identifier is a JWK thumbprint. The drafts require at least one of @authority or @target-uri to be covered, otherwise a signature captured from one host could be replayed against another, and require the signature-agent member to be signed whenever it is present, so nobody can swap the directory pointer in flight. created/expires bound the replay window; an optional nonce narrows it further.
Building the Signature Base
Here is the base being constructed and signed. Nothing about this is exotic, it is string assembly, a hash, and an Ed25519 signature.
import base64
import hashlib
import json
import time
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
def b64u(raw: bytes) -> str:
return base64.urlsafe_b64encode(raw).decode().rstrip("=")
private_key = Ed25519PrivateKey.generate() # persist this; it is your identity
public_raw = private_key.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
# keyid is the RFC 7638 JWK thumbprint: canonical JSON, required members, sorted.
jwk = {"crv": "Ed25519", "kty": "OKP", "x": b64u(public_raw)}
keyid = b64u(hashlib.sha256(
json.dumps(jwk, separators=(",", ":"), sort_keys=True).encode()
).digest())
authority = "publisher.example"
directory = "https://crawler.example"
created = int(time.time())
params = (
'("@authority" "signature-agent";key="agent1")'
f';created={created};expires={created + 300}'
f';keyid="{keyid}";alg="ed25519";tag="web-bot-auth"'
)
# One line per covered component, @signature-params always last, LF separators.
signature_base = "\n".join([
f'"@authority": {authority}',
f'"signature-agent";key="agent1": "{directory}"',
f'"@signature-params": {params}',
])
signature = base64.b64encode(private_key.sign(signature_base.encode())).decode()
print(f'Signature-Agent: agent1="{directory}"')
print(f"Signature-Input: sig1={params}")
print(f"Signature: sig1=:{signature}:")
print("\n--- signature base ---\n" + signature_base)Two things to notice. The key ID is derived from the key, not assigned by anyone, a thumbprint, so it is stable, collision-resistant, and requires no registry to mint. And the @signature-params line is the parameter string repeated verbatim inside the base, which is what stops an attacker from taking a valid signature and re-labelling it as covering different components.
Use a real structured-fields library for production serialisation rather than f-strings; the canonicalisation rules have corners, and a signature that fails to verify because of a stray space is a miserable afternoon.
The Directory: From Key ID to Known Operator
A signature proves the holder of a key sent the request. It does not tell an origin who that is. The companion draft defines the missing half: a well-known directory, served at /.well-known/http-message-signatures-directory, returning a JWKS of the keys an operator currently signs with. The Signature-Agent header points at it, the keyid selects within it.
A further registry draft adds a Signature Agent Card: a JSON document where an operator describes itself, identity, purpose, expected rate, contact, keys, so an origin resolving an unfamiliar key ID gets something more useful than a public key. That is the layer that turns cryptography into a decision an operator can actually make.
Identity Is Not Authorisation
The architecture draft is deliberate about its scope: it proves who is asking. It says nothing about whether they may have the thing.
This looks like an omission and is the opposite. Authorisation is where all the disagreement lives, whether training is different from retrieval, whether an agent inherits its user's rights, what a paywall means for a machine. Bundling any of that into the identity layer would have guaranteed the identity layer never shipped. Keeping them separate means an origin can verify a signature with a standard library and then apply whatever policy it likes.
Cloudflare's deployment shows the split working in practice. In its 2026 model, being a verified bot establishes identity only; whether you are served depends on your classification and the site owner's policy. You can be perfectly, cryptographically identified and still be turned away, and that is the design functioning correctly.
Delegation: A Crawler and an Assistant Are Not the Same Thing
The unresolved question is not cryptographic. A training crawler pulling ten million pages and a user's assistant fetching one page because that user asked for it are both "automated traffic", and today's vocabulary flattens them into the same category.
If a person asks an agent to read a page they could have opened themselves, whose access is it? The user's, exercised through a tool, or the agent operator's? Sites answer differently, terms of service mostly do not answer at all, and the answer determines whether blanket AI-crawler blocks are catching a lot of legitimate user-initiated traffic as collateral. The signalling primitives being built, separate identities for training versus user-triggered fetching, and Forwarded-style transitive trust so an origin can see the original requester behind an intermediary, are attempts to give sites enough information to decide. They do not decide it for them.
Rotation, Revocation and Directory Trust
The unglamorous parts will determine whether this works.
- Rotation. Directories are meant to hold multiple current keys so an operator can publish a new one before retiring the old. Origins that cache aggressively will reject rotated keys unless the cache policy is right.
- Revocation. A compromised key needs to stop being honoured faster than a cached JWKS expires. Removing it from the directory is the mechanism; the propagation delay is the risk.
- Directory trust. Anyone can stand up a directory and sign requests. That makes you consistent, not trusted. Reputation still has to come from somewhere, a CDN programme, a registry, or an origin's own experience of your behaviour.
- Not a rate limit. A verified identity that hammers an expensive endpoint is a verified problem.
What People Get Wrong
- Thinking a signature grants access. It answers "who", not "may they". Expect to be identified and still blocked.
- Assuming the whole request is protected. Only listed components are covered. Path, method and body are unsigned unless you name them.
- Treating the drafts as final. RFC 9421 is stable; the Web Bot Auth documents on top of it are Internet-Drafts in an IETF working group chartered in 2026, and details including header shapes and directory format can still change. Build behind an interface.
- Waiting for the standard to settle. Generating an Ed25519 key, publishing a directory and signing requests is an afternoon's work with libraries that already exist.
- Skipping the boring identity work. A documented user-agent with a contact URL costs nothing and still resolves more blocks than anything cryptographic.
Wrapping Up
The direction of travel is not subtle: identity on the web is moving from where you came from to what you can prove, and being unidentifiable is shifting from the default state to the anomalous one. RFC 9421 gives you signatures that survive real intermediaries, the Web Bot Auth drafts give an origin a way to turn a key ID into a known operator, and the standard's refusal to touch authorisation is what makes it deployable. Generate a key, publish a directory, sign your requests, and keep the honest fundamentals underneath it, a real user-agent, a contact URL, and traffic that behaves the way you said it would. The drafts will move; none of that will be wasted.
