Verified Bots: How Googlebot Proves It's Googlebot

Your access log shows Googlebot/2.1 pulling four hundred pages a minute out of a faceted search route that no search engine has any reason to touch. Blocking it feels dangerous, deindexing yourself is a bad afternoon. Letting it run feels worse.

Both instincts are wrong, because the question you are asking is unanswerable from the log line in front of you. A User-Agent string is a claim. It is a piece of text the client chose to send, with no more authority behind it than a name written on a form. The entire good-bot ecosystem, allowlists, crawl-rate exemptions, WAF rules that wave search engines through, rests on a verification step, and it is a step a lot of production systems quietly skip.

A User-Agent Is a Claim, Not a Credential

Impersonating Googlebot has always been worth doing, because sites treat Googlebot differently on purpose:

• Content that is gated for humans and open for indexing. Plenty of sites serve the full article to crawlers and a prompt to everyone else. If the gate keys off the User-Agent, the gate is a suggestion.
• Rate limits and bot rules that allowlist search engines. A crawler wearing Googlebot's name inherits an exemption written for someone else.
• Scanners hiding in expected noise. Vulnerability scanning traffic labelled as a search engine is less likely to get a second look from whoever reads the logs.
• Competitive scraping of sites that specifically permit search indexing and prohibit everything else.

None of that is exotic. It is a one-line header change, which is exactly why the check on the other side has to be something a header cannot fake.

Forward-Confirmed Reverse DNS

Google's documented answer, unchanged for years, is a DNS round trip:

1. Take the IP address from your logs and run a reverse lookup (a PTR query) to get a hostname.
2. Confirm the hostname is under googlebot.com, google.com, or googleusercontent.com. Common crawler traffic looks like crawl-66-249-66-1.googlebot.com.
3. Run a forward lookup on that hostname.
4. Confirm the address it returns is the same IP you started with.

If all four steps hold, the request came from an address whose reverse record claims Google and whose forward record, published in a zone only Google controls, agrees. That is a verified bot. If any step fails, you have a client wearing a costume.

Why the Round Trip Is Not Optional

This is the part people skip, and skipping it makes the check worthless.

Reverse DNS is delegated. When a registry assigns an IP block, it delegates the corresponding in-addr.arpa zone to whoever holds the block. The owner of an IP address controls that address's PTR record. Nothing stops someone with their own allocation from publishing a PTR that reads crawl-203-0-113-7.googlebot.com. Step 2 alone verifies nothing, you asked the suspect for their name and believed the answer.

What they cannot do is make googlebot.com's forward zone resolve that hostname to their address, because that zone is authoritative under Google's control. The forward lookup is the step that actually proves something. Steps 1 and 2 narrow the candidate; step 3 and 4 are the proof.

Doing It Properly in Python

The mechanism is four lines; the production version is the failure handling and the cache. Every branch below corresponds to a real thing that happens in logs.

import ipaddress
import socket
import time

# Suffixes each operator publicly documents for its crawlers.
VERIFIED_SUFFIXES = {
    "googlebot":   (".googlebot.com", ".google.com", ".googleusercontent.com"),
    "bingbot":     (".search.msn.com",),
    "applebot":    (".applebot.apple.com",),
    "duckduckbot": (".duckduckgo.com",),
}

POS_TTL, NEG_TTL = 3600.0, 60.0   # trust a pass for an hour, a fail for a minute
_cache: dict[tuple[str, str], tuple[bool, float]] = {}


def _forward_ips(hostname: str) -> set[str]:
    try:
        infos = socket.getaddrinfo(hostname, None, proto=socket.IPPROTO_TCP)
    except socket.gaierror:
        return set()                      # hostname from the PTR does not resolve
    return {info[4][0] for info in infos}


def verify(ip: str, operator: str, timeout: float = 3.0) -> bool:
    """Forward-confirmed reverse DNS. True only if the full round trip closes."""
    key = (ip, operator)
    cached = _cache.get(key)
    if cached is not None:
        result, stamp = cached
        if time.monotonic() - stamp < (POS_TTL if result else NEG_TTL):
            return result

    result = False
    socket.setdefaulttimeout(timeout)
    try:
        canonical = ipaddress.ip_address(ip).compressed   # reject junk before DNS
        hostname, _, _ = socket.gethostbyaddr(canonical)  # 1. PTR
        hostname = hostname.rstrip(".").lower()
        if hostname.endswith(VERIFIED_SUFFIXES[operator]):  # 2. suffix match
            result = canonical in _forward_ips(hostname)    # 3 + 4. forward, compare
    except socket.herror:
        result = False        # no PTR record at all — extremely common
    except (ValueError, KeyError):
        result = False        # not an IP address, or operator we don't know
    except OSError:
        result = False        # resolver timeout or transient network failure
    finally:
        socket.setdefaulttimeout(None)

    _cache[key] = (result, time.monotonic())
    return result


if __name__ == "__main__":
    for addr in ("66.249.66.1", "203.0.113.7"):
        print(addr, "->", verify(addr, "googlebot"))

Three details that matter more than the happy path. The suffixes carry a leading dot, so notgoogle.com cannot slip through a naive endswith("google.com"). Negative results get a short TTL, because a resolver hiccup that pins a real Googlebot to "unverified" for an hour is a self-inflicted deindexing. And socket.setdefaulttimeout is process-global and blunt — if this sits on a request path, use dnspython with per-query timeouts and an async resolver instead of the stdlib.

An important operational note: never do this synchronously in the request path if you can avoid it. Verify out of band, cache the verdict against the IP, and let the first request through on a default policy.

The IP Range Files: Faster, Staler

Google also publishes its crawler ranges as JSON, which turns a DNS round trip into a prefix match:

• https://developers.google.com/static/crawling/ipranges/common-crawlers.json — Googlebot itself
• .../special-crawlers.json — AdsBot, Google-Safety and similar
• .../user-triggered-fetchers.json and .../user-triggered-fetchers-google.json — fetches initiated by a user action rather than by the crawl scheduler

The tradeoff is freshness. A prefix file you cached yesterday does not contain a range Google added this morning, and the failure mode is refusing a genuine crawler. DNS is authoritative and always current, but costs a lookup and depends on your resolver being healthy. The usual production answer is both: match against the prefix file first, fall back to FCrDNS on a miss, and refresh the files daily.

The Same Pattern Everywhere Else

Every major operator runs a version of this, which makes it worth implementing once as a generic function rather than as a Googlebot special case.

• Bingbot verifies against hostnames under search.msn.com, with the same round trip; Microsoft also publishes a JSON range list and a verification tool in Bing Webmaster Tools.
• Applebot uses applebot.apple.com and publishes ranges.
• DuckDuckBot and Yandex publish their own documented suffixes.
• Common Crawl's CCBot runs from documented infrastructure but is the case where reading the operator's published guidance matters more than pattern-matching a hostname.

Verified Bots at the CDN, and Why It Matters More Than It Should

The bigger shift is that most sites never implement any of this, because their CDN did it for them. Cloudflare's Verified Bots programme maintains a central list of known-good crawlers, and a listed operator is treated differently across a very large share of the web at once.

That makes the listing criteria genuinely consequential. Cloudflare publishes a policy: meaningful traffic volume across multiple domains, a documented and stable user-agent or signature identity, no impersonation of another verified service, and, pointedly, rejection for a search crawler that ignores robots.txt. Verification is also no longer the same thing as access. In Cloudflare's 2026 model, verification establishes identity, and whether you are let in depends on your classification (Search, Agent, Training) and the site owner's policy.

For a crawler operator that is the whole strategic picture in one sentence: being identifiable is now a prerequisite for being allowed, and it is separate from being allowed.

The Cryptographic Successor

DNS-based verification has an obvious ceiling. It requires stable, published, operator-owned IP space, which rules out anything on ephemeral cloud egress, anything running behind a CDN, and every agent executing on a user's own device. The replacement being standardised at the IETF drops IP identity entirely in favour of signing requests with a key an origin can resolve to a known operator. That is Web Bot Auth, and Cloudflare already accepts it as a verification method alongside IP-based checks.

What People Get Wrong

• Checking the reverse lookup and stopping. The PTR record is controlled by the IP's owner. Without the forward confirmation you have verified nothing at all.
• Substring matching the hostname. endswith("google.com") without the leading dot accepts fakegoogle.com. Anchor on the dot.
• Treating a DNS failure as a failed verification, permanently. Cache negatives for seconds, not hours.
• Verifying inline on every request. DNS in the hot path adds latency and a hard dependency on your resolver. Verify asynchronously and cache by IP.
• Assuming verified means harmless. A verified crawler can still hammer an expensive endpoint. Verification answers "who", not "how much".
• Publishing a User-Agent with no contact URL, if you run a crawler. A bare product token gives an operator no way to reach you before they block you.

Wrapping Up

The mechanism is small and the reasoning behind it is the valuable part: reverse DNS is a claim by the address owner, forward DNS is a claim by the domain owner, and only agreement between two parties with different interests proves anything. Implement it generically across operators, cache the results with asymmetric TTLs, back it with the published prefix files, and keep it off the request path. And if you are on the crawling side of this, the bar is lower than people assume, a documented user-agent with a contact URL, stable published egress addresses or a signable identity, and behaviour that matches what you say you do is most of what any verification programme is asking for.