Respecting robots.txt and Crawl-Delay: Ethical Scraping That Still Scales

Here is the uncomfortable truth about robots.txt: it can't stop you. It's a text file at a known URL. Your scraper can fetch it, ignore it, and keep going, the server has no technical way to enforce a single line, and in most jurisdictions it carries no direct legal weight on its own.

Which is exactly the point. Because nothing forces you to honor it, whether you do is the clearest signal of whether you're a professional running a sustainable operation or a problem waiting to get blocked. Abuse teams notice the difference.

This post is about reading and respecting robots.txt and crawl-rate signals correctly, without turning your crawler into something that takes a week to do an hour's work. Politeness and throughput are not opposites; done right, the polite crawler is the one still running next month.

What robots.txt Actually Is

The Robots Exclusion Protocol (REP) started as an informal convention in 1994 and was written down as RFC 9309 in 2022. It's a voluntary standard: a compliant crawler reads the file and obeys it, assuming good faith on the client side.

A file is a set of groups. Each starts with one or more User-agent lines, followed by Allow and Disallow rules:

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: BadBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
Crawl-delay: 5

The pieces:

• User-agent names which crawler the group applies to. * is the default for any agent without a more specific match.

• Disallow lists path prefixes the agent should not fetch. An empty Disallow: means "nothing is blocked."

• Allow carves exceptions out of a broader Disallow (standardized in RFC 9309).

• Sitemap points to one or more sitemap URLs. It's global, not tied to a group.

• Crawl-delay asks for N seconds between requests. This is not part of RFC 9309, it's a widely-deployed extension some crawlers honor and others ignore.

What robots.txt is not, and this matters:

• It is not access control. A Disallow requires no authentication and blocks nothing. If you don't want a page fetched, don't serve it publicly.

• It is not law. It's not a contract you agreed to. (A site's Terms of Service are a separate question, see below.)

• It is not a security boundary. Treating it as one is a server-side mistake, not a scraper's permission slip.

So why honor something purely advisory? Because the directives encode what the operator wants automated traffic to avoid, usually endpoints that are expensive, useless to you, or sensitive. Respecting them keeps you out of trouble and off the slow paths.

Reading It Correctly

Most "robots.txt bugs" are misreadings, not parse failures.

Match the right user-agent group. A crawler picks the single most specific group whose User-agent token is a case-insensitive prefix match of its own product token. If you call yourself MyCrawler and there's a group for MyCrawler and one for *, you obey MyCrawler and ignore * entirely, groups do not merge or inherit. Fall back to * only if nothing more specific matches.

Longest match wins. When both an Allow and a Disallow could apply, the rule with the longest matching path wins. For the file above, /search/help is allowed (longer Allow) even though /search is disallowed. Equal-length ties resolve in favor of Allow.

Wildcards. RFC 9309 supports * (any sequence) and $ (end of URL): Disallow: /*?sort= blocks any URL with that query fragment; Disallow: /*.pdf$ blocks PDFs.

Treat the Sitemap line as a gift. It's a structured index of canonical URLs — exactly what you'd otherwise crawl blindly to discover. Using it means fewer wasted requests: more polite and more efficient. Always check for it.

Parsing It in Python

The standard library ships urllib.robotparser, zero-dependency and fine for simple cases.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()  # fetches and parses

UA = "MyCrawler/1.0"
print(rp.can_fetch(UA, "https://example.com/search/help"))  # True
print(rp.can_fetch(UA, "https://example.com/cart/"))        # False

# crawl_delay() returns the value if present (Python 3.6+), else None
print(rp.crawl_delay(UA))

The standard library has real limits, though: no built-in refresh policy, historically weaker wildcard handling than RFC 9309 requires, and crawl_delay() support that depends on your Python version. For anything serious, reach for a dedicated library.

Protego is the parser Scrapy uses, and it tracks RFC 9309 closely (wildcards, longest-match, the lot):

import httpx
from protego import Protego

UA = "MyCrawler/1.0"
text = httpx.get("https://example.com/robots.txt").text
rp = Protego.parse(text)

print(rp.can_fetch("https://example.com/search/help", UA))  # True
print(rp.crawl_delay(UA))                                    # e.g. 5.0
print(list(rp.sitemaps))                                     # discovery for free

reppy is another option focused on performance, though it has historically lagged on maintenance. For most projects: protego for correctness, the stdlib for zero dependencies.

Every parser leaves one thing to you: fetch robots.txt once per host, cache it, and refresh periodically (a few hours is typical). Handle the edge cases too, a 404 conventionally means "everything allowed," while a 5xx or timeout is best treated as "back off, the site is having a bad time."

Crawl-Delay and Choosing a Polite Rate

Honoring Crawl-delay is the easy part: if the file asks for 5 seconds, your per-host scheduler waits at least 5 seconds between requests to that host. When there's no directive (the common case), you pick a default. Sane starting points:

• 1 request per second per host, a reasonable conservative default for most sites.

• One every 2-5 seconds for small sites, single-server setups, or anything that feels hand-built.

• Faster only with evidence the host can take it, large CDN-backed platforms absorb far more, but let response times tell you, not optimism.

The number that matters is requests per host, not your global rate. That's the single most important idea here.

Be adaptive. A fixed delay is a guess; the server's responses are data. Back off when you see trouble:

import time

def polite_delay(state, response, base_delay):
    if response.status_code == 429 or response.status_code >= 500:
        # exponential backoff, capped
        state["delay"] = min(state["delay"] * 2, 60)
    else:
        # ease back toward the baseline on success
        state["delay"] = max(base_delay, state["delay"] * 0.9)
    time.sleep(state["delay"])
    return state

A 429 Too Many Requests (honor any Retry-After header) or a wave of 503s is the server explicitly telling you to slow down. Ignoring it is how you go from "rate-limited" to "permanently blocked."

Cap per-host concurrency. Politeness isn't only about delay between requests — it's also about not opening 50 parallel connections to one origin. One or two concurrent per host is plenty. Scale by adding more hosts, not more pressure on each one.

Where Proxy Rotation Fits

This is the part people get wrong. Rotating IPs, datacenter, ISP, or residential, spreads requests across addresses so no single IP looks like a firehose. That's legitimate for distributing load and avoiding IP-reputation false positives.

But rotation is not a license to hammer. Fire 100 requests/second at one host across 100 IPs and you're still putting 100 requests/second of real load on their infrastructure. IP diversity hides you from naive per-IP rate limits, but it doesn't change the load you impose, and behavioral detection still flags the aggregate pattern. Per-host pacing has to live in your scheduler, independent of how many IPs you rotate through. A clean pool, say, residential IPs like Evomi's, keeps you off shared deny-lists, but the pacing discipline is yours to enforce.

The Business Case for Being Polite

If the ethics don't move you, the economics should. Polite crawlers:

• Get blocked less. Aggressive patterns are the number-one trigger for rate limits, IP bans, and CAPTCHA walls. Slow down and you trip fewer of them, and every challenge you provoke costs a solver call, a retry, or a dead request.

• Keep long-term access. Most data work needs a stable feed over months, not one heroic scrape before a ban. Sustainable rate beats peak rate.

• Lower legal and PR risk. Staying off the "unusual traffic" radar keeps a scrape out of a complaint, and out of a headline.

Politeness is also just better engineering. Per-host scheduling, backoff, caching, and sitemap-driven discovery are the same primitives that make a crawler robust, how you survive flaky networks and slow origins regardless of etiquette.

The Legal and Ethical Landscape, Briefly

This is informational, not legal advice, talk to a lawyer for your case. The broad strokes:

• Public vs. authenticated data is the biggest line. Scraping behind a login, paywall, or "click to agree" wall is a meaningfully different risk profile than fetching public pages.

• Terms of Service can bind you in ways robots.txt doesn't, particularly if you accepted them by creating an account.

• Personal data (GDPR, CCPA, and similar) doesn't care whether data was public. Collecting and storing PII carries obligations regardless of how you got it.

• The case law moves. The long hiQ Labs v. LinkedIn arc, where scraping public profiles was found not to violate the US Computer Fraud and Abuse Act, even as other claims survived, narrowed CFAA's reach for public data but didn't make scraping consequence-free. Treat any single case as a data point, not a green light.

When in doubt, check for an official API first. It's often cheaper, more stable, and explicitly permitted.

A Good-Citizen Checklist

• Identify yourself honestly in your User-Agent where appropriate, a product name and contact URL let an operator reach you instead of blocking you.

• Cache aggressively. Don't re-fetch what hasn't changed; honor ETag and Last-Modified.

• Request only what you need. Skip assets, tracking endpoints, and out-of-scope pages. Use the sitemap.

• Crawl off-peak for the target's timezone when you can.

• Honor every signal: robots.txt, Crawl-delay, 429/Retry-After, and 5xx backoff.

• Ask first when it's high-volume. A short email requesting access (or an API key) often beats any amount of clever evasion.

Wrapping Up

robots.txt can't make you do anything, and that's the whole point, honoring it marks you as someone worth granting access to. Parse it correctly (protego for RFC 9309 fidelity, the stdlib for zero dependencies), cache it per host, and read the Sitemap line as the free discovery index it is.

Then put the discipline where it counts: per-host pacing in your scheduler, adaptive backoff on 429/5xx, and a per-host concurrency cap that proxy rotation never overrides. That single habit, pacing per host, not per IP, is what lets a crawler be both polite and fast, and what keeps it running long after the impatient ones get banned.