Robots.txt for Web Scraping: A Practical Proxy Guide

Nearly every website you'll ever scrape publishes a small text file at /robots.txt. Append it to any root domain (for example, https://example.com/robots.txt) and you'll see plain-text instructions aimed at automated visitors. It's easy to skim past this file, but for anyone building crawlers or collecting public data, it's the first thing worth reading. This guide covers what robots.txt actually says, how to parse it in Python, and how to fold those rules into a well-behaved scraping workflow.

What robots.txt is (and isn't)

Robots.txt is a convention rooted in the Robots Exclusion Protocol, which was standardised in RFC 9309. It tells bots which parts of a site the owner would prefer they leave alone. The most common directive is a simple pairing of a User-agent line and one or more Disallow rules.

To block every bot from a directory called /confidential/, a site might publish this:

User-agent: *
Disallow: /confidential/

Owners can also write more targeted rules — permitting one crawler while restricting the rest:

User-agent: *
Disallow: /confidential/

User-agent: SpecificBot
# Allow access only to this page within the restricted directory
Allow: /confidential/public-info.html

Here, all bots are kept out of /confidential/ except SpecificBot, which is explicitly allowed to fetch /confidential/public-info.html.

One point people misunderstand often: robots.txt is a request, not an access control. It doesn't stop a bot technically — a script can reach any publicly served page regardless of what the file says. What it does is signal intent. Reputable search engines and responsible scrapers honour those signals because ignoring them is poor practice and can strain a site's infrastructure. If you need actual security, that belongs behind authentication, not in a public text file.

It's also worth knowing that a disallowed page can still get visited accidentally if an allowed page links to it, since crawlers follow the links they find. That's why robots.txt does double duty: it guides crawler behaviour and it shapes what search engines index for results.

The syntax, piece by piece

The format is deliberately minimal. Three building blocks do most of the work:

• User-agent — names the bot (or bots) the block applies to. Every rule block starts with one.

• Allow / Disallow — grants or withholds access to a path for the current user agent.

• Wildcards — robots.txt isn't full regex, but the asterisk (*) matches any sequence of characters and the dollar sign ($) anchors the end of a URL.

Back to our earlier example:

User-agent: *
Disallow: /confidential/

User-agent: SpecificBot
Allow: /confidential/public-info.html

The first block's wildcard * applies the disallow to every user agent. The second block carves out an exception for SpecificBot.

Why block anything at all? Search engines work with a crawl budget — a rough cap on how many URLs they'll fetch per visit. Steering crawlers away from low-value or duplicate sections keeps that budget focused on content that matters, which is important for SEO on large sites.

The $ anchor gives finer control. You can target file types:

User-agent: *
Disallow: /*.pdf$

That tells every bot to skip any URL ending in .pdf. A related use is heading off duplicate content from tracking parameters:

User-agent: *
Disallow: /*?ref=

Now any URL carrying a ref= query parameter is off-limits to all bots.

Fetching robots.txt in Python

Because it's plain text, pulling the file down is trivial. The requests library is the usual go-to:

import requests

target_url = "http://example.com/robots.txt"

try:
    response = requests.get(target_url)
    response.raise_for_status() # Check if the request was successful
    print(response.text)
except requests.exceptions.RequestException as e:
    print(f"Error fetching robots.txt: {e}")

For projects that touch many domains, it pays to cache each file locally so you're not re-fetching it on every run:

import requests
import os

# URL for the robots.txt file
robots_url = "http://example.com/robots.txt"

# Define filename based on domain or other convention
file_name = "example_com_robots.txt"

try:
    # Fetch the content
    response = requests.get(robots_url)
    response.raise_for_status()  # Ensure request was successful

    # Save the content locally
    with open(file_name, "w", encoding='utf-8') as f:
        f.write(response.text)

    print(f"{file_name} downloaded successfully.")
except requests.exceptions.RequestException as e:
    print(f"Failed to download {robots_url}: {e}")

Give your files a predictable naming scheme (the domain works well) so you can look them up later without confusion. If you run your own site, Google's Robots.txt Tester in Search Console will confirm your file parses the way you expect.

Parsing the rules that apply to you

Search engines treat robots.txt as authoritative, and your scraper should aim for the same standard — check the file before crawling anything. Owners disallow sections for good reasons: reducing server load, keeping bots out of infinite spaces like calendar pages, or protecting endpoints that were never meant for automated traffic.

Since a custom scraper almost never has its own named entry (unlike Googlebot or Bingbot), you'll usually care about the rules under the wildcard user agent (*). Here's a self-contained example that extracts those disallowed paths and checks whether a given URL is permitted:

import requests
import re
from urllib.parse import urlparse

# Assume robots_content holds the text from a downloaded robots.txt file
# robots_content = """
# User-agent: *
# Disallow: /admin/
# Disallow: /private_stuff/
# Disallow: /*?sessionid=*
#
# User-agent: Googlebot
# Allow: /private_stuff/allowed-for-google.html
# """

def get_disallowed_paths(robots_content, user_agent='*'):
    disallowed = []
    current_ua = None
    lines = robots_content.splitlines()

    for line in lines:
        line = line.strip()
        if not line or line.startswith('#'):
            continue

        parts = line.split(':', 1)
        if len(parts) != 2:
            continue

        directive = parts[0].strip().lower()
        value = parts[1].strip()

        if directive == 'user-agent':
            current_ua = value
        elif directive == 'disallow' and current_ua == user_agent:
            if value: # Ensure Disallow value is not empty
                disallowed.append(value)

    return disallowed

# Example Usage:
# disallowed_for_all = get_disallowed_paths(robots_content, '*')
# print(f"Disallowed paths for '*': {disallowed_for_all}")

def can_fetch(url_path, disallowed_paths):
    # Ensure url_path starts with /
    if not url_path.startswith('/'):
        url_path = '/' + url_path

    for path_pattern in disallowed_paths:
        # Simple pattern matching: '*' wildcard, '$' end anchor
        regex_pattern = re.escape(path_pattern)
        regex_pattern = regex_pattern.replace(r'\*', '.*')
        if regex_pattern.endswith(r'\$'):
            regex_pattern = regex_pattern[:-2] + '$'
        else:
            # Ensure partial matches on directories work (e.g., /private/ matches /private/page.html)
            if not regex_pattern.endswith('$'):
                 regex_pattern += '.*' # Match anything following if it's a directory/prefix
        # Ensure the pattern matches from the start of the path
        if re.match('^' + regex_pattern, url_path):
            return False # Found a matching disallow rule

    return True # No disallow rule matched

# Example Check:
# url_to_check = "/private_stuff/some_page.html"
# is_allowed = can_fetch(urlparse(url_to_check).path, disallowed_for_all)
# print(f"Can fetch {url_to_check}? {is_allowed}") # Should be False based on example content

This pulls the disallow list for the wildcard agent and gives you a can_fetch helper to test any path against it. For production work you don't have to roll your own — Python's standard library ships urllib.robotparser, and frameworks like Scrapy respect robots.txt automatically when you enable the setting. The hand-written version above is still useful when you want to see exactly what's being matched and why.

Rate limits matter as much as the rules

Honouring the disallow list is only half of good behaviour. The other half is pacing. A scraper can fire off hundreds of requests in the time a person clicks through a handful of pages, and that volume can degrade a site for its real visitors — especially during peak hours.

A few habits keep you on the right side of this:

• Add a delay between requests, and watch Crawl-delay if the file specifies one.

• Monitor response times. If they climb, back off — that's a signal you're pushing too hard.

• Spread requests over time rather than hammering an endpoint in a tight loop.

Sending traffic through ethically sourced residential proxies helps distribute load and keeps your infrastructure stable, but it doesn't replace considerate pacing. Proxies are a tool for reliability and geographic coverage on public data, not a substitute for respecting a site's limits. If you'd rather not manage rate control and browser fingerprints yourself, Evomi's managed Scraping Browser handles the plumbing while you focus on the data.

If you're moving on to specific targets, our walkthroughs on Python scraping with Beautiful Soup and scraping reviews responsibly build on the same principles covered here.

Wrapping up

Whether you're running a search crawler or a focused scraper, reading and respecting robots.txt is a baseline of responsible automation. Search engines do it by default; your projects should match that standard. The file is easy to fetch, the syntax is small, and for most custom scrapers the wildcard (*) rules are all you need to check. Pair that with sensible rate limiting and clean, ethically sourced proxies, and you can collect the public data you need without putting strain on the sites you rely on.