Python Web Crawling with Scrapy and Proxies: A Guide

Search engines like Google and Bing know about billions of pages because they run fleets of automated bots that follow links across the web, recording what they find. That process is called web crawling, and you don't need Google's infrastructure to do it well. With Python and Scrapy, an open-source crawling framework, you can build a focused crawler in a single afternoon.

In this guide we'll build a working CrawlSpider that maps out every Wikipedia article reachable within two clicks of a starting page. Then we'll wire it up to rotating proxies so it can run at reasonable scale without hammering a single IP or overloading the target server.

Crawling vs. Scraping: What's the Difference?

People use these terms interchangeably, but they describe two distinct jobs. Web crawling is about discovery: finding URLs and following them to find more URLs, like mapping the streets of a city. Web scraping is about extraction: pulling specific pieces of information out of the pages you land on, like recording the name and address of every shop.

Most real projects use both. A price-comparison service, for example, first crawls e-commerce sites to discover product pages, then scrapes those pages for prices and availability. Crawling typically outputs a list of URLs or raw HTML; scraping outputs structured data ready for a spreadsheet or database. If you want the full breakdown, see our comparison of web crawling vs. web scraping.

What We're Building

Our crawler starts on a chosen Wikipedia article and discovers every article within two "degrees of separation" — two clicks — from that starting point. Along the way it records the title and URL of each page it visits. It's a small, contained project, but it exercises the same mechanics you'd use for a much larger crawl.

Prerequisites

You'll need Python installed. If you don't have it, grab the latest release from the official Python website. Then install Scrapy from your terminal:

A little familiarity with basic scraping concepts helps. If you're starting from zero, our guide to Python web scraping in 2025 is a solid foundation.

Setting Up the Scrapy Project

Navigate to your working directory and scaffold a new project:

This creates a wikinav directory with the standard Scrapy layout:

Create a new file called wikispider.py inside wikinav/spiders/ and open it in your editor. Start with the imports — the spider base classes, the link extractor, and Python's regular expression module:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
import re

Defining the Spider

In Scrapy, a spider is a Python class whose attributes and methods define its behavior. We'll subclass CrawlSpider, which is purpose-built for following links across pages:

class WikiNavSpider(CrawlSpider):
    # Spider configuration and logic will go here

Every spider needs a unique name and at least one entry in start_urls. Pick any article you like as the starting point:

    name = 'wikinav'

    # Starting point for our crawl
    start_urls = ['https://en.wikipedia.org/wiki/Web_crawler']

The rules attribute is where CrawlSpider earns its keep. It tells the crawler which links to extract, how to process them, and whether to keep following:

    rules = (
        # Rule to extract, process, and follow links
        Rule(
            LinkExtractor(
                allow=r"https:\/\/en\.wikipedia\.org\/wiki\/(?!Main_Page$)[^:]*$",
                # Allow article links, exclude Main_Page and special namespaces
                deny=(
                    r"Special:", r"Portal:", r"Help:",
                    r"Wikipedia:", r"Wikipedia_talk:",
                    r"Talk:", r"Category:",
                )  # Deny specific non-article namespaces
            ),
            callback='parse_page_content',  # Function to call for each extracted link
            follow=True  # Allow the spider to follow links found on these pages
        ),
    )

That's dense, so let's unpack the Rule:

• LinkExtractor finds links on a page based on the criteria you supply.

• allow is a regular expression matching the URLs you want. Here it targets English Wikipedia article pages (/wiki/Something) while excluding the Main Page and any URL containing a colon — colons signal non-article namespaces like File: or Template:. If regular expressions are unfamiliar, this overview is worth a read.

• deny lists patterns to skip even if they match allow. We drop Wikipedia's administrative and meta namespaces.

• callback names the method (as a string) that processes each page's response.

• follow, set to True, lets the crawler keep discovering links on the pages it fetches — that's what makes multi-level crawling work.

Writing rules is usually the fiddliest part of a CrawlSpider. It takes some trial and error, but it gives you a clean, declarative way to control crawling without managing links by hand.

Limiting Crawl Depth

We only want pages within two clicks of the start URL. Scrapy makes this trivial with custom_settings:

    custom_settings = {
        'DEPTH_LIMIT': 2,  # Limit crawl depth to 2 levels from start_urls
    }

Without this cap, the spider would try to crawl an enormous chunk of Wikipedia — polite crawling means bounding your scope.

Parsing Each Page

Now we implement the callback we referenced, parse_page_content. It receives the response object for each crawled page — this is where you extract data. We'll keep it simple: grab the title and URL, and yield them as a dictionary.

    def parse_page_content(self, response):
        # Extract the main title from the <title> tag, removing the " - Wikipedia" suffix
        page_title = response.css('title::text').get().replace(" - Wikipedia", "")
        page_url = response.url

        # Yield the data as a dictionary
        yield {
            'title': page_title,
            'url': page_url
        }

Note yield rather than return. Scrapy runs asynchronously, handling many requests at once, and yield lets it stream data out without blocking the spider's progress.

The Complete Spider

Here's the full wikispider.py:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
import re

class WikiNavSpider(CrawlSpider):
    name = 'wikinav'

    # Starting point for our crawl
    start_urls = ['https://en.wikipedia.org/wiki/Web_crawler']

    # Rules define how to follow links and which pages to process
    rules = (
        Rule(
            LinkExtractor(
                allow=r"https:\/\/en\.wikipedia\.org\/wiki\/(?!Main_Page$)[^:]*$",
                # Allow article links, exclude Main_Page and special namespaces
                deny=(
                    r"Special:", r"Portal:", r"Help:",
                    r"Wikipedia:", r"Wikipedia_talk:",
                    r"Talk:", r"Category:"
                )  # Deny specific non-article namespaces
            ),
            callback='parse_page_content',  # Function to call for each extracted link
            follow=True  # Allow the spider to follow links found on these pages
        ),
    )

    # Custom settings specific to this spider
    custom_settings = {
        'DEPTH_LIMIT': 2  # Limit crawl depth to 2 levels from start_urls
    }

    # Callback function to process the response from each crawled page
    def parse_page_content(self, response):
        # Extract the main title from the <title> tag, removing the " - Wikipedia" suffix
        page_title = response.css('title::text').get().replace(" - Wikipedia", "")
        page_url = response.url

        # Yield the data as a dictionary
        yield {
            'title': page_title,
            'url': page_url
        }

Crawl Responsibly: Delays and Rate Limiting

The spider works, but running it full-throttle against a live site will get your IP throttled fast — and rightly so. Aggressive crawling puts real load on someone else's servers. Before you touch proxies, slow down. Add a request delay to the project's settings.py (wikinav/settings.py):

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 1.5  # Wait 1.5 seconds between requests

It's also good practice to keep Scrapy's default ROBOTSTXT_OBEY = True in place so your crawler respects each site's robots.txt directives, and to set a descriptive USER_AGENT. A well-behaved crawler is one that identifies itself and stays within the limits a site publishes.

Why Add Proxies?

Delays keep you polite, but for larger crawls they aren't enough on their own. Sending thousands of requests from one IP address can trip a site's automated rate limits regardless of how careful you are, and it can leave your own connection blacklisted for legitimate use.

Proxies solve this by routing requests through different IP addresses. Rotating proxies spread your traffic across a pool so no single address carries the whole load — that means smoother, more reliable crawls of public data and less risk to your own IP. At Evomi we provide ethically sourced residential, mobile, datacenter, and static ISP proxies, all backed from our Swiss home base. Residential proxies route through real consumer devices, so requests come from genuine geographic locations — useful when you need results that reflect how a page looks to real users in a given region.

Adding Evomi Proxies to Your Scrapy Spider

We'll use rotating residential proxies, which change the exit IP automatically per request — ideal for spreading out a larger crawl. You'll need your Evomi proxy credentials plus the endpoint. For rotating residential proxies the endpoint is rp.evomi.com, with ports 1000 (HTTP), 1001 (HTTPS), and 1002 (SOCKS5).

Handle credentials carefully. Your username, password, endpoint, and port are sensitive. Don't hardcode them into your script, especially if the code goes into version control. The cleanest approach is environment variables. Scrapy automatically honors the standard http_proxy and https_proxy variables.

On Linux or macOS (bash/zsh), set them with export in the same terminal before running the spider:

# Replace YOUR_USERNAME and YOUR_PASSWORD with your actual Evomi credentials
export http_proxy="http://YOUR_USERNAME:YOUR_PASSWORD@rp.evomi.com:1000"
export https_proxy="http://YOUR_USERNAME:YOUR_PASSWORD@rp.evomi.com:1001"

On Windows Command Prompt use set http_proxy=...; in PowerShell use $env:http_proxy=.... Once the variables are set, Scrapy routes every request through the proxy with no changes to your Python code.

Now run the crawler in that same session. This executes the wikinav spider and writes the yielded data to a JSON file:

scrapy crawl wikinav -o

Depending on your starting page and network conditions, the crawl may take a while. If you want to confirm your proxy is live before running, our free proxy tester checks connectivity and exit location in seconds.

When Scrapy Isn't the Right Tool

Scrapy is excellent for crawling and scraping traditional, server-rendered websites, and it scales beautifully. Its one blind spot: it doesn't run JavaScript. Many modern sites build their content client-side, so the HTML Scrapy receives is nearly empty. For those, a browser-driven tool is the better fit — see our guide to dynamic web scraping with Selenium, or use a managed headless browser like Evomi's Scraping Browser when you'd rather not manage the infrastructure yourself.

Wrapping Up

You now have a working Scrapy CrawlSpider: it defines link-extraction rules, follows them to a bounded depth, parses each page, and routes traffic through rotating proxies for stable, respectful crawling. From here you can point it at other public datasets, expand the parsing logic to capture more fields, or adjust the depth and rules to fit a bigger project — the pattern stays the same.