Python Web Scraping in 2025: Tools, Tips & Proxies

Python remains the most practical language for pulling public data off the web. The syntax stays out of your way, and the library ecosystem covers everything from a quick one-off script to a full crawling framework. In this guide you'll build a working scraper with the Requests library and Beautiful Soup, then apply it to a real task: collecting post titles from the classic r/programming interface and counting which programming languages get mentioned most.

Everything here targets publicly available pages and respects the server with polite request rates. Before scraping any site at scale, check its robots.txt and terms of service, and prefer an official API when one exists.

What web scraping actually involves

Web scraping is the automated extraction of information from websites. Instead of copying data by hand, you run a script that downloads a page's HTML and pulls out the specific fields you need. More advanced scrapers drive a headless browser to render JavaScript-heavy pages the way a real user would.

Two honest caveats. First, scrapers are brittle: change the target site's HTML or CSS and your selectors can break overnight. Second, an API is almost always the better choice when it's available, because it gives you structured data and a stable contract. When there's no API, scraping public pages is a solid way to gather data for market research, competitive analysis, price monitoring, or academic study.

Why Python is a natural fit

Plenty of languages can scrape (you really only need an HTTP client and an HTML parser), but Python's tooling is unusually good for it. Requests handles HTTP cleanly, Beautiful Soup makes navigating messy HTML painless, and for larger jobs the Scrapy framework gives you a full crawling architecture. When a page needs a real browser, Playwright or Selenium step in for automation.

These libraries are mature, well documented, and backed by big communities. The gentle learning curve also means you can prototype an idea in minutes even if writing code isn't your day job. If you're new to the parser side of things, our dedicated walkthrough on Python web scraping with Beautiful Soup goes deeper on selectors.

A hands-on example: scraping r/programming

Our goal is to collect titles from roughly the first several pages of r/programming, then count how often popular languages appear. We'll target the classic interface (old.reddit.com) because its HTML structure is simpler and friendlier for learning.

Setting up your environment

Make sure Python is installed; grab it from the official Python website if needed. Then install the two libraries with pip:

Create a new file named reddit_scraper.py. That's where the code lives.

Fetching the page content

The workflow is two steps: get the HTML, then parse it. To download the r/programming front page, we use Requests.

import requests

The requests.get() function fetches the page. Always send a descriptive User-Agent header. Many sites, Reddit included, throttle requests that arrive with a default script user agent, and a clear, honest agent string also tells the site's operators who you are.

target_url = "https://old.reddit.com/r/programming/"
headers = {'User-agent': 'Python Scraping Bot - Learning Project 1.0'}
response = requests.get(target_url, headers=headers)

The raw HTML lives on the .content attribute of the response object.

page_html = response.content

So far, your script looks like this:

import requests

target_url = "https://old.reddit.com/r/programming/"
headers = {'User-agent': 'Python Scraping Bot - Learning Project 1.0'}
response = requests.get(target_url, headers=headers)
page_html = response.content  # We'll add parsing logic next

Parsing HTML with Beautiful Soup

Import the library at the top of your script:

from bs4 import BeautifulSoup

Then create a parser object from the HTML we fetched:

soup_parser = BeautifulSoup(page_html, "html.parser")

This object lets you navigate the document with methods like find() and find_all(). To know what to search for, open r/programming in your browser (private/incognito mode keeps you logged out), right-click a post title, and choose "Inspect". The developer tools highlight the matching HTML. Look for tags and attributes that consistently mark the data you want.

On old Reddit, titles sit inside an anchor tag inside a paragraph tag with the class title. Grab all of those paragraphs:

title_paragraphs = soup_parser.find_all("p", class_="title")

Each paragraph holds an anchor tag whose text is the title. Extract just the text with a list comprehension:

extracted_titles = [p_tag.find("a").get_text() for p_tag in title_paragraphs]

Print the results to check:

print(extracted_titles)

Here's the complete single-page script. Note that response.content is the correct attribute (a common typo is response.contents, which doesn't exist):

import requests
from bs4 import BeautifulSoup

target_url = "https://old.reddit.com/r/programming/"
headers = {'User-agent': 'Python Scraping Bot - Learning Project 1.0'}
response = requests.get(target_url, headers=headers)
page_html = response.content

soup_parser = BeautifulSoup(page_html, "html.parser")
title_paragraphs = soup_parser.find_all("p", class_="title")
extracted_titles = [p_tag.find("a").get_text() for p_tag in title_paragraphs]
print(extracted_titles)

Running this prints the titles from the first page. To build a bigger dataset, we need to walk through several pages.

Handling pagination

Let's extend the script to cover the first 15 pages. The pattern is simple: scrape the current page, find the link to the next page, follow it, and repeat.

Inspecting old Reddit again, the "next" button is an anchor inside a span with the class next-button. Extract that link like so:

next_button_span = soup_parser.find("span", class_="next-button")

# Check if the button exists before trying to get the link
if next_button_span and next_button_span.find("a"):
    next_page_url = next_button_span.find("a")['href']
else:
    next_page_url = None  # No more pages

We'll also use the time library to pause between requests. Spacing requests out is both courteous to the server and easier on your own success rate.

import time

Initialize a list for all titles and a variable for the current page URL:

all_post_titles = []
current_page_url = "https://old.reddit.com/r/programming/"
scrape_page_count = 15  # How many pages to scrape

Now loop through the pages, fetching, parsing, collecting titles, finding the next URL, and pausing between each request:

headers = {'User-agent': 'Python Scraping Bot - Learning Project 1.0'}

for page_num in range(scrape_page_count):
    if not current_page_url:
        print("No more pages found. Stopping.")
        break

    print(f"Scraping page {page_num + 1}: {current_page_url}")

    try:
        response = requests.get(current_page_url, headers=headers)
        response.raise_for_status()  # Check for HTTP errors (like 404, 500)
        page_html = response.content

        soup_parser = BeautifulSoup(page_html, "html.parser")
        title_paragraphs = soup_parser.find_all("p", class_="title")
        page_titles = [p_tag.find("a").get_text() for p_tag in title_paragraphs]
        all_post_titles.extend(page_titles)  # Use extend to add elements from list

        # Find the next page URL
        next_button_span = soup_parser.find("span", class_="next-button")
        if next_button_span and next_button_span.find("a"):
            current_page_url = next_button_span.find("a")['href']
        else:
            current_page_url = None  # Reached the end

        # Be polite and wait before the next request
        time.sleep(4)  # Pause for 4 seconds

    except requests.exceptions.RequestException as e:
        print(f"Error fetching page {current_page_url}: {e}")
        break  # Stop if there's a network/HTTP error
    except Exception as e:
        print(f"An error occurred during parsing: {e}")
        break

print(f"\nFinished scraping. Collected {len(all_post_titles)} titles.")

Here's the consolidated multi-page script:

import requests
from bs4 import BeautifulSoup
import time

all_post_titles = []
current_page_url = "https://old.reddit.com/r/programming/"
scrape_page_count = 15  # How many pages to scrape
headers = {'User-agent': 'Python Scraping Bot - Learning Project 1.0'}

print("Starting scraper...")

for page_num in range(scrape_page_count):
    if not current_page_url:
        print("No more pages found. Stopping.")
        break

    print(f"Scraping page {page_num + 1}: {current_page_url}")

    try:
        response = requests.get(current_page_url, headers=headers)
        response.raise_for_status()  # Raise an exception for bad status codes
        page_html = response.content

        soup_parser = BeautifulSoup(page_html, "html.parser")
        title_paragraphs = soup_parser.find_all("p", class_="title")
        page_titles = [p_tag.find("a").get_text() for p_tag in title_paragraphs]
        all_post_titles.extend(page_titles)

        # Find the next page URL
        next_button_span = soup_parser.find("span", class_="next-button")
        if next_button_span and next_button_span.find("a"):
            current_page_url = next_button_span.find("a")['href']
        else:
            print("Could not find 'next' button. Assuming last page.")
            current_page_url = None

        # Be polite!
        print("Pausing for 4 seconds...")
        time.sleep(4)

    except requests.exceptions.RequestException as e:
        print(f"Error fetching page {current_page_url}: {e}")
        break
    except Exception as e:
        print(f"An error occurred: {e}")
        break

print(f"\nFinished scraping. Collected {len(all_post_titles)} titles.")
# Optional: print all titles
# print(all_post_titles)

Analyzing the results: most-mentioned languages

With a list of titles in hand, let's count mentions of popular programming languages. Start with a dictionary of the languages to track, all lowercase for case-insensitive matching:

language_mentions = {
    "python": 0, "javascript": 0, "java": 0, "c#": 0, "c++": 0,
    "c": 0, "go": 0, "rust": 0, "php": 0, "swift": 0, "kotlin": 0,
    "ruby": 0, "typescript": 0, "html": 0, "css": 0, "sql": 0
}

Now process the titles: lowercase each one, split it into words, and count matches against the dictionary.

import re  # Import regular expressions for better word splitting

all_words = []
for title in all_post_titles:
    # Split title into words, convert to lowercase, remove basic punctuation
    words_in_title = re.findall(r'\b\w+\b', title.lower())
    all_words.extend(words_in_title)

# Count mentions
for word in all_words:
    if word in language_mentions:
        language_mentions[word] += 1

# Print the results nicely
print("\nProgramming Language Mention Counts:")

# Sort results by count descending
sorted_mentions = sorted(language_mentions.items(), key=lambda item: item[1], reverse=True)

for lang, count in sorted_mentions:
    if count > 0:  # Only show languages that were actually mentioned
        print(f"- {lang.capitalize()}: {count}")

Append this block to the end of reddit_scraper.py, after the scraping loop. Running the full script scrapes the pages, then prints a ranked list of language mentions.

Two honest limitations. The counts vary depending on which posts are active when you run it. And naive word matching catches false positives: "go" in "let's go" or the bare letter "c" will inflate results. For a serious analysis you'd add context checks, weighting, or a small tokenizer, but this is enough to see the idea working.

Making scraping reliable with proxies

Once you scrape more frequently or across many pages, you'll hit rate limits. Sending many rapid requests from a single IP address strains the target server and often triggers temporary throttling or CAPTCHA challenges, which stall your script.

Proxies help here in a legitimate way. A proxy acts as an intermediary: your request goes to the proxy, which forwards it to the target site. Distributing requests across a pool of rotating residential IPs spreads the load naturally, keeps you well within reasonable request rates per address, and improves your success rate on public data collection.

Free proxy lists tend to be slow, unstable, and of unknown origin. For anything you rely on, a reputable paid service pays for itself in reliability. If your targets render content with JavaScript, a browser-driven approach is worth reading up on in our guide to dynamic scraping with Selenium and Python.

Adding Evomi proxies to the script

Evomi offers residential, mobile, datacenter, and static ISP proxies, all ethically sourced and operated under Swiss privacy standards. Residential proxies start at $0.49/GB, and there are free trials on residential, mobile, and datacenter plans so you can test before committing. After signing up you'll pick a proxy type and get your endpoint, port, username, and password.

Wiring proxies into Requests is a one-liner change. Define a dictionary that maps HTTP and HTTPS traffic to your proxy URL (swap in your real credentials):

# Replace with your actual Evomi credentials and endpoint
evomi_proxy_user = "your-evomi-username"
evomi_proxy_pass = "your-evomi-password"
evomi_proxy_endpoint = "rp.evomi.com"  # Residential proxy endpoint
evomi_proxy_port_http = "1000"          # HTTP port for residential

proxy_url_http = f"http://{evomi_proxy_user}:{evomi_proxy_pass}@{evomi_proxy_endpoint}:{evomi_proxy_port_http}"

# The same HTTP proxy URL typically works for HTTPS via CONNECT tunneling:
proxy_url_https = proxy_url_http

PROXIES = {
    "http": proxy_url_http,
    "https": proxy_url_https
}

Then add the proxies argument to your requests.get() call inside the loop:

response = requests.get(
    current_page_url,
    headers=headers,
    proxies=PROXIES
)

Your traffic now routes through the proxy. With a rotating residential endpoint, each request can come from a different IP in the pool, which keeps per-address request rates low. It's good practice to verify your setup with a quick tool like Evomi's proxy tester before running a long job.

Where to go next

You now have a complete, working scraper: fetch, parse, paginate, analyze, and route through proxies. From here you can adapt the selectors to other public pages, add data storage (CSV, JSON, or a database), or move to Scrapy for larger crawls. If you scrape sites with heavy client-side rendering, look into a managed Scraping Browser setup that handles headless Chromium for you. And whatever you scrape, keep it to public data, respect robots.txt and each site's terms, and pace your requests so you're a considerate visitor.