Async Web Scraping with Aiohttp & Proxies: A Guide

If you're pulling public data at any real scale, the speed of your HTTP client matters. A scraper that fetches one page, waits, then fetches the next spends most of its life idle. That's fine for a handful of URLs and painful when you have thousands. This is where aiohttp earns its place: it's an asynchronous HTTP client built on Python's asyncio framework, so it can keep dozens of requests in flight at once instead of blocking on each one.

This guide walks through building a practical async scraper with aiohttp, parsing the results properly with BeautifulSoup, and routing traffic through rotating proxies so your request volume stays polite and distributed. Everything here is aimed at collecting publicly available data for legitimate purposes like price monitoring, research, and QA testing, within each target site's terms of service.

Why Async Makes Sense for Scraping

Web scraping is overwhelmingly I/O-bound. Your program isn't crunching numbers; it's waiting on network round-trips. When you send a request with a synchronous library, the interpreter sits idle until the server responds, then it moves on to the next URL. Multiply that wait by a few thousand pages and you've built a very slow tool.

Asynchronous code flips this. With aiohttp and asyncio, your program fires off a request and immediately starts the next one instead of waiting. When a response comes back, its handler picks up where it left off. The result is many concurrent connections managed by a single thread, with dramatically better throughput for network-heavy work. If you want a broader look at how aiohttp compares to alternatives like httpx and requests, the trade-offs come down to sync vs. async and API ergonomics.

Why Proxies Belong in the Picture

Concurrency is a double-edged sword. Firing off many requests quickly from a single IP address is exactly the traffic pattern that public sites throttle or rate-limit. Sending everything from one address is also just impolite: it concentrates load and gives the target server no way to distinguish your research traffic from anything else.

Routing requests through proxies spreads that load across many IP addresses. For most public-data work, rotating residential proxies are the natural fit because they use IP addresses assigned to real households, so your requests look like ordinary visitors from real geographies. Evomi's residential pool is ethically sourced and starts at $0.49/GB, and if a task is less sensitive, datacenter proxies from $0.30/GB are a cheaper option. There's a free trial on residential, mobile, and datacenter tiers if you want to test before committing.

A quick note on etiquette: proxies are a load-distribution and privacy tool, not a license to hammer a site. Respect robots.txt, keep request rates reasonable, and stay inside the site's terms.

What You'll Need

• Python 3.7 or newer — asyncio's modern syntax and asyncio.run() require it.

• aiohttp — the async HTTP client.

• BeautifulSoup — for parsing HTML properly rather than fishing through raw strings.

• Proxy access — rotating residential or datacenter proxies. You can grab a free trial to follow along.

Installing the Libraries

Install both packages with pip:

Confirm aiohttp is in place and check its version:

A Basic Aiohttp Scraper with a Proxy

Let's start with a minimal script that fetches laptop product names from a public scraping sandbox, routing the request through a single Evomi residential proxy. We'll target the well-known webscraper.io test site, which exists specifically for this kind of practice.

Step 1: Import the Libraries

import aiohttp
import asyncio

Step 2: Configure Your Proxy

Define the proxy endpoint. Evomi's residential proxies accept credentials embedded in the URL. Swap in your own username and password.

# Evomi Residential Proxy Configuration (replace with your details)
# Format: http://username:password@hostname:port
proxy_url = "http://user-xyz:pass123@rp.evomi.com:1000"

# If you prefer to pass credentials separately, aiohttp supports BasicAuth:
# proxy_url = "http://rp.evomi.com:1000"
# proxy_auth = aiohttp.BasicAuth('user-xyz', 'pass123')
# For this example, we embed auth in the proxy_url.

Ports: Evomi residential proxies use 1000 for HTTP, 1001 for HTTPS, and 1002 for SOCKS5. Pick the protocol you need.

Step 3: Write the Async Fetching Function

This function performs the request through the proxy and returns the raw HTML. We keep the fetch and the parsing separate, which is a cleaner pattern than mixing extraction logic into the request.

# Async function to fetch a page's HTML through the proxy
async def fetch_html(session, url):
    print(f"Attempting to fetch data from: {url}")
    try:
        async with session.get(url, proxy=proxy_url) as response:
            response.raise_for_status()  # Raises for 4xx/5xx status codes
            html_content = await response.text()
            print("Successfully fetched data.")
            return html_content
    except aiohttp.ClientError as e:
        print(f"An error occurred: {e}")
        return None

A few things worth calling out: async def marks the function as non-blocking; session.get(..., proxy=proxy_url) routes the request through your proxy; response.raise_for_status() turns failed responses into exceptions you can catch; and await response.text() yields control back to the event loop while the body downloads, letting other tasks run in the meantime.

Step 4: Parse the HTML Properly

The original approach of scanning raw HTML lines for class="title" is fragile — one change in whitespace or markup breaks it. BeautifulSoup parses the document into a real tree so you can select elements by class and read their attributes cleanly. On the test site, each product title lives in an anchor with the class title and stores the full name in its title attribute.

from bs4 import BeautifulSoup

def parse_product_names(html_content):
    if not html_content:
        return []
    soup = BeautifulSoup(html_content, "html.parser")
    # Each product title is an  with the full name in its title attribute
    return [a.get("title", a.get_text(strip=True)) for a in soup.select("a.title")]

Step 5: Orchestrate with a Main Function

The main() coroutine creates a reusable session, fetches the page, parses it, and prints the results.

async def main():
    target_url = "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops"
    async with aiohttp.ClientSession() as session:
        print("Client session started.")
        html = await fetch_html(session, target_url)
        product_names = parse_product_names(html)

        if product_names:
            print("\n--- Extracted Product Names ---")
            for name in product_names:
                print(name)
            print("-------------------------------")
        else:
            print("No product names extracted.")
    print("Client session closed.")

Using a single ClientSession is important: it pools connections and reuses them, which is far more efficient than opening a fresh connection per request.

Step 6: Run It

if __name__ == "__main__":
    print("Starting scraper...")
    asyncio.run(main())
    print("Scraper finished.")

The Complete Basic Script

import aiohttp
import asyncio
from bs4 import BeautifulSoup

# Evomi Residential Proxy Configuration (replace with your details)
# Format: http://username:password@hostname:port
proxy_url = "http://user-xyz:pass123@rp.evomi.com:1000"


async def fetch_html(session, url):
    # Hide credentials in the log output
    print(f"Fetching {url} via proxy {proxy_url.split('@')[-1]}")
    try:
        async with session.get(url, proxy=proxy_url) as response:
            response.raise_for_status()
            return await response.text()
    except aiohttp.ClientError as e:
        print(f"An error occurred while fetching {url}: {e}")
        return None


def parse_product_names(html_content):
    if not html_content:
        return []
    soup = BeautifulSoup(html_content, "html.parser")
    return [a.get("title", a.get_text(strip=True)) for a in soup.select("a.title")]


async def main():
    target_url = "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops"
    async with aiohttp.ClientSession() as session:
        print("Client session started.")
        html = await fetch_html(session, target_url)
        product_names = parse_product_names(html)

        if product_names:
            print("\n--- Extracted Product Names ---")
            for name in product_names:
                print(name)
            print("-------------------------------")
        else:
            print("No product names extracted.")
    print("Client session closed.")


if __name__ == "__main__":
    print("Starting scraper...")
    asyncio.run(main())
    print("Scraper finished.")

Run this and it connects through your Evomi proxy, downloads the laptop listings, and prints clean product names. That's the foundation. Now let's make it robust.

Rotating Proxies and Scraping Concurrently

A single proxy is a single point of failure and a single IP for the target to rate-limit. The stronger pattern is to keep a list of proxies, pick one per request, and run multiple fetches concurrently — which is exactly what aiohttp was built for.

Step 1: Import Extra Libraries

import aiohttp
import asyncio
import random
from bs4 import BeautifulSoup

Step 2: Define Your Proxy List

Here we use Evomi datacenter proxy endpoints as an example. Datacenter proxies are cost-effective for lighter targets, while residential is the better call for stricter sites.

# List of Evomi Datacenter Proxies (replace with your actual proxies)
# Format: http://username:password@hostname:port
proxy_list = [
    "http://user-dc1:pass123@dc.evomi.com:2000",
    "http://user-dc2:pass456@dc.evomi.com:2000",
    "http://user-dc3:pass789@dc.evomi.com:2000",
    "http://user-dc4:passabc@dc.evomi.com:2000",
    "http://user-dc5:passdef@dc.evomi.com:2000",
]

Ports: Evomi datacenter proxies use 2000 (HTTP), 2001 (HTTPS), and 2002 (SOCKS5).

Step 3: A Fetching Function That Takes a Proxy

Now the function accepts both a URL and a specific proxy, adds a timeout so a slow proxy can't stall the whole run, and handles proxy-specific errors distinctly.

async def fetch_page_data(session, page_url, proxy):
    # Log the proxy host only, never the credentials
    proxy_host = "N/A"
    if proxy:
        try:
            proxy_host = proxy.split('@')[-1].split(':')[0]
        except IndexError:
            proxy_host = "Invalid Format"

    print(f"Fetching {page_url} using proxy {proxy_host}...")
    try:
        async with session.get(
            page_url, proxy=proxy, timeout=aiohttp.ClientTimeout(total=15)
        ) as response:
            response.raise_for_status()
            html_content = await response.text()
            soup = BeautifulSoup(html_content, "html.parser")
            product_names = [
                a.get("title", a.get_text(strip=True))
                for a in soup.select("a.title")
            ]
            print(f"Successfully fetched {page_url}")
            return product_names

    except aiohttp.ClientProxyConnectionError as e:
        print(f"Proxy Connection Error for {proxy_host}: {e}")
        return None
    except aiohttp.ClientError as e:
        print(f"Client Error fetching {page_url} via {proxy_host}: {e}")
        return None
    except asyncio.TimeoutError:
        print(f"Timeout fetching {page_url} via {proxy_host}")
        return None

Note the layered except blocks: a proxy connection failure is worth distinguishing from a general client error or a timeout, because each suggests a different fix — a dead proxy, a bad request, or an unresponsive server.

Step 4: Run Requests Concurrently in Main

This is where async pays off. Instead of looping and awaiting each request one at a time, we build a list of coroutines and hand them to asyncio.gather(), which runs them concurrently. Each request picks a random proxy from the pool.

async def main():
    base_url = "https://webscraper.io/test-sites/e-commerce/allinone/computers/laptops"

    # Build a batch of target URLs. On a paginated site you would vary
    # these; here we reuse the base URL to demonstrate concurrency.
    num_requests = 5
    target_urls = [base_url] * num_requests

    async with aiohttp.ClientSession() as session:
        tasks = [
            fetch_page_data(session, url, random.choice(proxy_list))
            for url in target_urls
        ]
        # Run all requests concurrently and collect the results
        results = await asyncio.gather(*tasks)

    # Flatten successful results, ignoring any that failed (returned None)
    all_products = []
    for result in results:
        if result:
            all_products.extend(result)

    unique_products = sorted(set(all_products))
    print(f"\nCollected {len(unique_products)} unique product names:")
    for name in unique_products:
        print(name)


if __name__ == "__main__":
    asyncio.run(main())

Because random.choice(proxy_list) is called for each request, the load spreads across your pool. asyncio.gather() collects every result once all the coroutines finish, and we filter out any None values from failed requests before deduplicating the names.

Good Practices for Aiohttp Scraping

• Always set a timeout. Without one, a single unresponsive proxy or server can hang your entire batch.

• Cap concurrency. For large jobs, use an asyncio.Semaphore to limit how many requests run at once. Firing thousands simultaneously is both hard on the target and easy to trip rate limits.

• Reuse one session. Create a single ClientSession for the whole run so connections are pooled.

• Parse, don't string-search. BeautifulSoup or lxml survives markup changes far better than raw text matching.

• Retry transient failures. A timeout or a 503 often succeeds on a second try with a different proxy.

• Respect the site. Honor robots.txt, throttle politely, and only collect data you're permitted to.

If you'd rather skip proxy management and JavaScript rendering entirely, Evomi's Scraping Browser offers a managed headless Chromium endpoint (Playwright- and Puppeteer-compatible) with rotation handled for you. And before deploying, it's worth confirming your setup with the free proxy tester to make sure your endpoints and credentials are working.

Where to Go Next

Once you're comfortable with the async fundamentals, the same patterns apply across sites and languages. If you want to see BeautifulSoup used in more depth, our Python and Beautiful Soup guide is a good next read. For a target that needs sessions and authentication, see scraping login-only sites with Python. And if JavaScript-heavy pages are your challenge, the JavaScript and Node.js scraping guide covers the browser-driven approach.