Playwright Web Scraping in Python, with Per-Page Proxies

Playwright drives a real Chromium, Firefox or WebKit browser from code. For web scraping, that matters when a page builds its content with JavaScript: a plain HTTP request gets an empty shell, while Playwright sees what a visitor sees. It also waits for elements on its own, which removes most of the timing bugs that made older browser scrapers fragile.

This guide builds a Playwright scraper in Python step by step. It covers a first scraper with pagination, an infinite-scroll page read through its own API, parallel pages with async workers, and proxies set per browser, per context or per page. Every script was run with Playwright 1.63 for Python in October 2026 against quotes.toscrape.com, a sandbox site made for scraping practice, and the outputs shown are from those runs.

Is Playwright good for web scraping?

Yes, for the right pages. Use it when:

• the data appears only after JavaScript runs (check the page source: if the values aren't in it, you need a browser),
• you need to click, type, scroll or log in to reach the data,
• or the page loads its data from an API you want to observe rather than reverse-engineer.

If the HTML the server sends already contains the data, skip the browser. requests with a parser is many times lighter, and the same scraper costs a fraction of the CPU, memory and bandwidth. A browser loads every script, stylesheet and image unless you tell it not to. For how Playwright compares with the alternatives, see Playwright vs Puppeteer for web scraping and our Puppeteer, Playwright and Selenium showdown.

Install Playwright for Python

python -m venv .venv && source .venv/bin/activate
python -m pip install playwright
python -m playwright install chromium

The last command downloads the browser build Playwright is tested against. On a fresh Linux server, python -m playwright install --with-deps chromium also installs the system libraries Chromium needs.

Your first Playwright scraper

The /js/ version of the practice site renders its quotes with JavaScript, ten per page, with a "Next" link. This script reads every page and writes a CSV:

import csv
from urllib.parse import urljoin

from playwright.sync_api import sync_playwright

START = "https://quotes.toscrape.com/js/"


def scrape():
    rows = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        url = START
        while url:
            page.goto(url, wait_until="domcontentloaded")
            cards = page.locator("div.quote")
            cards.first.wait_for()  # the quotes are rendered by JavaScript
            for card in cards.all():
                rows.append({
                    "text": card.locator("span.text").inner_text(),
                    "author": card.locator("small.author").inner_text(),
                    "tags": ";".join(card.locator("a.tag").all_inner_texts()),
                })
            next_link = page.locator("li.next a")
            url = urljoin(page.url, next_link.get_attribute("href")) if next_link.count() else None
            page.wait_for_timeout(1000)  # stay polite: about one page a second
        browser.close()
    return rows


if __name__ == "__main__":
    quotes = scrape()
    with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
        writer.writeheader()
        writer.writerows(quotes)
    print(f"Saved {len(quotes)} quotes")

$ python first_scrape.py
Saved 100 quotes

What each part does:

• page.locator("div.quote") describes elements without fetching them yet. A locator is evaluated when you use it, so it always reflects the current page.
• cards.first.wait_for() waits until the first quote is attached to the page. The site's JavaScript runs after domcontentloaded, so reading immediately would find nothing.
• inner_text() and all_inner_texts() return the text a visitor sees, with whitespace normalised.
• urljoin(page.url, href) turns the relative "Next" link into a full URL. When there is no "Next" link, count() is 0 and the loop ends.
• wait_for_timeout(1000) is a deliberate pause between pages, not a way to wait for content. Keep the request rate low on any site you scrape.

Wait for the right thing

Most broken browser scrapers either read too early or sleep too long. Playwright gives you better options than a fixed delay:

Avoid networkidle on pages with analytics beacons, chat widgets or long polling. They never go quiet, and the call times out after 30 seconds. Waiting for the element you need is almost always the better signal.

Scrape infinite scroll by reading the page's own API

The practice site's /scroll page loads more quotes as you scroll. You could scroll and parse the HTML, but the page is fetching JSON from /api/quotes?page=N, and that JSON is cleaner than anything you'd extract from the DOM. Playwright can listen to those responses while it scrolls:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    # Skip images, fonts and media: the data is in the JSON, not the pixels.
    page.route(
        "**/*",
        lambda route: route.abort()
        if route.request.resource_type in {"image", "font", "media"}
        else route.continue_(),
    )

    batches = {}

    def collect(response):
        if "/api/quotes" in response.url and response.ok:
            data = response.json()
            batches[data["page"]] = data

    page.on("response", collect)
    page.goto("https://quotes.toscrape.com/scroll", wait_until="networkidle")

    # Scroll until the newest batch says there is nothing left (capped at 50 scrolls).
    for _ in range(50):
        if batches and not batches[max(batches)]["has_next"]:
            break
        page.mouse.wheel(0, 20000)
        page.wait_for_timeout(1000)

    quotes = [q for n in sorted(batches) for q in batches[n]["quotes"]]
    print(f"{len(quotes)} quotes from {len(batches)} API responses")
    print(quotes[0]["author"]["name"], "-", quotes[0]["text"][:60])
    browser.close()

$ python scroll_api.py
100 quotes from 12 API responses
Albert Einstein - “The world as we have created it is a process of our thinkin

Three things make this reliable:

• Responses are stored by page number, so a batch that arrives twice or out of order is counted once. Our first version waited for one response per scroll with expect_response. It failed intermittently, because a single long scroll can trigger two loads.
• The loop stops on the API's own has_next flag, not on a guess about page height, with a hard cap so a broken page can't scroll forever.
• page.route aborts images, fonts and media. The data doesn't need them, and through a proxy billed by traffic they are most of the bytes.

Once you know the endpoint, check whether you need the browser at all. If the API answers a plain HTTP request, call it directly and drop Playwright for that site.

Run pages in parallel with async Playwright

One browser can hold several isolated contexts, each with its own cookies and, optionally, its own proxy. The async API lets a few workers share a queue of URLs:

import asyncio
import os

from playwright.async_api import async_playwright

URLS = [f"https://quotes.toscrape.com/js/page/{n}/" for n in range(1, 11)]
CONCURRENCY = 3


def proxy_for(worker: int):
    server = os.environ.get("PROXY_SERVER")
    if not server:
        return None
    # One sticky session per worker keeps each context on one IP.
    password = f"{os.environ['PROXY_PASS']}_session-worker{worker}"
    return {"server": server, "username": os.environ["PROXY_USER"], "password": password}


async def worker(n, browser, queue, results):
    context = await browser.new_context(proxy=proxy_for(n))
    page = await context.new_page()
    while True:
        try:
            url = queue.get_nowait()
        except asyncio.QueueEmpty:
            break
        for attempt in range(3):
            try:
                await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
                await page.locator("div.quote").first.wait_for(timeout=10_000)
                results[url] = await page.locator("span.text").all_inner_texts()
                break
            except Exception as exc:
                print(f"worker {n}: {url} attempt {attempt + 1} failed: {exc.__class__.__name__}")
                await asyncio.sleep(2 ** attempt)
        await asyncio.sleep(1)  # per-worker pause between pages
    await context.close()


async def main():
    queue = asyncio.Queue()
    for url in URLS:
        queue.put_nowait(url)
    results = {}
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        await asyncio.gather(*(worker(n, browser, queue, results) for n in range(CONCURRENCY)))
        await browser.close()
    print(f"{sum(len(v) for v in results.values())} quotes from {len(results)} pages")


asyncio.run(main())

$ python scrape_async.py
100 quotes from 10 pages

It ran in about 6 seconds, both directly and through a proxy. Keep CONCURRENCY small. Three workers, each pausing a second between pages, is still three times the load of the first scraper, and the right number depends on the site, not on your machine. Retries back off exponentially (1, 2, then 4 seconds), so a slow page doesn't turn into a burst of repeat requests.

One naming trap: don't call this file concurrent.py. That shadows Python's standard concurrent package, which asyncio imports, and the script fails before Playwright even starts. We hit it, and renamed the file scrape_async.py.

Proxies in Playwright: browser, context or page

Playwright accepts a proxy at three levels. The option is the same dictionary each time, server plus optional username and password:

The page level works because browser.new_page() creates a context of its own behind the scenes, so it takes the same options a context does. Here are all three with Evomi's residential proxies, where targeting options go at the end of the password:

import os

from playwright.sync_api import sync_playwright

SERVER = os.environ.get("PROXY_SERVER", "http://rp.evomi.com:1000")
USER = os.environ["PROXY_USER"]
PASS = os.environ["PROXY_PASS"]
CHECK = os.environ.get("CHECK_URL", "https://ip.evomi.com/s")


def exit_ip(page):
    page.goto(CHECK)
    return page.inner_text("body").strip()


with sync_playwright() as p:
    # 1. Browser level: every page in this browser uses the proxy.
    browser = p.chromium.launch(proxy={"server": SERVER, "username": USER, "password": PASS})
    print("browser-level:", exit_ip(browser.new_page()))
    browser.close()

    # 2. Context level: one browser, a different proxy identity per context.
    browser = p.chromium.launch()
    us = browser.new_context(proxy={"server": SERVER, "username": USER, "password": f"{PASS}_country-US"})
    de = browser.new_context(proxy={"server": SERVER, "username": USER, "password": f"{PASS}_country-DE"})
    print("context US:", exit_ip(us.new_page()))
    print("context DE:", exit_ip(de.new_page()))

    # 3. Page level: new_page() accepts the same proxy option, because it creates
    #    a context of its own behind the scenes.
    page = browser.new_page(proxy={"server": SERVER, "username": USER, "password": f"{PASS}_session-abc123"})
    print("page-level, sticky session:", exit_ip(page), exit_ip(page))
    browser.close()

We ran this script against a local proxy that requires a username and password and logs what it receives. It received exactly the four passwords the script sends: the plain one, then _country-US, _country-DE and _session-abc123, each from the browser, context or page that set it. The log also shows how Chromium authenticates. It first opens the tunnel without credentials, gets a 407 from the proxy, and retries with the username and password. That's normal, and you don't need to handle it.

Sessions: keep one IP where the flow needs it

By default an Evomi residential request can leave from a different IP each time. For a browser that's often the wrong default: a site that sets a cookie on one request and checks it on the next, a multi-step form, or a logged-in area of your own account expects one visitor on one address. Add _session-<id> to the password (6 to 10 letters and digits) and every request with that ID keeps the same exit, by default for up to 30 minutes. _lifetime-<minutes> sets a different duration, up to 1,440. In the async scraper above, each worker gets its own session, so each context behaves like one consistent visitor.

Which endpoint to use

Use the HTTP endpoint, http://rp.evomi.com:1000, even for https:// sites. The browser opens a CONNECT tunnel and TLS runs end to end between Chromium and the site. Don't switch to SOCKS5 to "be safer": Chromium can't authenticate to a SOCKS5 proxy, and Playwright refuses at launch with Browser does not support socks5 proxy authentication. If you use the HTTPS proxy port (1001), the hostname is rp.evomi-proxy.com, which is the name its certificate covers. Full settings are in the Evomi Playwright integration guide.

To confirm what a context is using before a long run, open https://ip.evomi.com/s in it as exit_ip() does above, or test the credentials first with the Evomi proxy checker.

Troubleshooting

• A locator finds 0 elements. It was read before the JavaScript ran, or the selector is wrong. Call wait_for() on the locator and check the selector in DevTools.
• goto times out with networkidle. The page never stops making requests. Wait for a specific element or response instead.
• Timeout 30000ms exceeded on the first page, only behind a proxy. Usually wrong proxy credentials. Check the username, password and options, then retry.
• "Browser does not support socks5 proxy authentication". SOCKS5 with credentials doesn't work in Chromium. Use the HTTP proxy endpoint.
• Infinite scroll stops early or double-counts. The script waits for one response per scroll. Collect responses by page number and stop on the API's own flag.
• The script fails at import with an asyncio error. A file named concurrent.py or asyncio.py shadows the standard library. Rename it.
• Browser executable not found. The browsers weren't installed in this environment. Run python -m playwright install chromium.

Wrong proxy credentials are worth singling out. In our tests, Playwright reported them as a navigation timeout, not as a 407, so a scraper that "suddenly can't load anything" behind a proxy should have its credentials checked before its selectors.

Scrape responsibly

Prefer an official API when the site offers one. Read the site's terms and robots.txt, keep the request rate low, and collect public data only. A browser makes it easy to load far more than you need, so block what you don't use and cache what you've already fetched. If you'd rather not run browsers yourself, a hosted option such as a scraping browser moves that work elsewhere, and how headless browsers work explains what is running under the hood.

FAQ

Is Playwright good for web scraping?

Yes, for JavaScript-rendered pages and anything that needs interaction. Its auto-waiting locators, network interception and isolated contexts make browser scrapers far less fragile than older tools. For static HTML, a plain HTTP client is lighter and faster.

Can Playwright be used for web scraping with Python?

Yes. Playwright has an official Python package with both a synchronous API (good for scripts) and an async API (good for many pages in parallel). Every example in this guide is Python.

Can I use a different proxy for each page in Playwright?

Yes. Pass proxy= to browser.new_page() for a single page, or to browser.new_context() for a group of pages. Each context keeps its own proxy, cookies and storage, so you can run several exits side by side in one browser.

Does Playwright support proxy authentication?

Yes, for HTTP and HTTPS proxies, through the username and password fields of the proxy option. Chromium doesn't support authenticated SOCKS5 proxies, so use the HTTP endpoint with credentials.

Is Playwright better than Selenium or Puppeteer for scraping?

Playwright's auto-waiting, its per-context proxies and its Python, Node.js, Java and .NET support make it the most flexible of the three for new scrapers. Puppeteer is Node.js-first and Chromium-first. Selenium is the standard for cross-browser testing and fine for existing suites. The data you get is the same. The difference is how much waiting and plumbing you write yourself.

Which is the best tool for web scraping?

It depends on the page. Use an official API when there is one, an HTTP client and parser for static HTML, and a browser tool like Playwright only for pages that need JavaScript or interaction. Many scrapers combine them, with the browser used to discover an API that the HTTP client then calls directly.