Expedia Scraping with Python & Proxies: A Guide

Sarah Whitmore

Scraping Techniques

Expedia publishes an enormous amount of public information: hotel names, ratings, nightly prices, availability windows, and more. If you're doing legitimate market research, tracking published rates for properties you manage, or building a travel comparison feature, that public data is genuinely useful. The catch is that Expedia is a modern JavaScript-heavy site, so a plain HTML parser won't get you very far.

This guide walks through a practical, working approach to collecting public hotel pricing data from Expedia using Python and Playwright. We'll build up the script step by step, handle the "Show More Results" pattern that Expedia uses to lazy-load listings, and then route everything through proxies so a larger collection job runs reliably. Always stay within Expedia's Terms of Service and only gather publicly visible information.

Why Expedia Is Harder to Scrape Than It Looks

Like most contemporary travel sites, Expedia renders much of its content with JavaScript after the initial page load. Prices, ratings, and even the list of hotels themselves often appear only once client-side code has executed and additional API calls have resolved.

That's a problem for traditional, static-HTML tools such as Beautiful Soup. They fetch the raw markup but can't run the JavaScript that populates the page, so the data you actually want simply isn't there yet. (Beautiful Soup is still excellent for simpler, server-rendered pages — see our Beautiful Soup and proxy tips guide for those cases.)

To read the fully rendered page, you need something that behaves like a real browser.

The Right Tool: A Headless Browser

Headless browsers that you control programmatically — Playwright or Puppeteer — are the natural fit here. They execute JavaScript, wait for network requests, interact with page elements, and expose the fully rendered DOM.

Playwright is a particularly strong choice for dynamic sites. It ships official support for Python, JavaScript, Java, and C#, has a clean waiting model, and handles multiple browser engines (Chromium, Firefox, WebKit). We'll use the Python bindings throughout. If you'd rather work in Selenium, the concepts map cleanly onto our dynamic scraping with Selenium guide.

Setting Up Your Environment

You'll need Python installed first. If you don't have it, grab it from the official Python website and follow the installation instructions for your OS.

Then install Playwright and its browser binaries. Open a terminal and run:

pip install playwright
playwright install  # Installs browser binaries (Firefox, Chromium, WebKit)

Loading an Expedia Search Results Page

We'll instruct Playwright to open an Expedia hotel search page for a chosen city and date range. The snippet below launches Firefox in non-headless mode (so you can watch it work) and navigates to search results for Rome. Switch headless to True once you're happy with the behaviour.

import time
from playwright.sync_api import sync_playwright

# Define search parameters
destination = "Rome (and vicinity), Lazio, Italy"
checkin_date = "2024-07-10"
checkout_date = "2024-07-17"
adults = 2
rooms = 1

# Construct the URL (simplified example - real URLs might be more complex)
expedia_url = f"https://www.expedia.com/Hotel-Search?destination={destination}&startDate={checkin_date}&endDate={checkout_date}&adults={adults}&rooms={rooms}&sort=RECOMMENDED"

print(f"Navigating to: {expedia_url}")

with sync_playwright() as playwright_instance:
    browser = playwright_instance.firefox.launch(
        headless=False,  # Set to True for background execution
    )
    page = browser.new_page()
    page.goto(expedia_url)

    # Allow time for dynamic content to load
    print("Page loaded, waiting for dynamic content...")
    time.sleep(5)  # Increased wait time for potentially slow loads

    print("Closing browser.")
    browser.close()

Note: We're using a pre-built URL for clarity. A more advanced version could automate filling in the search form on Expedia's homepage and clicking Search — a good exercise once you've finished this tutorial.

Locating and Extracting Hotel Cards

Once the results load, each hotel is rendered as a card. Playwright uses selectors to find elements; Expedia exposes stable data-stid attributes we can target. First, collect all the cards:

# Inside the 'with sync_playwright() as ...' block:
hotel_cards = page.locator('[data-stid="lodging-card-responsive"]').all()
print(f"Found {len(hotel_cards)} initial hotel cards.")

Now loop through the cards and pull out the title, rating, and nightly price using locators scoped to each card:

# Inside the 'with sync_playwright() as ...' block, after locating cards:
extracted_hotels = []

for card in hotel_cards:
    # Use locators relative to the card element
    content_section = card.locator('div.uitk-card-content-section')

    # Extract title
    hotel_title_element = content_section.locator('h3')
    hotel_title = hotel_title_element.text_content().strip() if hotel_title_element.is_visible() else "N/A"

    # Extract rating (handle cases where it might be missing)
    rating_element = content_section.locator('span.uitk-badge-base-text')
    hotel_rating = rating_element.text_content().strip() if rating_element.is_visible() else "No Rating"

    # Extract price (handle cases where it might be missing)
    # More specific selector example
    price_element = content_section.locator('div[data-test-id="price-summary"] .uitk-text .uitk-type-500')

    # Fallback if the primary price selector fails
    if not price_element.is_visible():
        # Original selector as fallback
        price_element = content_section.locator('div.uitk-type-500')

    hotel_price = price_element.text_content().strip() if price_element.is_visible() else "Price Unavailable"

    hotel_data = {
        'title': hotel_title,
        'rating': hotel_rating,
        'price': hotel_price
    }
    extracted_hotels.append(hotel_data)
    # Optional: Print progress for each hotel
    # print(f"Extracted: {hotel_data}")

Notice the .is_visible() checks. Not every hotel shows a rating (new listings) or a price (sold out for the selected dates). These guards stop the script from crashing when an element is absent and instead store a placeholder like No Rating or Price Unavailable. This defensive style is essential on real-world sites where markup varies from row to row.

Finally, print the collected data:

# Inside the 'with sync_playwright() as ...' block, after the loop:
print("\n--- Extracted Hotel Data ---")
for hotel in extracted_hotels:
    print(hotel)
print("--------------------------")

The Combined First Version

Here's the full script for finding and extracting the initially loaded hotels, with a longer navigation timeout for slower connections:

import time
from playwright.sync_api import sync_playwright

# Define search parameters
destination = "Rome (and vicinity), Lazio, Italy"
checkin_date = "2024-07-10"
checkout_date = "2024-07-17"
adults = 2
rooms = 1

# Construct the URL
expedia_url = f"https://www.expedia.com/Hotel-Search?destination={destination}&startDate={checkin_date}&endDate={checkout_date}&adults={adults}&rooms={rooms}&sort=RECOMMENDED"

print(f"Navigating to: {expedia_url}")

with sync_playwright() as playwright_instance:
    browser = playwright_instance.firefox.launch(
        headless=False,  # Set to True for background execution
    )
    page = browser.new_page()
    # Increase default timeout for navigation
    page.set_default_navigation_timeout(60000)  # 60 seconds
    page.goto(expedia_url)

    print("Page loaded, waiting for dynamic content...")
    time.sleep(5)  # Wait for initial load

    # --- Scrape hotels ---
    hotel_cards = page.locator('[data-stid="lodging-card-responsive"]').all()
    print(f"Found {len(hotel_cards)} initial hotel cards.")

    extracted_hotels = []
    for card in hotel_cards:
        content_section = card.locator('div.uitk-card-content-section')

        hotel_title_element = content_section.locator('h3')
        hotel_title = hotel_title_element.text_content().strip() if hotel_title_element.is_visible() else "N/A"

        rating_element = content_section.locator('span.uitk-badge-base-text')
        hotel_rating = rating_element.text_content().strip() if rating_element.is_visible() else "No Rating"

        # Attempt primary price selector first
        price_element = content_section.locator('div[data-test-id="price-summary"] .uitk-text .uitk-type-500')
        if not price_element.is_visible():
            # Fallback selector
            price_element = content_section.locator('div.uitk-type-500')

        hotel_price = price_element.text_content().strip() if price_element.is_visible() else "Price Unavailable"

        hotel_data = {
            'title': hotel_title,
            'rating': hotel_rating,
            'price': hotel_price
        }
        extracted_hotels.append(hotel_data)

    print("\n--- Initial Extracted Hotel Data ---")
    for hotel in extracted_hotels:
        print(hotel)
    print("------------------------------------")

    print("Closing browser.")
    browser.close()

Running it produces something like the following (prices and availability change constantly):

[
  { 'title': 'Hotel Artemide', 'rating': '9.6', 'price': '$450' },
  { 'title': 'iQ Hotel Roma', 'rating': '9.2', 'price': '$380' },
  { 'title': 'UNAHOTELS Decò Roma', 'rating': '8.8', 'price': '$320' },
  { 'title': 'The Hive Hotel', 'rating': '8.6', 'price': '$295' },
  ...
]

This first pass usually doesn't capture every hotel. Expedia loads additional results as you scroll or when you click a "Show More Results" button — which is exactly where a real browser earns its keep.

Loading All Results by Clicking "Show More"

To gather the complete list, we repeatedly click the "Show More Results" button until it disappears (or an error occurs), then scrape the full set of cards. Add this before the card-scraping loop:

# Inside the 'with sync_playwright() as ...' block, before card scraping:
print("Checking for 'Show More Results' button...")
show_more_button_selector = 'button[data-stid="show-more-results"]'  # Verify in browser dev tools

while page.locator(show_more_button_selector).is_visible():
    print("Found 'Show More Results' button, clicking...")
    try:
        page.locator(show_more_button_selector).click(timeout=10000)  # 10 second timeout for click
        print("Waiting for more results to load...")
        # Wait for network activity to settle or just a fixed delay
        page.wait_for_load_state('networkidle', timeout=15000)  # Wait up to 15s for network to be idle
        # Alternative fixed wait: time.sleep(4)
    except Exception as e:
        print(f"Could not click 'Show More' or timed out waiting: {e}")
        break  # Exit loop if button disappears or errors occur

print("'Show More Results' button no longer visible or process finished.")
# Now proceed to scrape *all* the cards that are currently loaded

The loop looks for the button, clicks it if visible, and waits for new content to settle using networkidle (or a fixed delay as a fallback). It keeps going until the button vanishes or something errors out. This is a reusable pattern for any site that paginates by lazy-loading.

Here's the complete script with the "Show More" logic folded in:

import time
from playwright.sync_api import sync_playwright

# Define search parameters
destination = "Rome (and vicinity), Lazio, Italy"
checkin_date = "2024-07-10"
checkout_date = "2024-07-17"
adults = 2
rooms = 1

# Construct the URL
expedia_url = f"https://www.expedia.com/Hotel-Search?destination={destination}&startDate={checkin_date}&endDate={checkout_date}&adults={adults}&rooms={rooms}&sort=RECOMMENDED"

print(f"Navigating to: {expedia_url}")

with sync_playwright() as playwright_instance:
    browser = playwright_instance.firefox.launch(
        headless=False,  # Set to True for background execution
    )
    page = browser.new_page()
    page.set_default_navigation_timeout(60000)  # 60 seconds
    page.goto(expedia_url)

    print("Page loaded, initial wait...")
    time.sleep(5)  # Wait for initial load

    # --- Handle "Show More Results" ---
    print("Checking for 'Show More Results' button...")
    # Note: Selector might change, inspect element if needed
    show_more_button_selector = 'button[data-stid="show-more-results"]'

    while page.locator(show_more_button_selector).is_visible():
        print("Found 'Show More Results' button, clicking...")
        try:
            page.locator(show_more_button_selector).click(timeout=10000)
            print("Waiting for more results to load...")
            # Wait for network activity to settle or just a fixed delay
            page.wait_for_load_state('networkidle', timeout=15000)
            # time.sleep(4)  # Alternative fixed wait
        except Exception as e:
            print(f"Could not click 'Show More' or timed out waiting: {e}")
            break

    print("'Show More Results' button no longer visible or process finished.")

    # --- Scrape all loaded hotels ---
    print("Scraping all loaded hotel cards...")
    hotel_cards = page.locator('[data-stid="lodging-card-responsive"]').all()
    print(f"Found {len(hotel_cards)} total hotel cards.")

    extracted_hotels = []
    for card in hotel_cards:
        content_section = card.locator('div.uitk-card-content-section')

        hotel_title_element = content_section.locator('h3')
        hotel_title = hotel_title_element.text_content().strip() if hotel_title_element.is_visible() else "N/A"

        rating_element = content_section.locator('span.uitk-badge-base-text')
        hotel_rating = rating_element.text_content().strip() if rating_element.is_visible() else "No Rating"

        price_element = content_section.locator('div[data-test-id="price-summary"] .uitk-text .uitk-type-500')
        if not price_element.is_visible():
            price_element = content_section.locator('div.uitk-type-500')  # Fallback

        hotel_price = price_element.text_content().strip() if price_element.is_visible() else "Price Unavailable"

        hotel_data = {
            'title': hotel_title,
            'rating': hotel_rating,
            'price': hotel_price
        }
        extracted_hotels.append(hotel_data)

    print("\n--- Final Extracted Hotel Data ---")
    # Print only the first few and the total count for brevity
    for i, hotel in enumerate(extracted_hotels):
        if i < 5:  # Print first 5
            print(hotel)
        elif i == 5:
            print("...")  # Indicate more data exists
    print(f"(Total: {len(extracted_hotels)} hotels)")
    print("---------------------------------")

    print("Closing browser.")
    browser.close()

Why Proxies Matter for Larger Jobs

Pulling one city and one date range now and then is unlikely to cause any friction — Expedia serves huge amounts of traffic every second. But the moment you want breadth (many cities, many date windows, price snapshots over time) all those requests originate from a single IP address. That concentration is what causes trouble: rate limits, CAPTCHAs, and eventually a temporarily blocked IP.

Web scraping at scale is fundamentally about distributing your requests so no single source hammers the server. A proxy server routes each request through a different IP, so your collection load spreads across many addresses instead of piling onto one. Residential and mobile proxies work well here because their addresses belong to real consumer connections, which is exactly what a travel site expects to see.

Evomi's proxies are ethically sourced and Swiss-based, with residential plans from $0.49/GB and free trials on residential, mobile, and datacenter pools so you can benchmark before committing. Rotating your IP frequently keeps request volume per address low and your job running smoothly.

Adding a Proxy to Playwright

Integrating a proxy is a one-line change to the browser launch. Grab your host, port, username, and password from the Evomi dashboard, then pass a proxy dict:

# Example using Evomi Residential Proxy (HTTP)
proxy_server = "rp.evomi.com:1000"  # Or HTTPS on 1001, SOCKS5 on 1002
proxy_username = "YOUR_EVOMI_USERNAME"
proxy_password = "YOUR_EVOMI_PASSWORD"

browser = playwright_instance.firefox.launch(
    headless=False,  # Keep False for testing, True for production
    proxy={
        'server': proxy_server,
        'username': proxy_username,
        'password': proxy_password,
    }
)
# ... rest of your script (new_page, goto, etc.)

Every request from this browser instance now travels through the specified proxy. To confirm the exit IP is what you expect, you can point the browser at a checker such as geo.evomi.com before running the full job. If you'd rather skip managing browser infrastructure entirely, Evomi's Scraping Browser is a managed, cloud-hosted headless Chromium endpoint (Playwright- and Puppeteer-compatible over wss://browser.evomi.com) with proxying built in.

Practical Tips for Reliable Collection

  • Verify selectors regularly. Expedia updates its front end often. If a run returns empty titles or prices, open dev tools and re-check the data-stid and class names.

  • Prefer explicit waits over long sleeps. wait_for_load_state('networkidle') and expect(locator).to_be_visible() are more robust than fixed time.sleep() calls.

  • Rotate IPs and pace requests. A steady, moderate request rate paired with rotating residential IPs is far more stable than bursts from one address.

  • Store data incrementally. Write results to CSV or a database inside the loop so a crash mid-run doesn't cost you everything.

  • Respect the site. Only collect publicly visible data, follow Expedia's Terms of Service, and keep your footprint reasonable.

Where This Leads

Once you have clean, structured hotel data, the applications are broad: dynamic pricing dashboards that compare rates across dates and providers, availability trend tracking for a market you operate in, or competitive research for a hotel you own. The same Playwright-plus-proxy pattern shown here transfers to plenty of other JavaScript-heavy sites — the selectors change, but the workflow of rendering, paginating, extracting defensively, and distributing requests stays the same.

Expedia publishes an enormous amount of public information: hotel names, ratings, nightly prices, availability windows, and more. If you're doing legitimate market research, tracking published rates for properties you manage, or building a travel comparison feature, that public data is genuinely useful. The catch is that Expedia is a modern JavaScript-heavy site, so a plain HTML parser won't get you very far.

This guide walks through a practical, working approach to collecting public hotel pricing data from Expedia using Python and Playwright. We'll build up the script step by step, handle the "Show More Results" pattern that Expedia uses to lazy-load listings, and then route everything through proxies so a larger collection job runs reliably. Always stay within Expedia's Terms of Service and only gather publicly visible information.

Why Expedia Is Harder to Scrape Than It Looks

Like most contemporary travel sites, Expedia renders much of its content with JavaScript after the initial page load. Prices, ratings, and even the list of hotels themselves often appear only once client-side code has executed and additional API calls have resolved.

That's a problem for traditional, static-HTML tools such as Beautiful Soup. They fetch the raw markup but can't run the JavaScript that populates the page, so the data you actually want simply isn't there yet. (Beautiful Soup is still excellent for simpler, server-rendered pages — see our Beautiful Soup and proxy tips guide for those cases.)

To read the fully rendered page, you need something that behaves like a real browser.

The Right Tool: A Headless Browser

Headless browsers that you control programmatically — Playwright or Puppeteer — are the natural fit here. They execute JavaScript, wait for network requests, interact with page elements, and expose the fully rendered DOM.

Playwright is a particularly strong choice for dynamic sites. It ships official support for Python, JavaScript, Java, and C#, has a clean waiting model, and handles multiple browser engines (Chromium, Firefox, WebKit). We'll use the Python bindings throughout. If you'd rather work in Selenium, the concepts map cleanly onto our dynamic scraping with Selenium guide.

Setting Up Your Environment

You'll need Python installed first. If you don't have it, grab it from the official Python website and follow the installation instructions for your OS.

Then install Playwright and its browser binaries. Open a terminal and run:

pip install playwright
playwright install  # Installs browser binaries (Firefox, Chromium, WebKit)

Loading an Expedia Search Results Page

We'll instruct Playwright to open an Expedia hotel search page for a chosen city and date range. The snippet below launches Firefox in non-headless mode (so you can watch it work) and navigates to search results for Rome. Switch headless to True once you're happy with the behaviour.

import time
from playwright.sync_api import sync_playwright

# Define search parameters
destination = "Rome (and vicinity), Lazio, Italy"
checkin_date = "2024-07-10"
checkout_date = "2024-07-17"
adults = 2
rooms = 1

# Construct the URL (simplified example - real URLs might be more complex)
expedia_url = f"https://www.expedia.com/Hotel-Search?destination={destination}&startDate={checkin_date}&endDate={checkout_date}&adults={adults}&rooms={rooms}&sort=RECOMMENDED"

print(f"Navigating to: {expedia_url}")

with sync_playwright() as playwright_instance:
    browser = playwright_instance.firefox.launch(
        headless=False,  # Set to True for background execution
    )
    page = browser.new_page()
    page.goto(expedia_url)

    # Allow time for dynamic content to load
    print("Page loaded, waiting for dynamic content...")
    time.sleep(5)  # Increased wait time for potentially slow loads

    print("Closing browser.")
    browser.close()

Note: We're using a pre-built URL for clarity. A more advanced version could automate filling in the search form on Expedia's homepage and clicking Search — a good exercise once you've finished this tutorial.

Locating and Extracting Hotel Cards

Once the results load, each hotel is rendered as a card. Playwright uses selectors to find elements; Expedia exposes stable data-stid attributes we can target. First, collect all the cards:

# Inside the 'with sync_playwright() as ...' block:
hotel_cards = page.locator('[data-stid="lodging-card-responsive"]').all()
print(f"Found {len(hotel_cards)} initial hotel cards.")

Now loop through the cards and pull out the title, rating, and nightly price using locators scoped to each card:

# Inside the 'with sync_playwright() as ...' block, after locating cards:
extracted_hotels = []

for card in hotel_cards:
    # Use locators relative to the card element
    content_section = card.locator('div.uitk-card-content-section')

    # Extract title
    hotel_title_element = content_section.locator('h3')
    hotel_title = hotel_title_element.text_content().strip() if hotel_title_element.is_visible() else "N/A"

    # Extract rating (handle cases where it might be missing)
    rating_element = content_section.locator('span.uitk-badge-base-text')
    hotel_rating = rating_element.text_content().strip() if rating_element.is_visible() else "No Rating"

    # Extract price (handle cases where it might be missing)
    # More specific selector example
    price_element = content_section.locator('div[data-test-id="price-summary"] .uitk-text .uitk-type-500')

    # Fallback if the primary price selector fails
    if not price_element.is_visible():
        # Original selector as fallback
        price_element = content_section.locator('div.uitk-type-500')

    hotel_price = price_element.text_content().strip() if price_element.is_visible() else "Price Unavailable"

    hotel_data = {
        'title': hotel_title,
        'rating': hotel_rating,
        'price': hotel_price
    }
    extracted_hotels.append(hotel_data)
    # Optional: Print progress for each hotel
    # print(f"Extracted: {hotel_data}")

Notice the .is_visible() checks. Not every hotel shows a rating (new listings) or a price (sold out for the selected dates). These guards stop the script from crashing when an element is absent and instead store a placeholder like No Rating or Price Unavailable. This defensive style is essential on real-world sites where markup varies from row to row.

Finally, print the collected data:

# Inside the 'with sync_playwright() as ...' block, after the loop:
print("\n--- Extracted Hotel Data ---")
for hotel in extracted_hotels:
    print(hotel)
print("--------------------------")

The Combined First Version

Here's the full script for finding and extracting the initially loaded hotels, with a longer navigation timeout for slower connections:

import time
from playwright.sync_api import sync_playwright

# Define search parameters
destination = "Rome (and vicinity), Lazio, Italy"
checkin_date = "2024-07-10"
checkout_date = "2024-07-17"
adults = 2
rooms = 1

# Construct the URL
expedia_url = f"https://www.expedia.com/Hotel-Search?destination={destination}&startDate={checkin_date}&endDate={checkout_date}&adults={adults}&rooms={rooms}&sort=RECOMMENDED"

print(f"Navigating to: {expedia_url}")

with sync_playwright() as playwright_instance:
    browser = playwright_instance.firefox.launch(
        headless=False,  # Set to True for background execution
    )
    page = browser.new_page()
    # Increase default timeout for navigation
    page.set_default_navigation_timeout(60000)  # 60 seconds
    page.goto(expedia_url)

    print("Page loaded, waiting for dynamic content...")
    time.sleep(5)  # Wait for initial load

    # --- Scrape hotels ---
    hotel_cards = page.locator('[data-stid="lodging-card-responsive"]').all()
    print(f"Found {len(hotel_cards)} initial hotel cards.")

    extracted_hotels = []
    for card in hotel_cards:
        content_section = card.locator('div.uitk-card-content-section')

        hotel_title_element = content_section.locator('h3')
        hotel_title = hotel_title_element.text_content().strip() if hotel_title_element.is_visible() else "N/A"

        rating_element = content_section.locator('span.uitk-badge-base-text')
        hotel_rating = rating_element.text_content().strip() if rating_element.is_visible() else "No Rating"

        # Attempt primary price selector first
        price_element = content_section.locator('div[data-test-id="price-summary"] .uitk-text .uitk-type-500')
        if not price_element.is_visible():
            # Fallback selector
            price_element = content_section.locator('div.uitk-type-500')

        hotel_price = price_element.text_content().strip() if price_element.is_visible() else "Price Unavailable"

        hotel_data = {
            'title': hotel_title,
            'rating': hotel_rating,
            'price': hotel_price
        }
        extracted_hotels.append(hotel_data)

    print("\n--- Initial Extracted Hotel Data ---")
    for hotel in extracted_hotels:
        print(hotel)
    print("------------------------------------")

    print("Closing browser.")
    browser.close()

Running it produces something like the following (prices and availability change constantly):

[
  { 'title': 'Hotel Artemide', 'rating': '9.6', 'price': '$450' },
  { 'title': 'iQ Hotel Roma', 'rating': '9.2', 'price': '$380' },
  { 'title': 'UNAHOTELS Decò Roma', 'rating': '8.8', 'price': '$320' },
  { 'title': 'The Hive Hotel', 'rating': '8.6', 'price': '$295' },
  ...
]

This first pass usually doesn't capture every hotel. Expedia loads additional results as you scroll or when you click a "Show More Results" button — which is exactly where a real browser earns its keep.

Loading All Results by Clicking "Show More"

To gather the complete list, we repeatedly click the "Show More Results" button until it disappears (or an error occurs), then scrape the full set of cards. Add this before the card-scraping loop:

# Inside the 'with sync_playwright() as ...' block, before card scraping:
print("Checking for 'Show More Results' button...")
show_more_button_selector = 'button[data-stid="show-more-results"]'  # Verify in browser dev tools

while page.locator(show_more_button_selector).is_visible():
    print("Found 'Show More Results' button, clicking...")
    try:
        page.locator(show_more_button_selector).click(timeout=10000)  # 10 second timeout for click
        print("Waiting for more results to load...")
        # Wait for network activity to settle or just a fixed delay
        page.wait_for_load_state('networkidle', timeout=15000)  # Wait up to 15s for network to be idle
        # Alternative fixed wait: time.sleep(4)
    except Exception as e:
        print(f"Could not click 'Show More' or timed out waiting: {e}")
        break  # Exit loop if button disappears or errors occur

print("'Show More Results' button no longer visible or process finished.")
# Now proceed to scrape *all* the cards that are currently loaded

The loop looks for the button, clicks it if visible, and waits for new content to settle using networkidle (or a fixed delay as a fallback). It keeps going until the button vanishes or something errors out. This is a reusable pattern for any site that paginates by lazy-loading.

Here's the complete script with the "Show More" logic folded in:

import time
from playwright.sync_api import sync_playwright

# Define search parameters
destination = "Rome (and vicinity), Lazio, Italy"
checkin_date = "2024-07-10"
checkout_date = "2024-07-17"
adults = 2
rooms = 1

# Construct the URL
expedia_url = f"https://www.expedia.com/Hotel-Search?destination={destination}&startDate={checkin_date}&endDate={checkout_date}&adults={adults}&rooms={rooms}&sort=RECOMMENDED"

print(f"Navigating to: {expedia_url}")

with sync_playwright() as playwright_instance:
    browser = playwright_instance.firefox.launch(
        headless=False,  # Set to True for background execution
    )
    page = browser.new_page()
    page.set_default_navigation_timeout(60000)  # 60 seconds
    page.goto(expedia_url)

    print("Page loaded, initial wait...")
    time.sleep(5)  # Wait for initial load

    # --- Handle "Show More Results" ---
    print("Checking for 'Show More Results' button...")
    # Note: Selector might change, inspect element if needed
    show_more_button_selector = 'button[data-stid="show-more-results"]'

    while page.locator(show_more_button_selector).is_visible():
        print("Found 'Show More Results' button, clicking...")
        try:
            page.locator(show_more_button_selector).click(timeout=10000)
            print("Waiting for more results to load...")
            # Wait for network activity to settle or just a fixed delay
            page.wait_for_load_state('networkidle', timeout=15000)
            # time.sleep(4)  # Alternative fixed wait
        except Exception as e:
            print(f"Could not click 'Show More' or timed out waiting: {e}")
            break

    print("'Show More Results' button no longer visible or process finished.")

    # --- Scrape all loaded hotels ---
    print("Scraping all loaded hotel cards...")
    hotel_cards = page.locator('[data-stid="lodging-card-responsive"]').all()
    print(f"Found {len(hotel_cards)} total hotel cards.")

    extracted_hotels = []
    for card in hotel_cards:
        content_section = card.locator('div.uitk-card-content-section')

        hotel_title_element = content_section.locator('h3')
        hotel_title = hotel_title_element.text_content().strip() if hotel_title_element.is_visible() else "N/A"

        rating_element = content_section.locator('span.uitk-badge-base-text')
        hotel_rating = rating_element.text_content().strip() if rating_element.is_visible() else "No Rating"

        price_element = content_section.locator('div[data-test-id="price-summary"] .uitk-text .uitk-type-500')
        if not price_element.is_visible():
            price_element = content_section.locator('div.uitk-type-500')  # Fallback

        hotel_price = price_element.text_content().strip() if price_element.is_visible() else "Price Unavailable"

        hotel_data = {
            'title': hotel_title,
            'rating': hotel_rating,
            'price': hotel_price
        }
        extracted_hotels.append(hotel_data)

    print("\n--- Final Extracted Hotel Data ---")
    # Print only the first few and the total count for brevity
    for i, hotel in enumerate(extracted_hotels):
        if i < 5:  # Print first 5
            print(hotel)
        elif i == 5:
            print("...")  # Indicate more data exists
    print(f"(Total: {len(extracted_hotels)} hotels)")
    print("---------------------------------")

    print("Closing browser.")
    browser.close()

Why Proxies Matter for Larger Jobs

Pulling one city and one date range now and then is unlikely to cause any friction — Expedia serves huge amounts of traffic every second. But the moment you want breadth (many cities, many date windows, price snapshots over time) all those requests originate from a single IP address. That concentration is what causes trouble: rate limits, CAPTCHAs, and eventually a temporarily blocked IP.

Web scraping at scale is fundamentally about distributing your requests so no single source hammers the server. A proxy server routes each request through a different IP, so your collection load spreads across many addresses instead of piling onto one. Residential and mobile proxies work well here because their addresses belong to real consumer connections, which is exactly what a travel site expects to see.

Evomi's proxies are ethically sourced and Swiss-based, with residential plans from $0.49/GB and free trials on residential, mobile, and datacenter pools so you can benchmark before committing. Rotating your IP frequently keeps request volume per address low and your job running smoothly.

Adding a Proxy to Playwright

Integrating a proxy is a one-line change to the browser launch. Grab your host, port, username, and password from the Evomi dashboard, then pass a proxy dict:

# Example using Evomi Residential Proxy (HTTP)
proxy_server = "rp.evomi.com:1000"  # Or HTTPS on 1001, SOCKS5 on 1002
proxy_username = "YOUR_EVOMI_USERNAME"
proxy_password = "YOUR_EVOMI_PASSWORD"

browser = playwright_instance.firefox.launch(
    headless=False,  # Keep False for testing, True for production
    proxy={
        'server': proxy_server,
        'username': proxy_username,
        'password': proxy_password,
    }
)
# ... rest of your script (new_page, goto, etc.)

Every request from this browser instance now travels through the specified proxy. To confirm the exit IP is what you expect, you can point the browser at a checker such as geo.evomi.com before running the full job. If you'd rather skip managing browser infrastructure entirely, Evomi's Scraping Browser is a managed, cloud-hosted headless Chromium endpoint (Playwright- and Puppeteer-compatible over wss://browser.evomi.com) with proxying built in.

Practical Tips for Reliable Collection

  • Verify selectors regularly. Expedia updates its front end often. If a run returns empty titles or prices, open dev tools and re-check the data-stid and class names.

  • Prefer explicit waits over long sleeps. wait_for_load_state('networkidle') and expect(locator).to_be_visible() are more robust than fixed time.sleep() calls.

  • Rotate IPs and pace requests. A steady, moderate request rate paired with rotating residential IPs is far more stable than bursts from one address.

  • Store data incrementally. Write results to CSV or a database inside the loop so a crash mid-run doesn't cost you everything.

  • Respect the site. Only collect publicly visible data, follow Expedia's Terms of Service, and keep your footprint reasonable.

Where This Leads

Once you have clean, structured hotel data, the applications are broad: dynamic pricing dashboards that compare rates across dates and providers, availability trend tracking for a market you operate in, or competitive research for a hotel you own. The same Playwright-plus-proxy pattern shown here transfers to plenty of other JavaScript-heavy sites — the selectors change, but the workflow of rendering, paginating, extracting defensively, and distributing requests stays the same.

Author

Sarah Whitmore

Digital Privacy & Cybersecurity Consultant

About Author

Sarah is a cybersecurity strategist with a passion for online privacy and digital security. She explores how proxies, VPNs, and encryption tools protect users from tracking, cyber threats, and data breaches. With years of experience in cybersecurity consulting, she provides practical insights into safeguarding sensitive data in an increasingly digital world.

Like this article? Share it.
You asked, we answer - Users questions:
Can I scrape Expedia with Beautiful Soup instead of Playwright?+
Do I really need proxies to scrape Expedia?+
Why do some hotels show 'No Rating' or 'Price Unavailable'?+
Is scraping Expedia legal?+
How do I check that my proxy is actually being used?+
What if the 'Show More Results' selector stops working?+

In This Article

Read More Blogs

7 min

Why Is My Proxy So Slow? Diagnosing Latency, Bandwidth & Bottlenecks

Your scraper crawls and every request takes eight seconds, but is it the proxy, the route, or the target? This guide separates latency from throughput from success rate, shows you how to measure each with Python and curl, and walks through the real causes in order of likelihood so you fix the right thing.

7 min

How Residential Proxy Networks Actually Work: Peer-Sourced IPs Explained

A residential proxy is just someone's real home IP, so whose is it, and how did the provider get permission to route your traffic through it? This explainer walks through what makes an IP residential, how providers source the pool, the ethics spectrum from opt-in SDKs to malware, and how to tell a clean network from a sketchy one.

3 min Read

Disable WebRTC to Stop Your Real IP Leaking Past Proxies

WebRTC can reveal your true IP even when you route traffic through a proxy. Here's how the leak works, how to test for it, and how to switch WebRTC off in Chrome, Firefox and Edge.

Get Started with Swiss Quality Proxies

Try for Free

View Pricing

Explore Our Proxy Blog

Tips on residential proxies, web scraping & security

Frequently Asked Questions

Read if you have any questions regarding proxies

Read about Ethics

Our Ethical Standards in the residential proxy market

Get Started with Swiss Quality Proxies

Try for Free

View Pricing

Explore Our Proxy Blog

Tips on residential proxies, web scraping & security

Frequently Asked Questions

Read if you have any questions regarding proxies

Read about Ethics

Our Ethical Standards in the residential proxy market

Get Started with Swiss Quality Proxies

Try for Free

View Pricing

Explore Our Proxy Blog

Tips on residential proxies, web scraping & security

Frequently Asked Questions

Read if you have any questions regarding proxies

Read about Ethics

Our Ethical Standards in the residential proxy market