Effortless Glassdoor Scraping with Python & Evomi Proxies

Extracting Insights: Scraping Glassdoor Job Data with Python and Evomi

Glassdoor is a well-known platform brimming with valuable information for job seekers and employers alike. It hosts company reviews, salary data, interview questions, and, importantly, job listings. Accessing this data programmatically can provide significant advantages.

But how do you tap into this wealth of data efficiently? Web scraping is the answer! This guide will walk you through gathering job listing information from Glassdoor using Python, the Playwright library, and the reliability of Evomi proxies.

Approaching Glassdoor Scraping: Techniques That Work

Navigating Glassdoor often leads to a prompt asking for company details before you can proceed. This might suggest that scraping the site is a complex challenge requiring intricate methods.

However, the reality is simpler. The core data you need is usually present in the HTML source code when the page initially loads. The popup merely obstructs the view in a standard browser. By employing standard HTML parsing techniques, you can bypass this visual block and extract the underlying information effectively.

A powerful combination for this task is Python alongside the Playwright library. Playwright excels at browser automation, allowing your script to interact with web pages much like a human user would. This mimicry, especially when paired with proxy servers to manage your digital footprint, forms a robust foundation for most Glassdoor scraping projects.

A Practical Guide: Scraping Glassdoor Job Listings with Python

Let's dive into a step-by-step tutorial on scraping Glassdoor's job listings using Python and Playwright. The principles shown here can be adapted to scrape other sections, like company reviews, as well.

Setting Up Your Environment

First things first, you'll need Python installed on your system. If it's not already set up, you can grab it from the official Python website and follow their installation instructions.

Next, install Playwright and its necessary browser components. Open your terminal or command prompt and run these commands:

Now, create a dedicated folder for your project, perhaps named evomi_glassdoor_scraper. Inside this folder, create a Python file named scrape_jobs.py. Open this file in your preferred code editor, and you're ready to start coding.

Fetching the Initial Job Search Page

The first step in our script is to use Playwright to open a browser and navigate to the Glassdoor job search page. This code snippet initiates a browser instance, sets a realistic viewport size, and opens the target URL:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False) # Set to True for background execution
    context = browser.new_context(
        viewport={"width": 1280, "height": 720} # Common resolution
    )
    page = context.new_page()
    print("Navigating to Glassdoor Jobs...")
    page.goto('https://www.glassdoor.com/Job/index.htm', wait_until='domcontentloaded')
    print("Page loaded.")

Once the page loads, the script needs to interact with the search form – filling in the desired job title and location.

The following code locates the input fields using their placeholder or label text, types in our search criteria (e.g., "Software Engineer" in the "USA"), simulates pressing Enter, and then waits for the search results page to finish loading network activity.

Feel free to change "Software Engineer" and "USA" to match the role and location you're interested in.

    print("Filling search criteria...")    # Use more robust selectors if placeholders change
    job_title_input = page.locator('input[placeholder*="job title"]')
    job_title_input.type("Software Engineer", delay=100) # Add small delay
    location_input = page.locator('input[aria-label*="Location"]')
    # Clear potential default location first
    location_input.fill("")
    location_input.type("USA", delay=100)
    print("Submitting search...")
    # Click search button or press Enter
    # Using Keyboard press might be more reliable sometimes
    location_input.press("Enter")
    print("Waiting for results...")
    page.wait_for_load_state('networkidle', timeout=60000) # Wait up to 60s
    print("Search results loaded.")

With the results displayed, we can now extract the data we need from each job listing. For this example, we'll focus on collecting:

• Job Title

• Company Name

• Company Rating (if available)

First, we locate all the individual job card elements on the page using a CSS selector:

job_listings = page.locator('li.react-job-listing').all()
print(f"Found {len(job_listings)} job listings on the page.")

Next, we'll initialize an empty list called extracted_jobs. We then loop through each located job listing element (job_card), extracting the desired pieces of information and adding them as a dictionary to our list.

We use specific locators (like CSS selectors) via Playwright's locator() method to pinpoint the title and company name. For the rating, which often includes a '★' symbol, we can try locating it directly. Since not all listings have a rating, we include a check (is_visible()) and provide a default value if it's missing.

extracted_jobs = []
for job_card in job_listings:
    try:
        title_element = job_card.locator('div[class*="job-title"]')
        title = title_element.text_content().strip() if title_element else "N/A"

        # Attempt to find rating first to help clean company name
        rating_element = job_card.locator('span[data-test="rating"]')
        rating = "Not Rated"
        if rating_element and await rating_element.is_visible():
            rating_text = await rating_element.text_content()
            # Extract only the number part if needed, e.g., '4.1 ★' -> '4.1'
            rating_match = re.search(r'(\d\.\d)', rating_text)
            rating = rating_match.group(1) if rating_match else rating_text.strip()

        company_element = job_card.locator('div[class*="job-employer"]')
        company_name = "N/A"
        if company_element:
            company_name_full = await company_element.text_content()
            # Basic cleaning: remove rating if present
            company_name = company_name_full.replace(rating, "").strip() if rating != "Not Rated" else company_name_full.strip()

        job_data = {"title": title, "company": company_name, "rating": rating}
        extracted_jobs.append(job_data)
    except Exception as e:
        print(f"Error processing one job card: {e}") # Basic error handling

# (Add printing/saving logic here)
# browser.close() # Moved closing outside loop

Finally, let's print the collected data and close the browser session:

print("\n--- Extracted Job Data ---")
for job in extracted_jobs:
    print(job)
print("--------------------------\n")
print("Closing browser.")
browser.close()

For clarity, here is the complete script incorporating these parts:

import re  # Import regex for potential cleaning
from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeoutError

def scrape_glassdoor_jobs():
    extracted_jobs = []
    with sync_playwright() as p:
        try:
            browser = p.chromium.launch(headless=False)  # Set True for production
            context = browser.new_context(
                user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36',  # Example User Agent
                viewport={"width": 1280, "height": 720}
            )
            page = context.new_page()

            print("Navigating to Glassdoor Jobs...")
            page.goto('https://www.glassdoor.com/Job/index.htm', wait_until='domcontentloaded', timeout=60000)
            print("Page loaded.")

            print("Filling search criteria...")
            job_title_input = page.locator('input[placeholder*="job title"], input[id="searchBar-jobTitle"]')
            job_title_input.type("Software Engineer", delay=150)

            location_input = page.locator('input[aria-label*="Location"], input[id="searchBar-location"]')
            location_input.fill("")  # Clear first
            location_input.type("USA", delay=150)

            print("Submitting search...")
            location_input.press("Enter")

            print("Waiting for results...")
            page.wait_for_load_state('networkidle', timeout=90000)  # Increased timeout
            print("Search results loaded.")

            # Wait for the list to be somewhat populated
            page.wait_for_selector('li.react-job-listing', timeout=30000)

            job_listings = page.locator('li.react-job-listing').all()
            print(f"Found {len(job_listings)} job listings on the page.")

            for job_card in job_listings:
                try:
                    title_element = job_card.locator('div[id^="job-title"]')  # More specific selector
                    title = title_element.text_content(timeout=5000).strip() if title_element else "N/A"

                    rating_element = job_card.locator('span[data-test="rating"]')
                    rating = "Not Rated"
                    if rating_element.is_visible():
                        rating_text = rating_element.text_content(timeout=5000)
                        rating_match = re.search(r'(\d\.\d)', rating_text)
                        rating = rating_match.group(1) if rating_match else rating_text.strip()

                    company_element = job_card.locator('div[id^="job-employer"]')
                    company_name = "N/A"
                    if company_element.is_visible():
                        company_name_full = company_element.text_content(timeout=5000)
                        # Improved cleaning
                        parts = company_name_full.split(rating) if rating != "Not Rated" else [company_name_full]
                        company_name = parts[0].strip()

                    job_data = {"title": title, "company": company_name, "rating": rating}
                    extracted_jobs.append(job_data)

                except PlaywrightTimeoutError:
                    print("Warning: Timeout locating element within a job card. Skipping.")
                except Exception as e:
                    print(f"Error processing one job card: {e}")

            print("\n--- Extracted Job Data ---")
            for job in extracted_jobs:
                print(job)
            print("--------------------------\n")

        except PlaywrightTimeoutError:
            print("Error: Timed out waiting for page load or elements.")
        except Exception as e:
            print(f"An unexpected error occurred: {e}")
        finally:
            if 'browser' in locals() and browser:
                print("Closing browser.")
                browser.close()

    return extracted_jobs

if __name__ == "__main__":
    scrape_glassdoor_jobs()

Running this script should produce output similar to this (details will vary based on current listings):

--- Extracted Job Data ---
{
  'title': 'Senior Software Engineer - Backend',
  'company': 'ExampleCorp',
  'rating': '4.2'
}
{
  'title': 'Frontend Developer (React)',
  'company': 'Another Company Inc.',
  'rating': '3.9'
}
{
  'title': 'Entry Level Software Engineer',
  'company': 'Startup Solutions',
  'rating': 'Not Rated'
}
...

Boosting Your Scraper: Integrating Evomi Proxies

While scraping a single page of job results might go unnoticed, scaling up your efforts will likely attract attention. If you plan to monitor job openings over time or gather data across many locations and roles, making frequent requests from the same IP address is a surefire way to get blocked by Glassdoor's anti-scraping systems.

This is where proxies become essential. Proxies act as intermediaries, routing your requests through different IP addresses. This masks your original IP and distributes your activity, making it much harder for Glassdoor to identify and block your scraping operations. Using proxies is crucial for reliable, large-scale data gathering.

For scraping dynamic websites like Glassdoor, Evomi's residential proxies are an excellent choice. These proxies use IP addresses associated with real home internet connections worldwide. Because they appear as genuine user traffic, they are significantly less likely to be detected and blocked compared to datacenter proxies. Furthermore, Evomi is committed to ethically sourcing its proxy pool and, being based in Switzerland, adheres to high standards of quality and reliability.

Integrating Evomi proxies into your Playwright script is straightforward. You'll need your Evomi proxy credentials (username, password) and the correct endpoint details.

For Evomi's residential proxies, the endpoint might look like rp.evomi.com with specific ports for different protocols (e.g., port 1000 for HTTP, 1001 for HTTPS). You can find your specific credentials and endpoint details within your Evomi dashboard after signing up. You might even want to start with Evomi's free trial to test the setup.

Update the p.chromium.launch() call in your script to include the proxy configuration:

# Replace with your actual Evomi credentials and desired port
    evomi_proxy_server = 'rp.evomi.com:1000' # Example using HTTP port
    evomi_username = 'YOUR_EVOMI_USERNAME'
    evomi_password = 'YOUR_EVOMI_PASSWORD'
    browser = p.chromium.launch(
        headless=True, # Recommended for proxy usage in production
        proxy={
            'server': evomi_proxy_server,
            'username': evomi_username,
            'password': evomi_password,
        }
    )
    # ... rest of the script

With this change, all requests made by your Playwright browser session will now be routed through the specified Evomi residential proxy server, significantly enhancing the reliability and stealth of your Glassdoor scraper.

Scraping Glassdoor Without Writing Code

If Python scripting isn't your cup of tea, several no-code web scraping tools can help you extract data from Glassdoor. While they might offer less customization than a bespoke script, they provide a user-friendly alternative for those new to web scraping.

Octoparse

Octoparse offers a visual interface where users can point and click on website elements to define the data they wish to extract. It handles common scraping challenges like pagination and data formatting, exporting results to formats like CSV or Excel.

Apify

Apify is a platform providing tools for web scraping and automation. It offers pre-built "Actors" for common tasks and sites, potentially including Glassdoor, alongside infrastructure for running and scaling scraping jobs.

Automatio

Automatio focuses on a no-code approach, enabling users to build web scraping "bots" through a visual interface. It's designed for users without programming experience to automate data extraction tasks.

Navigating the Legal Landscape of Glassdoor Scraping

Generally, accessing publicly available information online via automated means (scraping) is considered legal in many jurisdictions. Using proxies to manage your IP address is also perfectly legal.

However, it's crucial to be aware of Glassdoor's Terms of Service. Like many websites, their terms explicitly prohibit scraping without written permission. While scraping public data without logging in primarily risks an IP block (which proxies help mitigate), creating an account and then scraping *could* potentially lead to account suspension if detected, as you've agreed to their terms.

Therefore, the recommended approach is to scrape only publicly accessible data, avoid logging into any Glassdoor account during scraping, and always use a reliable proxy service like Evomi.

Are There Data Scraping Limits on Glassdoor?

Glassdoor implements measures to prevent excessive automated traffic. Without precautions, you'll likely encounter IP blocks or CAPTCHAs if you scrape aggressively.

However, by employing a robust proxy network, such as Evomi's residential proxies, you can effectively bypass these limitations. Rotating IPs makes your traffic appear organic, allowing you to access and scrape company reviews, salary data, job listings, and other valuable information without hitting prohibitive walls. Responsible scraping practices, combined with quality proxies, enable extensive data collection for market analysis, competitor research, or refining recruitment efforts.