Headless Selenium in Python for Proxies & Web Testing

Getting Started with Headless Selenium in Python

Selenium is a powerful suite of tools designed for automating web browsers. Available for various programming languages, including Python, it's a cornerstone for developers and QA engineers. Essentially, Selenium allows you to script browser actions, even without displaying the graphical user interface (GUI).

Running browsers in "headless" mode, meaning without the visual interface, is particularly popular for automated testing and web scraping. Why? Because headless browsers typically run faster and consume fewer system resources compared to their visible counterparts. Using Selenium headless with Python also opens up more possibilities for fine-tuning and programmatic control over browser behavior.

Setting Up Your Python Environment for Selenium

Before diving into headless browser automation with Selenium, you'll need a couple of things: Python itself and an Integrated Development Environment (IDE). You can grab the latest Python version directly from the official Python website.

Numerous IDEs cater to Python development, each with its own set of features. For getting started with browser automation in Python, PyCharm Community Edition is an excellent, free choice. It offers robust features suitable for most Python projects.

Once Python and PyCharm are installed, create a new project within PyCharm.

Next, you need to install the Selenium library for your project. Open the Terminal panel within PyCharm (usually found at the bottom) and execute the following command:

PyCharm will handle the download and installation of the Selenium package. Note that this installation is typically specific to the current project environment. While system-wide installation is possible, project-specific installations are common practice and often preferred for managing dependencies.

Lastly, you'll need a WebDriver. Selenium interacts with browsers via specific WebDriver executables. Modern Selenium versions often manage this automatically, but sometimes you might need to download one manually, especially if you encounter errors. We'll be using Chrome in our examples.

Running Selenium with a Visible Browser (Headful Mode)

Let's start by running Selenium in its standard "headful" mode (with the browser window visible) to confirm our setup is working correctly.

First, import the necessary components from the Selenium library:

from selenium import webdriver

Now, let's define a simple Python function to open a browser and navigate to a specific URL:

def check_website_loading(target_url: str):
    # Initialize the Chrome WebDriver
    driver = webdriver.Chrome()
    # Navigate to the specified URL
    driver.get(target_url)
    print(f"Successfully loaded: {driver.title}")
    # Close the browser window and end the WebDriver session
    driver.quit()

# Example usage: Test loading Evomi's IP checker tool
check_website_loading('https://geo.evomi.com/')

Let's break down this code:

• The first line defines our function check_website_loading, which accepts one argument: target_url. Adding : str is a type hint, suggesting the function expects a string input, which helps catch potential errors early.

• driver = webdriver.Chrome() creates an instance of the Chrome WebDriver. This driver object is our primary tool for interacting with the browser.

• driver.get(target_url) instructs the browser controlled by our driver to navigate to the URL passed into the function.

• print(f"Successfully loaded: {driver.title}") is a simple action to confirm the page loaded by printing its title to the console.

• driver.quit() closes the browser window and gracefully shuts down the WebDriver process.

• Finally, we call the function check_website_loading, passing the URL of Evomi's free IP geolocation checker as the target.

Execute this script (often via a 'Run' button or command in your IDE). If everything is configured correctly, a new Chrome browser window should pop up, navigate to https://geo.evomi.com/, print the page title to your console, and then automatically close.

If you encounter an error message about a missing WebDriver, you might need to download it manually. For Chrome, you can usually find the correct version on the Chrome for Testing availability dashboard. Ensure you download the `chromedriver` compatible with your operating system and Chrome browser version.

Switching to Headless Mode with Selenium

While seeing the browser pop up is great for initial testing, most automation tasks, like large-scale testing or web scraping, benefit from running without the visual overhead. Let's modify our script to run Selenium in headless mode.

We need to import another component to manage browser options:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

Now, we'll adjust our function to configure and use these options:

def check_website_headless(target_url: str):
    # Initialize Chrome options
    options = ChromeOptions()
    # Add the argument to run in headless mode
    options.add_argument("--headless=new")
    # Initialize the WebDriver with the specified options
    driver = webdriver.Chrome(options=options)

    try:
        driver.get(target_url)
        print(f"Headless check successful for: {target_url}")
        print(f"Page title: {driver.title}")
    except Exception as e:
        print(f"An error occurred: {e}")
    finally:
        # Ensure the browser closes even if errors occur
        driver.quit()

# Example usage: Test loading the Evomi homepage headless
check_website_headless('https://evomi.com/')

Here's what changed:

• options = ChromeOptions() creates an object to hold our configuration settings for the Chrome browser.

• options.add_argument("--headless=new") is the key change. This method adds a command-line argument that tells the Chrome WebDriver to start in the modern headless mode.

• driver = webdriver.Chrome(options=options) modifies how we initialize the WebDriver, passing our configured options object to it.

• We also added a basic try...finally block to ensure driver.quit() is called even if an error occurs during page load.

Run this updated script. This time, you won't see a browser window appear. The script will execute in the background, connect to the website, print the success message and page title to your console, and then terminate. This demonstrates the core principle of headless browsing – performing browser actions without the visual display.

Processing Multiple URLs Sequentially

Often, you'll need to automate actions across a list of URLs rather than just one. We can adapt our script using a loop.

Let's modify the function to accept a list of URLs:

def check_multiple_websites(url_list):
    options = ChromeOptions()
    options.add_argument("--headless=new") # Keep it headless for efficiency
    driver = webdriver.Chrome(options=options)
    print(f"Starting checks for {len(url_list)} websites...")

    for url in url_list:
        try:
            driver.get(url)
            print(f"  [OK] Loaded: {driver.title} ({url})")
        except Exception as e:
            print(f"  [ERROR] Loading {url}: {e}")

    print("Finished all checks.")
    driver.quit()

The main change here is the for url in url_list: loop. This iterates through each item (expected to be a URL string) in the input url_list. Inside the loop, driver.get(url) navigates to the current URL. Note that we removed the type hint : str from the function definition because it now expects a sequence (like a list) rather than a single string.

Now, create a list of target websites and call the function:

target_sites = [
    'https://evomi.com/',
    'https://evomi.com/pricing',
    'https://check.evomi.com/'
]
check_multiple_websites(target_sites)

This code defines a Python list named target_sites containing a few URLs. Then, it passes this list to our check_multiple_websites function. The script will sequentially visit each URL in the list using the headless browser.

Here's the combined code block for clarity:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

def check_multiple_websites(url_list):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    driver = webdriver.Chrome(options=options)

    print(f"Starting checks for {len(url_list)} websites...")

    for url in url_list:
        try:
            driver.get(url)
            print(f"  [OK] Loaded: {driver.title} ({url})")
        except Exception as e:
            print(f"  [ERROR] Loading {url}: {e}")

    print("Finished all checks.")
    driver.quit()

target_sites = [
    'https://evomi.com/',
    'https://evomi.com/pricing', # Example link to pricing
    'https://check.evomi.com/'
]

check_multiple_websites(target_sites)

Tip: While developing or debugging, you might want to temporarily disable headless mode to visually confirm the script's actions. You can do this by commenting out the options.add_argument("--headless=new") line (add a # at the beginning of the line).

Conclusion: Headless Selenium and Beyond

As you've seen, setting up and running a headless browser using Selenium in Python requires just a few lines of code. This opens the door to powerful automation for web testing, data gathering, and more, all while being efficient with system resources.

While simply visiting pages is a start, Selenium offers a vast API for interacting with web elements, filling forms, clicking buttons, and executing JavaScript. For more demanding tasks like large-scale web scraping or testing applications across different geographic locations, combining Selenium automation with robust proxy infrastructure is often necessary. Using reliable residential, mobile, or ISP proxies, like those offered by Evomi, can help manage connections and access geo-specific content effectively, ensuring your automation tasks run smoothly and avoid detection. Exploring options like Evomi's ethically sourced proxies, competitive pricing (starting from $0.49/GB for residential), and Swiss-based reliability can be a solid next step for scaling your projects.