Essential User-Agent Strings for Web Scraping

The Curious Case of the User-Agent: Your Stealth Cloak for Web Scraping

Ever felt like your web scraping scripts hit an invisible wall? You write the code, launch it, and... blocked. It's frustratingly common. Websites are savvy; they can often sniff out automated scripts that don't behave like typical browsers, shutting the door before you get the data you need.

But fear not, intrepid data gatherer! There's a simple yet effective trick up our sleeves: tweaking the User-Agent header. Stick around, and we'll dive into what User-Agents are, why they matter for scraping, explore some common ones, and show you how to modify your scripts to blend in like a digital chameleon.

Decoding the User-Agent: What Is It and Why Should Scrapers Care?

Whenever your browser (or a script using an HTTP client like Python's excellent requests library) talks to a web server, it sends a little package of information called HTTP headers. These headers contain metadata about the request itself.

Tucked inside these headers is the User-Agent string. Think of it as the request's calling card, telling the server which software (browser, script, app) is making the request.

Want to see what your script is sending? Try this Python snippet using the requests library. It sends a basic request and then prints the headers it sent:

import requests

target_url = "https://httpbin.org/headers" # A handy site for checking headers
response = requests.get(target_url)
print(response.json()['headers'])

The output might look something like this (details may vary):

{
  "Accept": "*/*",
  "Accept-Encoding": "gzip, deflate",
  "Host": "httpbin.org",
  "User-Agent": "python-requests/2.28.1",
  "X-Amzn-Trace-Id": "Root=1-..."
}

See that User-Agent? By default, requests announces itself quite clearly. This is often a dead giveaway for anti-scraping systems.

It's also worth noting that headers can reveal other details, like whether you're using a proxy. You can get a glimpse of what information your browser sends using tools like Evomi's Browser Fingerprint Checker.

Real web browsers send much more detailed User-Agent strings. They typically include the browser name and version, the operating system, and rendering engine details. This helps servers tailor the content (e.g., serving a mobile version of a site).

For instance, here's a plausible User-Agent for a recent version of Firefox on a Mac:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/117.0

And contrast that with one from Safari on an iPhone:

Mozilla/5.0 (iPhone; CPU iPhone OS 16_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Mobile/15E148 Safari/604.1

The key takeaway? By swapping the default script User-Agent with one resembling a common browser, you make your scraping requests look much more like legitimate user traffic, significantly reducing the chances of being blocked outright.

Mastering the Switch: How to Change Your User-Agent

Thankfully, most HTTP client libraries used for web scraping make it straightforward to customize the User-Agent string, effectively disguising your script as a standard browser.

Let's stick with the popular Python requests library to see how it's done.

Imagine you have this basic script:

import requests

target_url = "https://httpbin.org/user-agent"  # This endpoint echoes the User-Agent
response = requests.get(target_url)
print(f"Default User-Agent: {response.json()['user-agent']}")

To send a custom User-Agent, you first create a Python dictionary containing your desired header. You only need the User-Agent key for this purpose:

custom_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

Then, you pass this dictionary to the headers parameter in your requests.get() (or post(), etc.) call:

import requests

custom_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}
target_url = "https://httpbin.org/user-agent"

response = requests.get(target_url, headers=custom_headers)

print(f"Custom User-Agent Sent: {response.json()['user-agent']}")

Run this, and you'll see the output confirming that your specified User-Agent was sent instead of the default python-requests one. Voila! Your script now looks like it's Chrome on Windows.

Custom User-Agent Sent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36

Choosing Your Disguise: Common User-Agents for Web Scraping

When selecting a User-Agent for your scraper, the path of least resistance is usually best: pick one that's widely used. Using a common browser's User-Agent helps your requests blend into the normal traffic flow, making them less likely to attract unwanted attention.

As of late 2023 / early 2024, Chrome running on Windows remains exceedingly common. A typical string looks like this:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

You might notice mentions of "Mozilla", "AppleWebKit", "Gecko", and "Safari" even in a Chrome string. This is mostly for historical compatibility reasons, a quirky legacy of the browser wars detailed elsewhere if you're curious.

Here are some other popular User-Agent strings you could consider, reflecting common browser/OS combinations:

1. Chrome 119 on macOS:

   Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

2. Firefox 119 on Windows:

   Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/119.0

3. Chrome 118 on Windows:

   Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36

4. Edge 119 on Windows:

   Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0

5. Chrome 119 on Linux:

   Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

6. Firefox 119 on macOS:

   Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/119.0

Crucially, browser versions change frequently. What's common today might be outdated tomorrow. It's good practice to periodically verify popular User-Agent strings and update your scrapers accordingly.

A quick pro-tip: Simply search "What is my user agent?" in your favorite search engine. The result will show the exact string your current browser is sending – a perfectly valid and up-to-date option for your script.

Finding More User-Agent Options

If you need a broader selection, especially historical or less common ones, resources like the list maintained at Will Shouse's Tech Blog can be very helpful. It's updated based on visitor data.

Going Mobile: User-Agents for Mobile Scraping

Just like their desktop counterparts, mobile browsers have their own distinct User-Agent strings indicating the mobile OS (iOS, Android) and browser.

Here are a couple of examples for mobile scraping tasks:

1. Chrome on a recent Android device:

   Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Mobile Safari/537.36

2. Safari on a recent iPhone:

   Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1

Using these can be essential if you need to scrape the mobile version of a website.

User-Agents and Anti-Scraping: A Piece of the Puzzle

Employing a realistic browser User-Agent is a fundamental step in bypassing simple anti-scraping checks that block requests identifying themselves as scripts. Some sites, for example, might block the default headers from common HTTP libraries.

However, let's be clear: changing the User-Agent isn't a silver bullet. Sophisticated websites use a multitude of techniques to detect scraping. They analyze request frequency, navigation patterns, JavaScript execution capabilities, IP address reputation, and more. Tools like Puppeteer or Selenium, which control real browsers, can still be detected if not configured carefully.

For any serious or large-scale web scraping project, managing your IP address footprint is paramount. This is where rotating proxies become indispensable. Proxies act as intermediaries, masking your script's true IP address. By rotating through a pool of different proxy IPs for consecutive requests, you make your traffic appear as if it's originating from many different, unrelated users, drastically reducing the risk of an IP ban.

If you need a robust and reliable proxy solution, consider Evomi's Residential Proxies. Sourced ethically from real user devices across the globe, our vast network provides the diversity needed to scale your scraping operations effectively. We're based in Switzerland, focusing on quality and support, and even offer a free trial so you can experience the difference.