Scraping Amazon Data at Scale With Proxies
David Foster
Scraping Techniques
You've written some clean Python that pulls data from a product page. It works flawlessly for one URL, maybe ten. Then the brief changes: now you need pricing and specs from thousands of products, refreshed on a schedule. That jump from a handful of pages to a serious volume is where most scraping projects stall.
The recurring question is always the same: how do you keep collecting public product data reliably without hammering a single IP into the ground? Sending a flood of requests from one address is bad manners and unreliable, quite apart from any terms-of-service considerations. This guide walks through a responsible, proxy-based approach to gathering public Amazon data at scale for legitimate purposes like price monitoring, market research, and internal QA. Before you start, read the platform's terms and robots directives and stick to data that's genuinely public.
Why proxies matter for large-scale data collection
A proxy server is an intermediary. It sits between your machine and the target site. Your request goes to the proxy first; the proxy forwards it to the site using its own IP, receives the response, and passes it back to you.
For scaling responsibly, that indirection does two useful things. First, it lets you distribute a large workload across many IPs and geographies, so you're not concentrating all your traffic on one connection. Second, it lets you fetch localized data — Amazon shows different prices and availability by region, so a proxy in the right country returns the version a real shopper there would see. Both are legitimate reasons to route traffic through proxies, and both help you build a dataset that's accurate and representative.
Evomi's residential proxies are a good fit here: the pool rotates automatically, they're ethically sourced, and they're geographically diverse. If you'd rather offload the browser and rendering entirely, the managed Scraping Browser handles headless Chromium in the cloud, but for straightforward HTML product pages the requests approach below is lean and fast.
What you'll need
We'll use two well-known Python libraries. If they aren't installed yet, pip handles it in one line:
Then import what we need. The csv module comes with the standard library and we'll use it to feed product IDs into the scraper.
# Import necessary libraries
import requests
from bs4 import BeautifulSoup
import csv # We'll use this later to handle product IDsIf Beautiful Soup and CSS selectors are new to you, our Beautiful Soup proxy guide covers the fundamentals in more depth.
Use a Session — it's faster and cleaner
A step beginners often skip: use a requests.Session object instead of calling requests.get directly each time. A session persists parameters like cookies and headers across requests, and — more importantly for throughput — it reuses the underlying TCP connection instead of opening a new one for every fetch. Over thousands of requests, that connection reuse adds up. The requests documentation explains the behaviour in detail. We'll also attach our proxy settings to the session, so every request routes through the proxy automatically.
Wiring in Evomi proxies
Evomi hands you credentials in the format username:password@endpoint:port. For rotating residential proxies — which are ideal for large collection jobs since the IP changes across requests — the setup looks like this:
# Basic structure with session setup
import requests
from bs4 import BeautifulSoup
# Placeholder functions we'll define later
def load_asins_from_csv(filepath): pass
def fetch_product_page(session, asin): pass
def extract_product_info(html_content, asin): pass
def run_scraper():
# Initialize the session
session = requests.Session()
# Set standard request headers
session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Upgrade-Insecure-Requests': '1'
})
# Configure Evomi proxies (replace with your actual credentials)
# Using residential proxies (rp.evomi.com) via HTTP (port 1000) as an example
proxy_user = 'YOUR_USERNAME'
proxy_pass = 'YOUR_PASSWORD'
proxy_host = 'rp.evomi.com'
proxy_port = '1000' # HTTP port for residential
proxies = {
'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}',
'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}' # Often the same for HTTP/S setup
}
session.proxies.update(proxies)
print("Session configured with headers and proxies:")
# print(session.headers) # Uncomment to verify headers
# print(session.proxies) # Uncomment to verify proxies
# --- Add scraping logic here ---
# Entry point for the script
if __name__ == "__main__":
run_scraper()A note on those headers. Setting an honest, current User-Agent and standard Accept headers is simply good HTTP citizenship — it tells the server what your client actually is and what it can handle, which helps you receive correctly formatted responses. Don't treat headers as a disguise; treat them as accurate metadata about your client. The residential proxy rotates IPs across requests, which is what lets you spread the load geographically rather than concentrating it on one address.
Working with ASINs (Amazon's product IDs)
Every product on Amazon has a unique identifier called an ASIN (Amazon Standard Identification Number). You'll find it in the product URL or the "Product details" section on the page. For a batch job, you'll typically keep your target ASINs in a CSV, one per row. Here's a small loader that reads them from the first column:
import csv
# Function to load ASINs from a CSV file
def load_asins_from_csv(filepath):
asin_list = []
try:
with open(filepath, mode='r', newline='', encoding='utf-8') as csvfile:
reader = csv.reader(csvfile)
# Skip header row if present (optional)
# next(reader, None)
for row in reader:
if row: # Ensure row is not empty
asin_list.append(row[0].strip())
except FileNotFoundError:
print(f"Error: File not found at {filepath}")
except Exception as e:
print(f"An error occurred reading the CSV: {e}")
return asin_listNext, a function that fetches a single product page through our configured session. We pass in the session and ASIN, build the URL, and return both the HTML and the ASIN so we always know which response belongs to which product.
# Function to fetch the product page HTML
def fetch_product_page(session, asin):
# Construct the URL for the Amazon product page (using amazon.com)
product_url = f"https://www.amazon.com/dp/{asin}"
try:
response = session.get(product_url, timeout=15) # Added timeout
response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)
return response.text, asin # Return HTML content and ASIN
except requests.exceptions.RequestException as e:
print(f"Request failed for ASIN {asin}: {e}")
return None, asinParsing the fields you actually want
With HTML in hand, we parse out the fields we care about — here, title and price. Beautiful Soup with the lxml parser plus CSS selectors is a concise way to target elements. Amazon's markup varies between layouts, so we try several price selectors in order and fall back gracefully.
# Function to parse HTML and extract product data
def extract_product_info(html_content, asin):
if not html_content:
return None
soup = BeautifulSoup(html_content, 'lxml') # Using lxml parser
product_data = {'asin': asin, 'title': None, 'price': None}
try:
# Extract product title (selector might need adjustment based on page structure)
title_element = soup.select_one('span#productTitle')
if title_element:
product_data['title'] = title_element.get_text(strip=True)
# Extract price (this selector often works, but can vary)
# It looks for common price patterns like elements with class 'a-offscreen'
# or specific price block elements.
price_element = soup.select_one('span.a-price > span.a-offscreen')
if not price_element: # Try alternative common selector
price_element = soup.select_one('span#priceblock_ourprice')
if not price_element: # Another alternative
price_element = soup.select_one('span#price_inside_buybox')
if price_element:
product_data['price'] = price_element.get_text(strip=True)
except Exception as e:
print(f"Error parsing data for ASIN {asin}: {e}")
# Basic validation: only return data if title and price were found
if product_data['title'] and product_data['price']:
return product_data
else:
print(f"Could not extract complete data for ASIN {asin}")
return None # Return None if essential data is missingWrapping the parsing in a try...except block matters here. Page structures change, some products lack certain fields, and you don't want one malformed page to crash a run of ten thousand.
Bringing it together and testing small
Now we update run_scraper to load the ASINs, loop through them, fetch, parse, and collect results. One rule that saves a lot of grief: test on a small handful of ASINs first. Don't point a fresh, untested scraper at thousands of products — validate your selectors and logic on five, then scale.
# Updated main function to run the scraper
def run_scraper():
# ... (Session and proxy setup code from earlier) ...
session = requests.Session()
# ... (Add headers and proxy config here) ...
session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Upgrade-Insecure-Requests': '1'
})
proxy_user = 'YOUR_USERNAME'
proxy_pass = 'YOUR_PASSWORD'
proxy_host = 'rp.evomi.com'
proxy_port = '1000'
proxies = {
'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}',
'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}'
}
session.proxies.update(proxies)
# --- Scraping Logic ---
asin_file = 'asins_to_scrape.csv' # Name of your CSV file
asins_to_process = load_asins_from_csv(asin_file)
if not asins_to_process:
print("No ASINs loaded. Exiting.")
return
print(f"Loaded {len(asins_to_process)} ASINs. Starting scraping...")
results = []
for asin in asins_to_process:
print(f"Processing ASIN: {asin}")
html_content, fetched_asin = fetch_product_page(session, asin)
if html_content:
product_info = extract_product_info(html_content, fetched_asin)
if product_info:
print(f"Successfully extracted: {product_info}")
results.append(product_info)
else:
print(f"Failed to extract data for ASIN: {asin}")
else:
print(f"Failed to fetch page for ASIN: {asin}")
# Optional: Add a small delay between requests to be polite
# import time
# time.sleep(1) # Sleep for 1 second
print("\nScraping complete.")
print(f"Successfully extracted data for {len(results)} products.")
# Here you would typically save 'results' to a file (CSV, JSON, database, etc.)
# print(results)
# Entry point
if __name__ == "__main__":
run_scraper()Notice the commented-out time.sleep(1). Uncomment it in production. Adding a deliberate pause between requests keeps your crawl rate reasonable and reduces load on the target — it's both courteous and more sustainable than firing requests as fast as the network allows.
To test, create asins_to_scrape.csv in the same directory as the script and add a few ASINs, one per line (for example B081FGTPB7, B07VGRJDFY).
The complete script
Here's everything assembled, with extra price-selector fallbacks and finer-grained error handling for timeouts and HTTP errors. It's a solid starting point for collecting public Amazon product data through Evomi proxies.
import requests
from bs4 import BeautifulSoup
import csv
import time # Optional: for adding delays
# Function to load ASINs from a CSV file
def load_asins_from_csv(filepath):
asin_list = []
try:
with open(filepath, mode='r', newline='', encoding='utf-8') as csvfile:
reader = csv.reader(csvfile)
for row in reader:
if row: # Ensure row is not empty
asin_list.append(row[0].strip())
except FileNotFoundError:
print(f"Error: File not found at {filepath}")
except Exception as e:
print(f"An error occurred reading the CSV: {e}")
return asin_list
# Function to fetch the product page HTML
def fetch_product_page(session, asin):
product_url = f"https://www.amazon.com/dp/{asin}"
try:
response = session.get(product_url, timeout=15)
response.raise_for_status() # Check for HTTP errors
print(f"Request successful for {asin} (Status: {response.status_code})")
return response.text, asin
except requests.exceptions.Timeout:
print(f"Request timed out for ASIN {asin}")
return None, asin
except requests.exceptions.HTTPError as e:
print(f"HTTP error for ASIN {asin}: {e.response.status_code}")
return None, asin
except requests.exceptions.RequestException as e:
print(f"Request failed for ASIN {asin}: {e}")
return None, asin
# Function to parse HTML and extract product data
def extract_product_info(html_content, asin):
if not html_content:
return None
soup = BeautifulSoup(html_content, 'lxml')
product_data = {'asin': asin, 'title': None, 'price': None}
try:
title_element = soup.select_one('span#productTitle')
if title_element:
product_data['title'] = title_element.get_text(strip=True)
# Try common price selectors sequentially
price_element = soup.select_one('span.a-price > span.a-offscreen')
if not price_element:
price_element = soup.select_one('span#priceblock_ourprice') # Older layout?
if not price_element:
price_element = soup.select_one('span#price_inside_buybox') # Inside buy box?
# Add more selectors here if needed based on page variations
if price_element:
product_data['price'] = price_element.get_text(strip=True)
else:
# If no price found, try getting text from a broader price container
price_container = soup.select_one('div#corePrice_feature_div span.a-price-whole')
if price_container:
price_fraction = soup.select_one('div#corePrice_feature_div span.a-price-fraction')
currency_symbol = soup.select_one('div#corePrice_feature_div span.a-price-symbol')
whole = price_container.get_text(strip=True)
fraction = price_fraction.get_text(strip=True) if price_fraction else '00'
symbol = currency_symbol.get_text(strip=True) if currency_symbol else '$' # Default symbol
product_data['price'] = f"{symbol}{whole}.{fraction}"
except Exception as e:
print(f"Error parsing data for ASIN {asin}: {e}")
if product_data['title'] and product_data['price']:
return product_data
else:
missing = []
if not product_data['title']: missing.append("title")
if not product_data['price']: missing.append("price")
print(f"Could not extract ({', '.join(missing)}) for ASIN {asin}")
return None
# Main execution function
def run_scraper():
# --- Evomi Proxy Configuration ---
# Replace with your actual Evomi credentials and desired proxy type/port
proxy_user = 'YOUR_USERNAME'
proxy_pass = 'YOUR_PASSWORD'
proxy_host = 'rp.evomi.com' # Example: Residential endpoint
proxy_port = '1000' # Example: HTTP port for residential
proxies = {
'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}',
'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}'
}
# --- Session Setup ---
session = requests.Session()
session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Upgrade-Insecure-Requests': '1',
'Referer': 'https://www.google.com/' # Add a referer
})
session.proxies.update(proxies)
print("Session configured. Proxies enabled.")
# --- Scraping Logic ---
asin_file = 'asins_to_scrape.csv' # Your input file
asins_to_process = load_asins_from_csv(asin_file)
if not asins_to_process:
print("No ASINs loaded or file not found. Exiting.")
return
print(f"Loaded {len(asins_to_process)} ASINs. Starting scraping...")
results = []
processed_count = 0
for asin in asins_to_process:
processed_count += 1
print(f"\n[{processed_count}/{len(asins_to_process)}] Processing ASIN: {asin}")
html_content, fetched_asin = fetch_product_page(session, asin)
if html_content:
product_info = extract_product_info(html_content, fetched_asin)
if product_info:
print(f"--> Success: Extracted {product_info['title']} - {product_info['price']}")
results.append(product_info)
# Be polite: pause between requests
time.sleep(1)
print("\nScraping complete.")
print(f"Successfully extracted data for {len(results)} products.")
# Save 'results' to CSV, JSON, or a database here.
# Entry point
if __name__ == "__main__":
run_scraper()Scaling responsibly
A few habits keep a large collection job healthy and above board:
Respect the source. Check the site's terms and robots directives, limit yourself to public data, and keep request rates moderate. The polite
time.sleeppause is there for a reason.Handle failures gracefully. Retries with backoff, timeouts, and clear logging mean one bad page won't sink a run of thousands.
Pick the right proxy type. Residential proxies suit geographically diverse consumer-facing pages; datacenter proxies are cheaper and faster where locality matters less. You can test IP geolocation at geo.evomi.com and check your setup at proxy-tester.evomi.com before a big run.
Store your output. Push
resultsinto a CSV, JSON file, or database as you go, rather than holding everything in memory.
If your target is reviews or ratings rather than pricing, the same principles apply — our guide to scraping reviews safely goes deeper on the ethics and mechanics of that specific use case.
With a session for connection reuse, ethically sourced rotating proxies for geographic reach and load distribution, and defensive parsing, you have a foundation that scales from ten products to tens of thousands — cleanly and responsibly.
You've written some clean Python that pulls data from a product page. It works flawlessly for one URL, maybe ten. Then the brief changes: now you need pricing and specs from thousands of products, refreshed on a schedule. That jump from a handful of pages to a serious volume is where most scraping projects stall.
The recurring question is always the same: how do you keep collecting public product data reliably without hammering a single IP into the ground? Sending a flood of requests from one address is bad manners and unreliable, quite apart from any terms-of-service considerations. This guide walks through a responsible, proxy-based approach to gathering public Amazon data at scale for legitimate purposes like price monitoring, market research, and internal QA. Before you start, read the platform's terms and robots directives and stick to data that's genuinely public.
Why proxies matter for large-scale data collection
A proxy server is an intermediary. It sits between your machine and the target site. Your request goes to the proxy first; the proxy forwards it to the site using its own IP, receives the response, and passes it back to you.
For scaling responsibly, that indirection does two useful things. First, it lets you distribute a large workload across many IPs and geographies, so you're not concentrating all your traffic on one connection. Second, it lets you fetch localized data — Amazon shows different prices and availability by region, so a proxy in the right country returns the version a real shopper there would see. Both are legitimate reasons to route traffic through proxies, and both help you build a dataset that's accurate and representative.
Evomi's residential proxies are a good fit here: the pool rotates automatically, they're ethically sourced, and they're geographically diverse. If you'd rather offload the browser and rendering entirely, the managed Scraping Browser handles headless Chromium in the cloud, but for straightforward HTML product pages the requests approach below is lean and fast.
What you'll need
We'll use two well-known Python libraries. If they aren't installed yet, pip handles it in one line:
Then import what we need. The csv module comes with the standard library and we'll use it to feed product IDs into the scraper.
# Import necessary libraries
import requests
from bs4 import BeautifulSoup
import csv # We'll use this later to handle product IDsIf Beautiful Soup and CSS selectors are new to you, our Beautiful Soup proxy guide covers the fundamentals in more depth.
Use a Session — it's faster and cleaner
A step beginners often skip: use a requests.Session object instead of calling requests.get directly each time. A session persists parameters like cookies and headers across requests, and — more importantly for throughput — it reuses the underlying TCP connection instead of opening a new one for every fetch. Over thousands of requests, that connection reuse adds up. The requests documentation explains the behaviour in detail. We'll also attach our proxy settings to the session, so every request routes through the proxy automatically.
Wiring in Evomi proxies
Evomi hands you credentials in the format username:password@endpoint:port. For rotating residential proxies — which are ideal for large collection jobs since the IP changes across requests — the setup looks like this:
# Basic structure with session setup
import requests
from bs4 import BeautifulSoup
# Placeholder functions we'll define later
def load_asins_from_csv(filepath): pass
def fetch_product_page(session, asin): pass
def extract_product_info(html_content, asin): pass
def run_scraper():
# Initialize the session
session = requests.Session()
# Set standard request headers
session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Upgrade-Insecure-Requests': '1'
})
# Configure Evomi proxies (replace with your actual credentials)
# Using residential proxies (rp.evomi.com) via HTTP (port 1000) as an example
proxy_user = 'YOUR_USERNAME'
proxy_pass = 'YOUR_PASSWORD'
proxy_host = 'rp.evomi.com'
proxy_port = '1000' # HTTP port for residential
proxies = {
'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}',
'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}' # Often the same for HTTP/S setup
}
session.proxies.update(proxies)
print("Session configured with headers and proxies:")
# print(session.headers) # Uncomment to verify headers
# print(session.proxies) # Uncomment to verify proxies
# --- Add scraping logic here ---
# Entry point for the script
if __name__ == "__main__":
run_scraper()A note on those headers. Setting an honest, current User-Agent and standard Accept headers is simply good HTTP citizenship — it tells the server what your client actually is and what it can handle, which helps you receive correctly formatted responses. Don't treat headers as a disguise; treat them as accurate metadata about your client. The residential proxy rotates IPs across requests, which is what lets you spread the load geographically rather than concentrating it on one address.
Working with ASINs (Amazon's product IDs)
Every product on Amazon has a unique identifier called an ASIN (Amazon Standard Identification Number). You'll find it in the product URL or the "Product details" section on the page. For a batch job, you'll typically keep your target ASINs in a CSV, one per row. Here's a small loader that reads them from the first column:
import csv
# Function to load ASINs from a CSV file
def load_asins_from_csv(filepath):
asin_list = []
try:
with open(filepath, mode='r', newline='', encoding='utf-8') as csvfile:
reader = csv.reader(csvfile)
# Skip header row if present (optional)
# next(reader, None)
for row in reader:
if row: # Ensure row is not empty
asin_list.append(row[0].strip())
except FileNotFoundError:
print(f"Error: File not found at {filepath}")
except Exception as e:
print(f"An error occurred reading the CSV: {e}")
return asin_listNext, a function that fetches a single product page through our configured session. We pass in the session and ASIN, build the URL, and return both the HTML and the ASIN so we always know which response belongs to which product.
# Function to fetch the product page HTML
def fetch_product_page(session, asin):
# Construct the URL for the Amazon product page (using amazon.com)
product_url = f"https://www.amazon.com/dp/{asin}"
try:
response = session.get(product_url, timeout=15) # Added timeout
response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)
return response.text, asin # Return HTML content and ASIN
except requests.exceptions.RequestException as e:
print(f"Request failed for ASIN {asin}: {e}")
return None, asinParsing the fields you actually want
With HTML in hand, we parse out the fields we care about — here, title and price. Beautiful Soup with the lxml parser plus CSS selectors is a concise way to target elements. Amazon's markup varies between layouts, so we try several price selectors in order and fall back gracefully.
# Function to parse HTML and extract product data
def extract_product_info(html_content, asin):
if not html_content:
return None
soup = BeautifulSoup(html_content, 'lxml') # Using lxml parser
product_data = {'asin': asin, 'title': None, 'price': None}
try:
# Extract product title (selector might need adjustment based on page structure)
title_element = soup.select_one('span#productTitle')
if title_element:
product_data['title'] = title_element.get_text(strip=True)
# Extract price (this selector often works, but can vary)
# It looks for common price patterns like elements with class 'a-offscreen'
# or specific price block elements.
price_element = soup.select_one('span.a-price > span.a-offscreen')
if not price_element: # Try alternative common selector
price_element = soup.select_one('span#priceblock_ourprice')
if not price_element: # Another alternative
price_element = soup.select_one('span#price_inside_buybox')
if price_element:
product_data['price'] = price_element.get_text(strip=True)
except Exception as e:
print(f"Error parsing data for ASIN {asin}: {e}")
# Basic validation: only return data if title and price were found
if product_data['title'] and product_data['price']:
return product_data
else:
print(f"Could not extract complete data for ASIN {asin}")
return None # Return None if essential data is missingWrapping the parsing in a try...except block matters here. Page structures change, some products lack certain fields, and you don't want one malformed page to crash a run of ten thousand.
Bringing it together and testing small
Now we update run_scraper to load the ASINs, loop through them, fetch, parse, and collect results. One rule that saves a lot of grief: test on a small handful of ASINs first. Don't point a fresh, untested scraper at thousands of products — validate your selectors and logic on five, then scale.
# Updated main function to run the scraper
def run_scraper():
# ... (Session and proxy setup code from earlier) ...
session = requests.Session()
# ... (Add headers and proxy config here) ...
session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Upgrade-Insecure-Requests': '1'
})
proxy_user = 'YOUR_USERNAME'
proxy_pass = 'YOUR_PASSWORD'
proxy_host = 'rp.evomi.com'
proxy_port = '1000'
proxies = {
'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}',
'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}'
}
session.proxies.update(proxies)
# --- Scraping Logic ---
asin_file = 'asins_to_scrape.csv' # Name of your CSV file
asins_to_process = load_asins_from_csv(asin_file)
if not asins_to_process:
print("No ASINs loaded. Exiting.")
return
print(f"Loaded {len(asins_to_process)} ASINs. Starting scraping...")
results = []
for asin in asins_to_process:
print(f"Processing ASIN: {asin}")
html_content, fetched_asin = fetch_product_page(session, asin)
if html_content:
product_info = extract_product_info(html_content, fetched_asin)
if product_info:
print(f"Successfully extracted: {product_info}")
results.append(product_info)
else:
print(f"Failed to extract data for ASIN: {asin}")
else:
print(f"Failed to fetch page for ASIN: {asin}")
# Optional: Add a small delay between requests to be polite
# import time
# time.sleep(1) # Sleep for 1 second
print("\nScraping complete.")
print(f"Successfully extracted data for {len(results)} products.")
# Here you would typically save 'results' to a file (CSV, JSON, database, etc.)
# print(results)
# Entry point
if __name__ == "__main__":
run_scraper()Notice the commented-out time.sleep(1). Uncomment it in production. Adding a deliberate pause between requests keeps your crawl rate reasonable and reduces load on the target — it's both courteous and more sustainable than firing requests as fast as the network allows.
To test, create asins_to_scrape.csv in the same directory as the script and add a few ASINs, one per line (for example B081FGTPB7, B07VGRJDFY).
The complete script
Here's everything assembled, with extra price-selector fallbacks and finer-grained error handling for timeouts and HTTP errors. It's a solid starting point for collecting public Amazon product data through Evomi proxies.
import requests
from bs4 import BeautifulSoup
import csv
import time # Optional: for adding delays
# Function to load ASINs from a CSV file
def load_asins_from_csv(filepath):
asin_list = []
try:
with open(filepath, mode='r', newline='', encoding='utf-8') as csvfile:
reader = csv.reader(csvfile)
for row in reader:
if row: # Ensure row is not empty
asin_list.append(row[0].strip())
except FileNotFoundError:
print(f"Error: File not found at {filepath}")
except Exception as e:
print(f"An error occurred reading the CSV: {e}")
return asin_list
# Function to fetch the product page HTML
def fetch_product_page(session, asin):
product_url = f"https://www.amazon.com/dp/{asin}"
try:
response = session.get(product_url, timeout=15)
response.raise_for_status() # Check for HTTP errors
print(f"Request successful for {asin} (Status: {response.status_code})")
return response.text, asin
except requests.exceptions.Timeout:
print(f"Request timed out for ASIN {asin}")
return None, asin
except requests.exceptions.HTTPError as e:
print(f"HTTP error for ASIN {asin}: {e.response.status_code}")
return None, asin
except requests.exceptions.RequestException as e:
print(f"Request failed for ASIN {asin}: {e}")
return None, asin
# Function to parse HTML and extract product data
def extract_product_info(html_content, asin):
if not html_content:
return None
soup = BeautifulSoup(html_content, 'lxml')
product_data = {'asin': asin, 'title': None, 'price': None}
try:
title_element = soup.select_one('span#productTitle')
if title_element:
product_data['title'] = title_element.get_text(strip=True)
# Try common price selectors sequentially
price_element = soup.select_one('span.a-price > span.a-offscreen')
if not price_element:
price_element = soup.select_one('span#priceblock_ourprice') # Older layout?
if not price_element:
price_element = soup.select_one('span#price_inside_buybox') # Inside buy box?
# Add more selectors here if needed based on page variations
if price_element:
product_data['price'] = price_element.get_text(strip=True)
else:
# If no price found, try getting text from a broader price container
price_container = soup.select_one('div#corePrice_feature_div span.a-price-whole')
if price_container:
price_fraction = soup.select_one('div#corePrice_feature_div span.a-price-fraction')
currency_symbol = soup.select_one('div#corePrice_feature_div span.a-price-symbol')
whole = price_container.get_text(strip=True)
fraction = price_fraction.get_text(strip=True) if price_fraction else '00'
symbol = currency_symbol.get_text(strip=True) if currency_symbol else '$' # Default symbol
product_data['price'] = f"{symbol}{whole}.{fraction}"
except Exception as e:
print(f"Error parsing data for ASIN {asin}: {e}")
if product_data['title'] and product_data['price']:
return product_data
else:
missing = []
if not product_data['title']: missing.append("title")
if not product_data['price']: missing.append("price")
print(f"Could not extract ({', '.join(missing)}) for ASIN {asin}")
return None
# Main execution function
def run_scraper():
# --- Evomi Proxy Configuration ---
# Replace with your actual Evomi credentials and desired proxy type/port
proxy_user = 'YOUR_USERNAME'
proxy_pass = 'YOUR_PASSWORD'
proxy_host = 'rp.evomi.com' # Example: Residential endpoint
proxy_port = '1000' # Example: HTTP port for residential
proxies = {
'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}',
'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}'
}
# --- Session Setup ---
session = requests.Session()
session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Upgrade-Insecure-Requests': '1',
'Referer': 'https://www.google.com/' # Add a referer
})
session.proxies.update(proxies)
print("Session configured. Proxies enabled.")
# --- Scraping Logic ---
asin_file = 'asins_to_scrape.csv' # Your input file
asins_to_process = load_asins_from_csv(asin_file)
if not asins_to_process:
print("No ASINs loaded or file not found. Exiting.")
return
print(f"Loaded {len(asins_to_process)} ASINs. Starting scraping...")
results = []
processed_count = 0
for asin in asins_to_process:
processed_count += 1
print(f"\n[{processed_count}/{len(asins_to_process)}] Processing ASIN: {asin}")
html_content, fetched_asin = fetch_product_page(session, asin)
if html_content:
product_info = extract_product_info(html_content, fetched_asin)
if product_info:
print(f"--> Success: Extracted {product_info['title']} - {product_info['price']}")
results.append(product_info)
# Be polite: pause between requests
time.sleep(1)
print("\nScraping complete.")
print(f"Successfully extracted data for {len(results)} products.")
# Save 'results' to CSV, JSON, or a database here.
# Entry point
if __name__ == "__main__":
run_scraper()Scaling responsibly
A few habits keep a large collection job healthy and above board:
Respect the source. Check the site's terms and robots directives, limit yourself to public data, and keep request rates moderate. The polite
time.sleeppause is there for a reason.Handle failures gracefully. Retries with backoff, timeouts, and clear logging mean one bad page won't sink a run of thousands.
Pick the right proxy type. Residential proxies suit geographically diverse consumer-facing pages; datacenter proxies are cheaper and faster where locality matters less. You can test IP geolocation at geo.evomi.com and check your setup at proxy-tester.evomi.com before a big run.
Store your output. Push
resultsinto a CSV, JSON file, or database as you go, rather than holding everything in memory.
If your target is reviews or ratings rather than pricing, the same principles apply — our guide to scraping reviews safely goes deeper on the ethics and mechanics of that specific use case.
With a session for connection reuse, ethically sourced rotating proxies for geographic reach and load distribution, and defensive parsing, you have a foundation that scales from ten products to tens of thousands — cleanly and responsibly.
Author
David Foster
Proxy & Network Security Analyst
About Author
David is an expert in network security, web scraping, and proxy technologies, helping businesses optimize data extraction while maintaining privacy and efficiency. With a deep understanding of residential, datacenter, and rotating proxies, he explores how proxies enhance cybersecurity, bypass geo-restrictions, and power large-scale web scraping. David’s insights help businesses and developers choose the right proxy solutions for SEO monitoring, competitive intelligence, and anonymous browsing.
