Multi-Region Crawling

Your price-monitoring job has been green for three weeks. Then someone in the commercial team opens the dashboard and asks why a mid-range headphone SKU is €199 in Germany, £199 in the UK, and $199 in the US, the same number, three currency symbols, twelve markets, no variation at all.

Nothing crashed. The crawler runs in us-east-1, every request exits through a US IP, and the site serves the same US variant each time. Your pipeline then formats that one number with whatever currency symbol the country column said it should have. It is one observation, copied twelve times.

Why One Region Gives You One Answer

Several mechanisms make a response region-dependent, and they operate independently:

IP geolocation. Your exit IP selects the catalogue, currency, tax display and shipping estimate. VAT-inclusive and VAT-exclusive are different numbers for the same product; so is "unavailable in your region."
Accept-Language negotiation. You advertise language preferences (RFC 9110 §12.5.4); the server picks a representation and names it in Content-Language (§8.5).
hreflang annotations. <link rel="alternate" hreflang="..."> tags, Link: headers or sitemap entries declare which localised versions exist. Google's documentation is explicit that they do not decide what gets served — a map, not proof of what arrived.
Geo-redirects. A 302 to /de/, de.example.com or a country-selector interstitial lands you somewhere other than the page you queued.
CDN edge variation and Vary. Edges cache per-variant, keyed on what the origin listed in Vary(§12.5.5). If the origin varies on a geography header the CDN injects but never declares, the right edge serves the wrong region's copy.
Consent banners. EEA variants often ship a different DOM, a modal, deferred scripts, nodes that appear only post-consent, and your selectors were written against the US DOM.

Because those mechanisms never check each other, a Frankfurt exit IP with Accept-Language: en-US and timezone America/New_York is a combination no real visitor produces. This is the coherence problem from fingerprinting, applied to geography: the page you get back is a hybrid nobody sees, so the price on it is not a price anyone is offered.

Getting a Genuine Regional View

Egress options, roughly by how much geography they buy:

Cloud regions. Cheap and already in your bill, but a datacenter ASN in Frankfurt is visibly a datacenter, and granularity stops at your provider's region list.
Proxy exit locations by country and city. The workhorse: residential and ISP pools give a consumer-looking exit in a chosen country, usually selected through the username string. Evomi, for one, publishes country, city and ASN targeting on its residential pool within the base per-GB rate, worth checking on any provider, since targeting surcharges and premium-location tiers are common.
Mobile carrier egress. A carrier's CGNAT range in a country; right when the site serves a distinct mobile experience, at several times the residential per-GB cost.

Set the region once and derive the rest. Playwright's locale is documented to affect navigator.language, the Accept-Language header and number/date formatting; timezone_id sets the browser timezone. One config object per region drives all three:

Shell

import asyncio
from playwright.async_api import async_playwright

# Endpoint host/port and the geo-targeting username syntax are provider-specific;
# check your provider's docs for the exact format.
PROXY_HOST = "http://proxy.example.net:8000"
PROXY_USER = "your-username"
PROXY_PASS = "your-password"

REGIONS = {
    "de": {"cc": "de", "locale": "de-DE", "tz": "Europe/Berlin"},
    "gb": {"cc": "gb", "locale": "en-GB", "tz": "Europe/London"},
    "jp": {"cc": "jp", "locale": "ja-JP", "tz": "Asia/Tokyo"},
    "us": {"cc": "us", "locale": "en-US", "tz": "America/New_York"},
}


async def observe(region: str, url: str) -> dict:
    cfg = REGIONS[region]
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        ctx = await browser.new_context(
            proxy={
                "server": PROXY_HOST,
                "username": f"{PROXY_USER}-country-{cfg['cc']}",
                "password": PROXY_PASS,
            },
            locale=cfg["locale"],      # navigator.language + Accept-Language
            timezone_id=cfg["tz"],     # browser timezone
        )
        page = await ctx.new_page()
        resp = await page.goto(url, wait_until="domcontentloaded")
        row = {
            "region": region,
            "status": resp.status,
            "requested_url": url,
            "final_url": page.url,
            "content_language": resp.headers.get("content-language"),
            "html_lang": await page.get_attribute("html", "lang"),
            "canonical": await page.get_attribute("link[rel=canonical]", "href"),
        }
        await browser.close()
        return row


print(asyncio.run(observe("de", "https://example.com/product/123")))

The return value carries not just the price, but evidence of which variant answered.

Fanning Out

Per-region workers means a full crawler deployment per region. Simple in isolation, painful to operate: N deployments to upgrade, N dashboards, N frontiers that drift until one region is silently crawling last month's SKU list.

One control plane with regional egress keeps a single scheduler and frontier; workers are thin fetchers pinned to one region config each. What to crawl stays central, only the fetch is regional, and the frontier remains the single source of truth about coverage.

Either way the schema consequence is the same: price is no longer a function of (product, time) but of (product, region, time). Region belongs in the primary key, and the evidence columns belong next to it, exit_country, locale_sent, content_language, final_url, currency_symbol. Someone will eventually ask whether a €40 France–Italy gap was real or a misrouted exit, and only those columns answer.

Verifying You Actually Got a Regional View

Geo-targeting fails quietly — it does not throw, it hands you the wrong variant. Assert, per response:

Content-Language matches what you asked for. You sent de-DE and got en? Negotiation did not go your way.
The currency symbol matches the region. A $ in a row tagged de is a failed observation, not a cheap product.
final_url is in the expected locale path or host, and canonical and hreflang agree with the variant you think you have. A canonical pointing at the US page signals a fallback or cached copy.
A canary page. One URL whose content you know differs by market, shipping info, a market-specific legal notice, a product unavailable in one country. Diff it against a stored per-variant fingerprint at the start of every regional run. When the canary stops varying, you find out before 200,000 rows land tagged wrong.

Fail the row, not the run. An unverifiable region should produce a null with a reason code, never a value.

Where the Data Lives

The first distinction is what you collect. Public, non-personal product and price data is a materially different question from personal data, and personal data under the GDPR is broad: Article 4(1) covers any information relating to an identifiable natural person. Reviews with usernames, seller profiles, Q&A threads, anything joinable to an individual. Which side of that line you sit on decides whether the rest applies at all.

If you handle personal data and move it out of the EEA, Chapter V of the GDPR governs. In outline:

Adequacy decisions (Article 45). The Commission may find a third country offers adequate protection, and transfers there need no further instrument; recognised countries include Switzerland, Japan, the UK and Uruguay. The EU–US Data Privacy Framework was adopted as Decision (EU) 2023/1795 on 10 July 2023, and its validity is still in litigation — the General Court dismissed the challenge in Latombe v Commission on 3 September 2025, now on appeal (C-703/25 P).
Appropriate safeguards (Article 46). Usually the Standard Contractual Clauses in Commission Implementing Decision (EU) 2021/914 of 4 June 2021, or Binding Corporate Rules under Article 47 within a group.
Transfer impact assessment. Article 46 brings Clause 14 of the 2021 SCCs, "Local laws and practices affecting compliance with the Clauses", requiring you to assess whether the importer's local law actually permits compliance. The EDPB's Recommendations 01/2020 give the method.

Then the rule that bites engineers: where you store and process matters as much as where you collected.A crawler exiting through a German IP that writes to a bucket in us-east-1 has moved the data. Storage region is a compliance decision, not a latency one, and provider commitments are worth reading against your real flows, AWS opened its European Sovereign Cloud in Brandenburg in January 2026, Microsoft completed its EU Data Boundary in February 2025, Google Cloud publishes per-service at-rest residency options.

Whether Chapter V applies at all, and which instrument fits, depends on exactly what you collect and where it goes; none of the above is a conclusion about your pipeline. Write the flows down, collected in, processed in, stored in, accessed from, and take that to counsel.

Cost Math: One Region vs Six

All figures are estimates from published list prices; substitute your own rate card. Assumptions: 200,000 pages/day/region, 300 KB per fetch, 30-day month, six regions. Bandwidth per region is 200,000 × 300 KB = 60 GB/day = 1,800 GB/month.

Proxy bandwidth, at a published residential rate of $0.49/GB:

One region: 1,800 × $0.49 = $882/month
Six regions: 10,800 × $0.49 = $5,292/month
If one of the six needs mobile egress at a published $2.20/GB (largest plan tier): 9,000 × $0.49 = $4,410 plus 1,800 × $2.20 = $3,960 → $8,370/month

That one substitution costs more than the other five regions combined — premium locations are where multi-region budgets go.

Compute. At AWS's published us-east-1 rate for an on-demand t3.medium, $0.0416/hour (other regions differ): 730 × $0.0416 = $30.37/month per worker, $182/month for six.

Cross-region transfer, at AWS's typical $0.02/GB inter-region rate, depends on what you ship. Parsed 2 KB rows: 5 remote regions × 200,000 × 2 KB = 2 GB/day = 60 GB/month = $1.20/month. Raw HTML: 5 × 200,000 × 300 KB = 300 GB/day = 9,000 GB/month = $180/month.

Storage duplication. Regional raw copies at roughly 4:1 gzip: 75 KB × 200,000 × 6 = 90 GB/day = 2,700 GB/month added. At S3 Standard's $0.023/GB-month (us-east-1, first 50 TB) month one is $62, but it accumulates: by month twelve you hold ~32,400 GB and pay about $745/month.

Rough totals. Single region ≈ $882 + $30 + $10 ≈ $922/month; six all-residential regions ≈ $5,292 + $182 + $1 + $62 ≈ $5,537. Cost is near-linear in region count because proxy bandwidth dominates, so decide per region — six regions you act on beats twelve you collect for completeness.

Mistakes That Waste Time

Setting only the proxy country. Exit IP in Japan, Accept-Language: en-US, timezone UTC — a hybrid page and a price nobody is quoted.
Setting only the header. Accept-Language: de-DE from a US IP usually gets the US variant in German — worse than English, because it looks like it worked.
Trusting hreflang as proof of what arrived. It declares what exists; Content-Language and the currency symbol say what you got.
Choosing storage regions for latency. Cheap and fast is not the constraint governing where personal data may sit.

Wrapping Up

Two changes fix most of this. Make region one config key that drives exit IP, Accept-Language, timezone and locale together, and put region in the primary key, a crawler that cannot say where it stood is not measuring anything. Then verify the variant on every response with Content-Language, the currency symbol, the final URL and a canary you know varies, nulling failed rows instead of storing them.

Before scaling the fan-out, write down where each region's data is collected, processed and stored, and take that document to counsel rather than to a blog post, this one included.