Why Your Scraping Architecture Should Look Like a Data Pipeline, Not a Script

Most scraping operations start as scripts. A Python file. A cron job. Maybe Scrapy with a few spiders. It works. Data lands in a CSV or a Postgres table. Everyone's happy.

Then the requirements grow. More targets. Higher frequency. Data quality requirements. Downstream consumers with SLAs. The script becomes a collection of scripts. The cron jobs multiply. Someone adds a retry loop. Then a second retry loop with different logic. Then a monitoring job to watch the monitoring job.

What started as a script is now a distributed system, designed accidentally, documented poorly, and understood by at most two people.

The teams that avoid this trajectory aren't smarter. They started with the right mental model: scraping is data engineering, not automation scripting. The architecture should reflect that from day one.

The Script Model and Why It Breaks

The canonical scraping script does several things in sequence: fetch a page, parse it, transform the data, write it somewhere. This works at low volume and low complexity. The problems emerge when any single dimension scales:

Volume scaling breaks sequential processing. A script that scrapes 500 pages serially starts hitting rate limits, timeout issues, and memory pressure. You add threading or async. Now you have concurrency bugs, shared state problems, and retry logic that doesn't compose cleanly.

Target scaling breaks the monolith. Ten spiders in one process means one bad spider can crash the entire job. Configuration for ten targets in one file becomes unmanageable. Deployment of changes to one target requires redeploying everything.

Reliability requirements expose the lack of checkpointing. If the job fails at page 47,000 of 50,000, a restart from scratch is expensive. Without durability, every failure restarts the entire run.

Downstream consumers require data guarantees that ad-hoc scripts can't provide. When a dashboard, a model, or a partner integration depends on your scraped data, "it usually works" becomes "it has a defined SLA."

The script model eventually demands all the properties of a data pipeline, idempotency, checkpointing, retry semantics, schema contracts, observability, but adds them piecemeal, on top of architecture that wasn't designed for them.

The Pipeline Mental Model

A scraping pipeline separates concerns that scripts conflate:

[Seed Generation] → [URL Queue] → [Fetch Workers] → [Parse Workers] → [Validate] → [Storage]
                                       ↑                                    ↓
                                  [Proxy Pool]                       [Quarantine]

Each stage has a defined contract:

• Seed Generation produces a set of URLs to process. It's idempotent, running it twice doesn't create duplicate work.

• URL Queue is a durable message queue (Cloud Pub/Sub, SQS, Redis streams). Items remain in the queue until explicitly acknowledged. Crashes don't lose work.

• Fetch Workers pull from the queue, execute requests through proxy rotation, and push raw HTML to the next stage. They don't parse. They fetch and forward.

• Parse Workers receive raw content and extract structured data. They don't know about proxies or HTTP. They parse and forward.

• Validate applies schema checks and range guards. Failed records go to quarantine. Passing records proceed.

• Storage receives clean, validated records. Idempotent upserts prevent duplicates on retry.

This separation means you can scale fetch workers independently of parse workers. You can replay a parse stage without re-fetching. You can swap proxy providers without touching parsing logic.

Implementing with Cloud Pub/Sub

Cloud Pub/Sub is the natural choice for GCP-based scraping pipelines. It's fully managed, scales to millions of messages, and provides at-least-once delivery with acknowledgment-based retry.

from google.cloud import pubsub_v1
import json

def publish_urls(project_id: str, topic_id: str, urls: list[str]):
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path(project_id, topic_id)
    
    for url in urls:
        message = {
            'url': url,
            'target': 'product_catalog',
            'priority': 'normal',
            'enqueued_at': datetime.utcnow().isoformat()
        }
        data = json.dumps(message).encode('utf-8')
        future = publisher.publish(topic_path, data=data)
        future.result()  # Block until confirmed

def fetch_worker(project_id: str, subscription_id: str):
    subscriber = pubsub_v1.SubscriberClient()
    subscription_path = subscriber.subscription_path(project_id, subscription_id)
    
    def callback(message: pubsub_v1.subscriber.message.Message):
        payload = json.loads(message.data.decode('utf-8'))
        try:
            html = fetch_with_proxy(payload['url'])
            publish_to_parse_topic(payload['url'], html)
            message.ack()
        except Exception as e:
            # NACK to retry with backoff
            message.nack()
    
    subscriber.subscribe(subscription_path, callback=callback)

The key property: if the fetch worker crashes mid-batch, unacknowledged messages are redelivered. No work is lost.

GCS as the Intermediate Store

For large-scale pipelines, raw HTML between fetch and parse stages shouldn't flow through the message queue directly, HTML payloads can be hundreds of KB, and message queues are optimized for small messages. The pattern: fetch workers write raw HTML to Google Cloud Storage, publish only the GCS object path to the queue.

from google.cloud import storage
import hashlib

def store_raw_html(url: str, html: str, bucket_name: str) -> str:
    client = storage.Client()
    bucket = client.bucket(bucket_name)
    
    # Content-addressed storage: same URL + same content = same key
    content_hash = hashlib.sha256(html.encode()).hexdigest()[:16]
    object_key = f"raw/{datetime.utcnow().strftime('%Y/%m/%d')}/{content_hash}.html"
    
    blob = bucket.blob(object_key)
    blob.upload_from_string(html, content_type='text/html')
    return object_key

This gives you a complete audit trail: every raw page you ever fetched, content-addressed. You can replay any parse run against historical fetches. When a bug in your parser corrupts three days of data, you don't re-fetch — you re-parse from the stored HTML.

BigQuery as the Destination

For analytics-oriented scraping pipelines, BigQuery is the right destination. Its columnar storage is optimized for the query patterns that matter: aggregations over large datasets, time-series analysis, cross-target comparisons.

from google.cloud import bigquery

def write_to_bigquery(records: list[dict], table_id: str):
    client = bigquery.Client()
    table = client.get_table(table_id)
    
    errors = client.insert_rows_json(table, records)
    if errors:
        raise ValueError(f'BigQuery insert errors: {errors}')

For the streaming case, real-time scraping results, BigQuery Storage Write API with committed mode provides exactly-once semantics. For batch loads, the standard load job is cheaper and better for high-volume historical backfills.

When Event-Driven Architecture Pays Off

The pipeline model above is still batch-oriented: seed, fetch, parse, store. For use cases that require lower latency, competitive price monitoring, real-time news aggregation, inventory tracking, the architecture needs to be reactive.

Event-driven scraping uses change detection as the trigger rather than a schedule. Instead of "scrape these 10,000 pages every 6 hours", the architecture is: "when a price changes, trigger dependent processes immediately."

The mechanism: run a lightweight sentinel scraper on a schedule to detect changes, publish change events to Pub/Sub when detected, and let downstream consumers subscribe to the events they care about. A pricing model re-runs. An alert fires. A cache is invalidated. The scraper's job is to detect changes and publish them; what happens in response is decoupled.

This is more complex to build but produces lower-latency data and reduces unnecessary scraping of unchanged pages, which matters for both cost and target friendliness.

The Proxy Layer as a First-Class Component

In the pipeline model, proxy management is a first-class component, not embedded in fetch logic. The proxy pool should expose a clean interface that fetch workers call, without caring about the underlying provider.

class ProxyPool:
    def __init__(self, provider_config: dict):
        self.host = provider_config['host']
        self.port = provider_config['port']
        self.username = provider_config['username']
        self.password = provider_config['password']

    def get_proxy_url(self, country: str = None) -> str:
        auth = f"{self.username}:{self.password}"
        if country:
            auth = f"{self.username}-country-{country}:{self.password}"
        return f"http://{auth}@{self.host}:{self.port}"

# Evomi residential proxy configuration
pool = ProxyPool({
    'host': 'rp.evomi.com',
    'port': 1000,  # HTTP (use 1001 for HTTPS, 1002 for SOCKS5)
    'username': 'your_username',
    'password': 'your_password'
})

This abstraction means switching proxy providers, testing multiple providers in parallel, or routing different target tiers to different proxy products is a configuration change, not a code change. Evomi's residential proxies plug in here naturally, the endpoint format is consistent, and geo-targeting by country code integrates cleanly into the proxy URL.

The Architecture Pays Dividends

The pipeline model costs more to set up than a script. It requires thinking through stage contracts, queue topology, and retry semantics upfront. Engineers trained on scripting find it unfamiliar.

It pays for itself at the first production incident.

When a parse bug corrupts two days of data, you replay from GCS. When a fetch worker crashes, the queue redelivers. When you add a new target, it's a new seed generator feeding the same pipeline. When a downstream consumer's requirements change, you add a new subscription, without touching the fetch layer.

Scraping is data engineering. Build it like data engineering, from the start.