Read Timeout vs Connect Timeout: Tuning Scraper Timeouts

Two scrapers, same bug, opposite symptoms. The first one ran for nine hours overnight and scraped 400 pages, it was stuck the whole time on a single request to an exit node that accepted the TCP connection and then went silent. No error, no exception, just a thread blocked forever. The second one threw ReadTimeout on roughly one request in twenty, including pages that loaded fine on a retry. Same root cause for both: a timeout that was set wrong, or not set at all.

A timeout is not one number. An HTTP request goes through several distinct phases, and "the request took too long" can mean any of them stalled. If you collapse all of that into a single value, or skip it entirely, you get exactly these two failure modes: hangs that never resolve, and premature kills on requests that were simply slow to start.

This post takes the request apart phase by phase, maps each phase to the actual parameter in requests, httpx, and aiohttp, and then explains why a proxy changes the arithmetic and what to set when one is in the path.

The Phases of a Request

When you call get(url), this happens in order:

1. DNS resolution. The hostname becomes an IP. Usually fast, occasionally not.

2. TCP connect. The three-way handshake. This is "can I reach the server at all."

3. TLS handshake. For HTTPS, certificate exchange and key negotiation on top of the TCP connection.

4. Sending the request. Writing your headers and body onto the socket.

5. Time to first byte (read). You've sent the request; now you wait for the server to start replying. This is where slow backends and overloaded proxy exits hurt.

6. Receiving the body. Reading the response bytes until the connection closes or content-length is met.

Phases 1 through 4 are roughly "establishing the connection." Phase 5 is the server thinking. Phase 6 is transfer. Most HTTP clients group these into two buckets you can control, connect (getting to the point where the request is sent) and read (waiting for and receiving the response), plus, in better libraries, a total ceiling over everything.

requests: the (connect, read) tuple

requests lets you pass timeout as a single number or, better, as a tuple:

import requests

# (connect timeout, read timeout) in seconds
resp = requests.get(
    "https://httpbin.org/delay/2",
    timeout=(5, 15),
)
print(resp.status_code)

The connect value covers DNS, TCP, and TLS, everything up to the moment the socket is ready to send. The read value is the maximum gap between bytes once you're reading the response. That second part is the gotcha that bites people.

The read timeout is not a total deadline. It resets every time a byte arrives. A server that trickles one byte every 14 seconds, forever, will never trip a read=15 timeout, each individual gap is under the limit, so requests waits indefinitely. That is exactly how the overnight-hang scraper got stuck: the exit accepted the connection (connect phase passed) and then dribbled data slowly enough to stay under the per-read window.

requests has no built-in total-time option. If you need a hard ceiling, you enforce it from outside, a concurrent.futures timeout, a signal alarm, or by moving to a client that has one. Which is a good reason to look at httpx.

httpx: four phases, named honestly

httpx splits the connection bucket apart and names each piece:

import httpx

timeout = httpx.Timeout(
    connect=10.0,   # TCP + TLS to the server (or proxy)
    read=20.0,      # max wait between received chunks
    write=10.0,     # max wait while sending the request body
    pool=5.0,       # max wait to get a connection from the pool
)

with httpx.Client(timeout=timeout) as client:
    resp = client.get("https://httpbin.org/delay/2")
    print(resp.status_code)

write matters when you POST large payloads; pool matters under concurrency, when all pooled connections are busy and a new request has to wait its turn. Splitting these out is more honest than the requests tuple, a slow pool checkout and a slow server are genuinely different problems, and now you can set them separately.

httpx still inherits the same per-read semantics: read is the gap between chunks, not a total. But you can set a single number to apply a sane default to all four, and combine it with your own overall deadline. For a true ceiling, wrap the call:

import httpx

# A blanket 30s applied to connect/read/write/pool...
client = httpx.Client(timeout=30.0)

# ...plus a hard wall-clock ceiling you control yourself.
# httpx has no single "total" param, so cap it at the call site
# (e.g. asyncio.wait_for in async code, or a watchdog thread sync)

If you want the framework to own the total deadline outright, aiohttp does.

aiohttp: ClientTimeout with a real total

import aiohttp

timeout = aiohttp.ClientTimeout(
    total=45,         # hard ceiling on the whole operation
    connect=15,       # acquiring a connection (incl. pool wait)
    sock_connect=10,  # just the socket connect to the host
    sock_read=20,     # max gap between received bytes
)

async def fetch(url):
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(url) as resp:
            return await resp.text()

total is the one the other two libraries lack: a wall-clock cap on connect plus think-time plus transfer. sock_connect is the pure socket connect, while connect also includes time spent waiting on the connection pool. sock_read is the per-chunk gap, same idea as read elsewhere. Setting total means the trickle-forever exit gets killed no matter how it stalls, which is the behavior you almost always want.

Why Proxies Change the Math

Without a proxy, "connect" is one hop: you to the target. With a residential or mobile proxy, connect now covers a chain, your client to the provider's gateway, the gateway to a chosen exit (a real device on a home or cellular network), and the exit out to the target. That is three legs instead of one, and the middle leg is the slow one.

Residential and mobile exits sit behind consumer connections. They are slow to establish a connection, the device might be on weak Wi-Fi, mid-NAT-traversal, or simply busy. A connect timeout that's fine for direct datacenter traffic (say 3 seconds) will reject a large fraction of perfectly usable residential exits before they finish the handshake.

So the proxy guidance inverts the usual instinct:

• Connect: generous. Give the chain time to assemble. A too-tight connect timeout doesn't make you faster; it just discards working exits and forces retries.

• Read: tighter. Once a good exit has connected and the target has started responding, data should flow at a normal pace. A long stall after connect usually means a dead or rate-limited exit, and you want to cut it loose quickly, not wait.

• Total: always set one. This is your safety net against the trickle-forever case that per-read timeouts cannot catch.

Starting Values (Then Measure)

These are starting points for proxied scraping, not gospel. Tune them against your own traffic.

• Direct (no proxy / datacenter): connect 5s, read 15s, total 30s.

• Residential / ISP proxy: connect 10s, read 20s, total 45s.

• Mobile proxy: connect 15s, read 25s, total 60s.

To tune from data, log time-to-first-byte for successful requests and look at the distribution, not the average. Set your read timeout near the 95th or 99th percentile of observed TTFB plus a margin, high enough that legitimately slow-but-fine responses survive, low enough that dead exits get cut. If your p99 TTFB is 6 seconds, a 20-second read timeout is reasonable; a 60-second one just means you wait a full minute on every dead exit before retrying.

Pair Timeouts With Retries on a Fresh Exit

A timeout should not crash the run, it should trigger a retry, and on a proxy a retry should go out through a different exit. The exit that just timed out is exactly the one you don't want to ask again. Pair every timeout with a backoff and a new connection:

import time, requests

def fetch(url, proxies, attempts=3):
    for i in range(attempts):
        try:
            return requests.get(url, proxies=proxies, timeout=(10, 20))
        except (requests.ConnectTimeout, requests.ReadTimeout):
            if i == attempts - 1:
                raise
            time.sleep(2 ** i)  # 1s, 2s, 4s backoff
            # rotate to a fresh exit before the next attempt
            proxies = next_exit()

With rotating residential pools, "a fresh exit" often just means making a new request through the same gateway endpoint, which assigns a different device. The point is that the retry must not reuse the dead path. Backoff (the 2 ** i here) keeps a struggling target from being hammered while it recovers.

Mistakes That Waste Time

• No timeout at all. The default in requests is no timeout, a single stalled connection blocks that thread forever. This is the overnight-hang bug. Always pass one.

• One tiny number for everything. timeout=3 through a residential proxy rejects most exits during connect. The connect phase legitimately needs more time than the read phase here.

• Treating the read timeout as a total deadline. It's a per-byte gap, not a wall clock. A slow trickle never trips it. Set a real total (or wrap the call).

• Retrying instantly through the same dead exit. No backoff, same proxy endpoint reused with a sticky session, you just hit the same broken device again and burn an attempt.

• Setting read so high you can't tell dead from slow. A 120-second read timeout means every dead exit costs two minutes before you learn it's dead. Cut it down toward your measured p99.

Wrapping Up

Split your timeout into phases and the two failure modes go away. For proxied scraping, set connect generously so slow residential exits get a chance to assemble the chain, set read tighter so a stalled exit gets dropped fast, and always set a total ceiling so nothing can trickle forever, only httpx (via your own wrapper) and aiohttp (via total) make that last one easy. Then measure TTFB and pull the read timeout down toward your real p99. Pair every timeout with a backoff and a fresh exit, and the request that hangs your scraper today becomes one quick retry instead.