Node-Fetch Proxies: Bypass Blocks & Boost Your Web Scraping

Getting Started with Node-Fetch and Proxies

Leveraging proxies with Node-Fetch can significantly elevate your web scraping projects and other automated tasks.

Node-Fetch is a widely-used module that ports the browser's Fetch API standard to the Node.js environment. It allows your backend applications to make HTTP requests, fetch web page content, and submit data – making it a cornerstone for activities like web scraping using Node.js. It's worth noting that Node.js has included native support for `fetch` since version 18, potentially removing the need for the separate `node-fetch` module in newer projects.

However, a common hurdle arises quickly: vanilla Node-Fetch doesn't have a built-in, straightforward option for routing requests through proxies. Sending numerous requests from your server's single IP address is a fast track to getting flagged and blocked by target websites.

This guide will walk you through integrating proxies with Node-Fetch using an additional library. We'll cover how to configure proxies, handle authentication, and even customize request headers like the User-Agent to make your requests look more natural and avoid detection.

Let's dive in!

Understanding Fetch in the Node.js Context

The Fetch API in Node.js, whether via the `node-fetch` module or the native implementation, mirrors the functionality of the Fetch API found in modern web browsers. The key distinction lies in the environment: the browser's Fetch API operates client-side, initiated by user interactions within the browser, while Node.js Fetch runs server-side, enabling programmatic HTTP communication from your backend.

So, while front-end developers utilize the Fetch API for asynchronous resource loading within the browser, backend developers employ Node-Fetch (or native fetch) to build automated processes, interact with external APIs, and, pertinent to our discussion, perform web scraping tasks directly from the server.

Is Node-Fetch Identical to Browser Fetch?

While not exactly the same underlying code, `node-fetch` (the module) and the native `fetch` in Node.js are designed to be syntactically and functionally very close to the browser's `window.fetch()`.

The primary difference remains the execution environment. `window.fetch()` is a browser-native function, part of the web platform APIs executed on the client machine. `node-fetch` (or Node's built-in fetch) is a library/feature for the Node.js runtime, allowing server-side code to make HTTP requests programmatically.

Implementing Proxies with Node-Fetch

There are a couple of primary strategies for channeling your Node-Fetch requests through a proxy. The most common involves using a helper library that modifies the request agent, while another approach might utilize a system-level or reverse proxy setup (though that's beyond the scope of direct Node-Fetch configuration).

Keeping Your Web Scraper Unblocked Using Proxies

Web scraping, while a powerful and often legitimate technique for data gathering, is frequently met with resistance from website operators. Websites employ various detection methods, but two major red flags they look for are the originating IP address and the characteristics of the request headers.

Repeated, rapid requests from a single IP address are a strong indicator of automated activity, differing significantly from typical human browsing patterns. Similarly, requests lacking standard browser headers (like `User-Agent`, `Accept-Language`, etc.) or having headers indicative of automated tools can trigger blocks.

Proxies are the go-to solution for the IP address problem. By routing your requests through a proxy server, the target website sees the proxy's IP, not your server's original IP. Using a pool of proxies, especially residential proxies which use IP addresses assigned to real home internet connections, makes your requests appear to originate from many different, legitimate users across various locations.

Providers like Evomi offer access to vast pools of ethically sourced residential, datacenter, and mobile proxies. When you sign up, you typically receive credentials (username/password) and endpoint details (hostname/IP and port number) specific to the proxy type you need. For example, Evomi's residential proxies might use an endpoint like rp.evomi.com with a specific port for HTTP, HTTPS, or SOCKS5 connections. Some providers also offer IP whitelisting, allowing you to connect from a pre-approved IP address without needing credentials for every request.

Now, let's see how to integrate these proxy details into your Node-Fetch code.

Using the `https-proxy-agent` Library

A practical way to tell Node-Fetch to use a proxy is by employing a custom agent. The `https-proxy-agent` library, created by Nathan Rajlich, is a popular choice for this, specifically designed for HTTPS proxies (though it often handles HTTP proxies too).

First, you'll need to install the necessary packages. If you're using Node.js v18 or newer, native `fetch` is available, but you still need the agent. If using an older Node version, you'll need `node-fetch` as well:

npm install https-proxy-agent
# Only if using Node.js < v18:
# npm install node-fetch

In your script, you import `HttpsProxyAgent` and instantiate it with your proxy server's URL. Then, you pass this agent instance in the `agent` option of your `fetch` call:

// Use native fetch (Node >= 18) or import if needed:
// const fetch = require('node-fetch'); // For Node < 18
const HttpsProxyAgent = require('https-proxy-agent');

(async () => {
  // Replace with your actual proxy URL (e.g., from Evomi dashboard)
  // This example assumes IP Whitelisting is configured for your server IP
  const proxyUrl = 'http://rp.evomi.com:1000'; // Example using Evomi residential HTTP endpoint
  const proxyAgent = new HttpsProxyAgent(proxyUrl);

  try {
    const response = await fetch('https://httpbin.org/ip', { agent: proxyAgent });
    const data = await response.json(); // httpbin.org/ip returns JSON
    console.log('IP address via proxy:', data.origin);
    // You can verify the IP using tools like https://geo.evomi.com/
  } catch (error) {
    console.error('Error fetching via proxy:', error);
  }
})();

Note that the example above doesn't include authentication details in the `proxyUrl`. This works if your proxy provider allows IP whitelisting and you've configured it for your server's IP address.

To use username/password authentication, simply embed them in the proxy URL like so:

// Use native fetch (Node >= 18) or import if needed:
// const fetch = require('node-fetch'); // For Node < 18
const HttpsProxyAgent = require('https-proxy-agent');

(async () => {
    // Replace with your actual proxy credentials and Evomi endpoint
    const proxyUser = 'your_username';
    const proxyPassword = 'your_password';
    const proxyHost = 'rp.evomi.com';
    const proxyPort = 1001; // Example using Evomi residential HTTPS endpoint

    const proxyUrl = `http://${proxyUser}:${proxyPassword}@${proxyHost}:${proxyPort}`;
    // Note: Even for HTTPS proxies, HttpsProxyAgent often uses 'http://' in the credential string format. Check library docs if unsure.
    // Or directly format as needed by the library, sometimes it expects an object.
    // Let's try the standard URL format first which is common.

    const proxyAgent = new HttpsProxyAgent(proxyUrl);

    try {
        const response = await fetch('https://httpbin.org/ip', { agent: proxyAgent });
        const data = await response.json();
        console.log('IP address via authenticated proxy:', data.origin);
    } catch (error) {
        console.error('Error fetching via authenticated proxy:', error);
    }
})();

The `https-proxy-agent` library offers other ways to pass credentials as well, consult its documentation for more advanced configurations if needed.

Customizing User Agents with Node-Fetch Proxies

Beyond just masking your IP, sending realistic browser headers is crucial for avoiding blocks. The `User-Agent` header is particularly important. You can easily add custom headers to your `fetch` request using the `headers` option, combining it with the `agent` option for proxy usage.

Instead of a basic proxied request:

const response = await fetch('https://httpbin.org/ip', { agent: proxyAgent });

You can include a `headers` object:

const response = await fetch('https://httpbin.org/ip', {
    agent: proxyAgent,
    headers: {
        'User-Agent': 'Some realistic browser User-Agent string'
        // Add other headers like Accept, Accept-Language etc.
    }
});

Here's a combined example using an authenticated proxy and a custom User-Agent:

// Use native fetch (Node >= 18) or import if needed:
// const fetch = require('node-fetch'); // For Node < 18
const HttpsProxyAgent = require('https-proxy-agent');

(async () => {
    // Replace with your actual proxy credentials and Evomi endpoint
    const proxyUser = 'your_username';
    const proxyPassword = 'your_password';
    const proxyHost = 'rp.evomi.com';
    const proxyPort = 1001; // Example using Evomi residential HTTPS endpoint
    const proxyUrl = `http://${proxyUser}:${proxyPassword}@${proxyHost}:${proxyPort}`;
    const proxyAgent = new HttpsProxyAgent(proxyUrl);

    const customHeaders = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.9'
        // Add other relevant headers
    };

    const requestOptions = {
        agent: proxyAgent,
        headers: customHeaders
    };

    try {
        // Let's fetch headers echoed back by httpbin to verify
        const response = await fetch('https://httpbin.org/headers', requestOptions);
        const data = await response.json();
        console.log('Headers sent via proxy:');
        console.log(data.headers);
    } catch (error) {
        console.error('Error fetching with custom headers via proxy:', error);
    }
})();

By combining proxy rotation with realistic request headers, you significantly increase the resilience and success rate of your Node-Fetch based web scraping operations.

Wrapping Up

In this post, we explored how to configure and use proxies with Node-Fetch (both the module and the native implementation). We covered using the `https-proxy-agent` library to route requests, handle authentication, and customize headers like the User-Agent. These techniques are essential for building robust web scrapers or any Node.js application that needs to make numerous web requests without hitting IP-based rate limits or blocks.

Armed with this knowledge, you can now more effectively gather data from the web using Node.js, integrating the fetched content with parsers or other data processing logic as needed.

Happy fetching!