Axios Retry for Failed Requests: Scraping & Examples

Handling Failed HTTP Requests with Axios Retry

When you're building applications that talk to external services, failed requests are just a fact of life. Networks hiccup, servers get overloaded – stuff happens. That's where Axios Retry comes in. It’s a handy plugin for the popular Axios HTTP client that automatically intercepts failed requests and tries them again according to rules you set. This is super useful in many scenarios, especially for tasks like web scraping where you're hitting lots of URLs and bumps are expected.

Think about it: without some kind of retry mechanism, your application might just grind to a halt the first time it encounters a temporary network glitch or a server returning a transient error. Axios, by itself, doesn't handle retries; it just throws an error. You could wrap every request in a try...catch block and build your own retry logic, but why reinvent the wheel?

Axios Retry simplifies this immensely. It's straightforward to use, plays nicely with asynchronous code (async/await), and offers a good deal of flexibility for customizing how and when retries should happen. Let's dive into how it works.

Why Use Axios Retry? Key Capabilities

Out of the box, Axios simply reports an error when a request fails. While JavaScript's native error handling can catch these, implementing a robust retry strategy from scratch takes effort. Axios Retry abstracts away that complexity.

Its main draw is the configuration power it gives you. You can define:

• How many times to retry a failed request.

• Which conditions trigger a retry (e.g., specific HTTP status codes like 5xx server errors, network timeouts).

• The delay between retries, including strategies like exponential backoff to avoid hammering a struggling server.

• Custom logic to decide if a specific request should be retried based on its details.

This level of control is invaluable for automated processes. Consider web scraping: a bot might need to fetch data from thousands of pages. Temporary errors are almost guaranteed. With Axios Retry configured, the bot can automatically retry problematic requests without interrupting the overall process, making the data collection far more reliable and efficient. Without it, you'd either lose data or face frequent manual interventions.

Getting Started: A Practical Axios Retry Example

First things first, you'll need Node.js installed, as Axios and its ecosystem run on it, allowing you to use JavaScript for server-side tasks or build tools.

Once Node.js is set up, navigate to your project directory in your terminal or command prompt and install Axios and Axios Retry using npm (Node Package Manager):

npm

If you hit any snags with npm, double-check your Node.js installation. Otherwise, you should be good to go.

Now, let's implement a basic retry setup:

const axios = require('axios');
// Note the .default when requiring axios-retry with CommonJS
const axiosRetry = require('axios-retry').default;

// Get the Axios instance and apply retry logic
const client = axios.create();

axiosRetry(client, {
  retries: 3, // Attempt the request up to 3 times if it fails
  retryCondition: (error) => {
    // Retry on network errors or specific server errors (e.g., 500, 502, 503, 504)
    return axiosRetry.isNetworkOrIdempotentRequestError(error) ||
      error.response?.status === 500 ||
      error.response?.status === 502 ||
      error.response?.status === 503 ||
      error.response?.status === 504;
  },
  retryDelay: (retryCount) => {
    // Add a small delay between retries
    return retryCount * 100; // 100ms, 200ms, 300ms
  }
});

// Make a request using the configured client
client.get('https://api.example.com/resource')
  .then(response => {
    console.log('Success:', response.data);
  })
  .catch(error => {
    // This catch block runs only if all retries fail
    console.error('Request failed after multiple retries:', error.message);
    if (error.response) {
      console.error('Status:', error.response.status);
    }
  });

Let's break this down:

1. We import `axios` and `axios-retry`. We use `require` here, common in Node.js projects (using `import` requires specific project configurations). Stick to one style (`require` or `import`) throughout your project.

2. We create an Axios instance using `axios.create()`. This is good practice as it allows applying configurations like retries to this specific instance without affecting the global Axios object.

3. We call `axiosRetry()`, passing our Axios instance (`client`) and a configuration object.

4. Inside the config object:

   • `retries: 3` specifies a maximum of 3 retry attempts after the initial failure.

   • `retryCondition` is a function that receives the error object. It returns `true` if a retry should be attempted, `false` otherwise. Here, we use a built-in helper `isNetworkOrIdempotentRequestError` and add checks for common server-side temporary errors (500, 502, 503, 504).

   • `retryDelay` is a function determining the wait time (in milliseconds) before the next retry. Here, we implement a simple linear delay based on the retry attempt number.

5. Finally, we use our configured `client` instance to make a `GET` request. The `.then()` block executes on success, and the `.catch()` block executes only if the request fails even after all the specified retries.

Note: Axios Retry typically doesn't retry requests explicitly aborted by the client (like closing a browser tab), as retrying those usually isn't the desired behavior.

Smarter Delays: Using Exponential Backoff

Constantly retrying immediately might not be wise if a server is temporarily overloaded. An exponential backoff strategy increases the delay between each retry, giving the server more time to recover. Axios Retry has a built-in helper for this:

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

const client = axios.create();

axiosRetry(client, {
  retries: 4, // Try 4 times
  retryDelay: axiosRetry.exponentialDelay, // Use built-in exponential backoff
  retryCondition: axiosRetry.isNetworkOrIdempotentRequestError // Default retry condition
});

client.get('https://api.another-example.com/items')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries with exponential backoff:', error.message));

Here, we replaced our custom `retryDelay` function with `axiosRetry.exponentialDelay`. By default, this waits `(2^retryCount * 100) + (random jitter)` milliseconds. The small random variation helps prevent multiple clients from retrying in perfect sync (known as the "thundering herd" problem).

You can also define your own custom backoff logic if needed:

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

// Custom backoff: Start with 500ms, increase by 500ms each time
function customLinearBackoff(retryNumber) {
  console.log(`Attempt ${retryNumber}: Waiting ${retryNumber * 500}ms`);
  return retryNumber * 500;
}

const client = axios.create();

axiosRetry(client, {
  retries: 3,
  retryDelay: customLinearBackoff, // Use our custom function
  retryCondition: axiosRetry.isNetworkOrIdempotentRequestError
});

client.get('https://api.yetanother-example.com/status')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries with custom linear backoff:', error.message));

Fine-Tuning Retry Conditions

Axios Retry allows precise control over *when* a retry happens. Maybe you only want to retry on a very specific error, like a rate limit response (HTTP 429):

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

const client = axios.create();

axiosRetry(client, {
  retries: 2, // Only retry twice for rate limits
  retryDelay: axiosRetry.exponentialDelay, // Give the server time
  retryCondition: (error) => {
    // Only retry if the response status is 429 (Too Many Requests)
    return error.response?.status === 429;
  }
});

client.get('https://api.strict-limits.com/data')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries on 429:', error.message));

In this example, the `retryCondition` checks if the error has a response (`error.response?`) and if that response's status code is exactly 429. Any other error (like a 500 or a network timeout) will fail immediately without retries.

Axios Retry in Web Scraping and Automation

As mentioned earlier, robust error handling is critical for web scraping. When your scraper interacts with hundreds or thousands of pages, temporary network issues, server hiccups, or intermittent blocks are common. Axios Retry ensures these don't derail your entire scraping job.

While proxies, like the high-quality residential proxies offered by Evomi, are essential for avoiding IP-based blocking and accessing geo-restricted content, Axios Retry complements them by handling the unavoidable, transient network failures. Combining reliable, ethically sourced proxies with smart retry logic creates a much more resilient and successful data gathering setup. If you're setting up a new scraping project, consider leveraging a free trial to test how different proxy types and retry configurations perform under real-world conditions.

Wrapping Up

Axios Retry is a valuable addition to any developer's toolkit when working with Axios for HTTP requests. It provides a simple yet powerful way to handle transient network errors and server issues automatically, making your applications more resilient.

Its ease of configuration, support for different retry strategies like exponential backoff, and the ability to define custom logic make it incredibly flexible. Whether you're building a web application, a data scraping bot, or any system relying on external APIs, implementing Axios Retry can save you headaches and improve reliability significantly.