Performing cURL GET Requests: A Proxy-Focused Overview

Understanding cURL and the GET Method

At its core, a cURL GET request is a way to fetch data from a specific online location, typically identified by a URL. GET is one of the most fundamental HTTP methods; it's the action your browser performs every time you navigate to a webpage or click a link.

While browsers render this data visually, the curl command-line tool lets you interact directly with web resources, skipping the graphical interface to retrieve the raw data, like HTML source code, directly in your terminal.

Getting cURL Set Up

Good news! Many modern operating systems, including macOS and recent Windows versions (10 and later), come with cURL pre-installed. Most Linux distributions also include it or make it readily available. If you're on a Debian-based Linux system (like Ubuntu) and don't have it, installing is usually straightforward:

sudo apt update && sudo apt install curl

For older Windows versions, you might need to download and install cURL manually. Once installed, you can verify it's working by opening your terminal (or Command Prompt/PowerShell on Windows) and asking for its help menu:

curl --help

Seeing a list of options confirms cURL is ready to go.

Executing a Basic cURL GET Request

Making a GET request is cURL's default behavior. Simply typing curl followed by a URL will initiate a GET request to that address:

curl

This command fetches the content from the specified URL and displays it in your terminal. You can also explicitly state the method using the -G or --get flag, although it's often redundant for simple GET requests:

curl --get

Adding Query Parameters

Often, you'll need to send additional data along with your GET request, usually in the form of query parameters appended to the URL (like ?key1=value1&key2=value2). cURL handles this using the -d flag. You can use multiple -d flags, and cURL will correctly format them into the request URL when combined with -G or --get.

curl -G \
  -d "search=gadgets" \
  -d "category=electronics" \
  -d "instock=true"

Notice that the data for each parameter is enclosed in quotation marks. This command would effectively make a request to https://api.example.com/products?search=gadgets&category=electronics&instock=true.

You can also target multiple URLs in a single command; cURL will execute the GET request sequentially for each URL provided:

curl \
    -G \
    -d "info=ip"

Handling Redirects

Websites frequently use redirects (like HTTP 301 or 302). By default, cURL requests the exact URL you provide and stops. If that URL redirects elsewhere, cURL won't automatically follow it. To make cURL follow redirects, use the -L or --location flag:

curl \
  -G \
  -L

Without -L, requesting `google.com` might just return the redirect information, not the final page content.

Working with HTTP Headers

HTTP headers contain crucial metadata about the request and the expected response. Modifying request headers, particularly the User-Agent, can be vital.

Setting Custom Headers

Some servers block requests that use the default cURL user agent string, as it often signals automated access. Sending a header that mimics a standard web browser can help bypass simple checks. Use the -A flag to set the User-Agent:

curl \
  -G \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"

Using realistic user agent strings is a common practice in web scraping and testing. You can set other custom headers using the -H flag:

curl \
  -G \
  -H "X-Custom-Header: MyValue"

Viewing Response Headers

Sometimes you need to inspect the headers sent back by the server. Use the -i (or --include) flag to display the response headers along with the body content:

curl \
  -G \
  -i

If you *only* want to see the response headers and not the body, use -I (or --head). This effectively sends an HTTP HEAD request instead of a GET:

curl -I

Requesting Specific Data Formats

You can request data in a particular format using the Accept header. This is common when interacting with APIs that can return data in formats like JSON. Use the -H flag to specify the desired format:

curl -G \
  -H "Accept: application/json"

Keep in mind this only works if the server is configured to provide a response in the requested format. Many standard websites primarily serve HTML and might ignore this header, but it's essential for APIs.

Handling Authentication

Accessing protected resources often requires authentication. cURL supports several methods.

For resources protected by Basic HTTP Authentication (username and password), use the --user flag:

curl \
  -G \
  --user "myusername:mypassword123"

For token-based authentication (common in APIs), you typically include the token in a header, like Authorization: Bearer YOUR_TOKEN or a custom header:

curl \
  -G \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

You can also manage cookies. For instance, sending a cookie might be necessary to maintain a logged-in state. Use the --cookie flag:

curl \
  -G \
  --cookie "sessionid=abc123xyz"

Using cURL GET Requests Through Proxies

Proxies act as intermediaries for your requests, offering benefits like masking your IP address, bypassing geo-restrictions, or distributing requests to avoid rate limits. Using cURL with a proxy is simple using the -x or --proxy flag.

The syntax typically follows this pattern: -x protocol://[username:password@]proxy_host:proxy_port.

Here’s how you might use an Evomi residential proxy:

# HTTP Proxy Example (using Evomi residential endpoint)
curl -x http://USERNAME:PASSWORD@rp.evomi.com:1000 -A "Mozilla/5.0..." -L https://geo.evomi.com/json

# SOCKS5 Proxy Example (using Evomi residential endpoint)
curl --proxy socks5h://USERNAME:PASSWORD@rp.evomi.com:1002 -A "Mozilla/5.0..." -L

Replace USERNAME and PASSWORD with your actual Evomi credentials. The `socks5h` scheme ensures DNS resolution also happens through the proxy. Using reliable proxies, like Evomi's ethically sourced residential proxies, is crucial for tasks requiring anonymity or large-scale data access. We even offer a free trial if you want to test how seamlessly cURL integrates with our proxy network.

Wrapping Up

cURL is an incredibly versatile tool for interacting with web resources directly from the command line, and the GET method is its most frequent operation. Whether you're testing APIs, downloading files, or scraping web data, understanding how to craft cURL GET requests with the right parameters, headers, and potentially proxies is invaluable.

A good starting point often includes setting a realistic User-Agent:

curl \
  -G \
  -L \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"

We've only scratched the surface here. cURL boasts a vast array of options. To explore everything it can do, you can always consult its extensive help documentation:

curl --help

If you found this useful, you might also be interested in learning about performing cURL POST requests.