Scraping GitHub with Python and Proxies: A Practical Guide

Nathan Reynolds

Scraping Techniques

GitHub is one of the richest sources of public technical data on the web: repository metadata, star counts, language trends, dependency graphs, and the READMEs that describe millions of open-source projects. If you want to track which Python projects are gaining traction, benchmark documentation quality, or build a research dataset of public repos, a small Python scraper gets you a long way. This guide walks through scraping public GitHub pages with Requests and Beautiful Soup, then shows how rotating proxies keep larger crawls fast and stable — all within GitHub's terms and rate limits.

One thing up front: GitHub publishes a first-class REST API. For most structured data (repos, stars, contributors, README contents), the API is cleaner, faster, and explicitly supported. Scraping HTML makes sense when you need something the API doesn't expose the way you want, such as the exact layout of the public trending page. Use the API where you can, and scrape public pages politely where you can't.

Why GitHub is friendly to scrape

A lot of GitHub renders as plain, server-side HTML. The core information on a repository or trending page isn't hidden behind heavy client-side JavaScript, so you rarely need a full browser automation stack like Selenium or Playwright for common tasks. That means the workflow stays simple:

  • Fetch the raw HTML of a public page with an HTTP client.

  • Parse that HTML to pull out the specific fields you care about.

We're using Python here, but the same two-step pattern applies in almost any language. If you're brand new to this, our Beautiful Soup and proxy guide covers the fundamentals before you dig into GitHub specifically.

Choosing your Python toolkit

For scraping GitHub, the two workhorses are Requests and Beautiful Soup. Requests retrieves a page's HTML source, acting like a stripped-down browser. Beautiful Soup then parses that HTML into a navigable tree so you can find and extract elements by tag, class, or attribute. Learn more about the technique itself on Wikipedia's web scraping overview.

Setting up your environment

Make sure you have Python installed — you can grab it from the official Python website. Then install the two libraries with pip:

Fetching the trending page

Our target is GitHub's trending Python repositories. The first job is to download the page HTML. It's worth sending a realistic User-Agent header — GitHub, like most sites, treats requests with no identifying header differently, and a descriptive agent is simply good manners:

import requests

# Define the target URL
target_url = 'https://github.com/trending/python'

# A descriptive User-Agent is polite and reduces friction
headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; ResearchScraper/1.0)'
}

# Send an HTTP GET request
page_response = requests.get(target_url, headers=headers)

# Check if the request was successful (Status code 200)
if page_response.status_code == 200:
    html_content = page_response.text
    # Proceed with parsing...
else:
    print(f"Failed to retrieve page. Status code: {page_response.status_code}")
    # Handle error appropriately

On a successful request, the full HTML source lives in html_content via the response object's .text attribute.

Parsing with Beautiful Soup

With the HTML in hand, create a Beautiful Soup object to parse it:

from bs4 import BeautifulSoup

# Assuming html_content holds the fetched HTML
soup_parser = BeautifulSoup(html_content, 'html.parser')

Each repository on the trending page is wrapped in an article element with the class Box-row. Grab all of them at once:

# Find all article elements, likely containing repo info
repository_elements = soup_parser.find_all('article', class_='Box-row')

Now loop through those elements and extract the repository name, star count, and a direct link:

trending_repos_data = []

for repo_element in repository_elements:
    try:
        # Extract Name (often in an h2 tag)
        name_element = repo_element.find('h2', class_='h3')
        # The name structure might be 'user / repo_name', so we split and clean
        full_name = name_element.a.text.strip().replace('\n', '').replace(' ', '')
        repo_name = full_name.split('/')[-1]

        # Extract Star Count (look for the star icon and its sibling/parent text)
        star_link = repo_element.find('a', href=lambda href: href and '/stargazers' in href)
        star_count_text = star_link.text.strip() if star_link else 'N/A'

        # Extract Link (get the href from the link in the h2)
        link_path = name_element.a['href']
        repo_link = 'https://github.com' + link_path

        repo_info = {
            'name': repo_name,
            'stars': star_count_text,
            'link': repo_link
        }
        trending_repos_data.append(repo_info)

    except AttributeError:
        # Handle cases where an element might be missing (e.g., sponsored repo without stars)
        print("Skipping an element due to missing attributes.")
        continue

The extraction logic in plain terms:

  • name: finds the h2 holding the repo name, reads the link text inside it, strips whitespace and newlines, and splits on / to keep just the repository name.

  • stars: locates the link whose href contains /stargazers and reads its text, with a fallback for when it isn't present.

  • link: pulls the href from the h2's anchor and prefixes the base GitHub URL.

A word on fragile selectors: GitHub occasionally changes class names and markup. If your scraper suddenly returns empty results, re-inspect the page and update the selectors. Wrapping extraction in try/except (as above) keeps one malformed row from crashing the whole run.

Finally, print the collected data:

import json

# Pretty print the JSON output
print(json.dumps(trending_repos_data, indent=2))

The output resembles a list of dictionaries (actual repos will vary):

[
  {
    "name": "some-cool-project",
    "stars": "12.3k",
    "link": "https://github.com/user/some-cool-project"
  },
  {
    "name": "another-trending-repo",
    "stars": "5,870",
    "link": "https://github.com/another-user/another-trending-repo"
  }
]

Here's the complete basic scraper in one piece:

import requests
from bs4 import BeautifulSoup
import json

# Define the target URL
target_url = 'https://github.com/trending/python'

headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; ResearchScraper/1.0)'
}

# Send an HTTP GET request
page_response = requests.get(target_url, headers=headers)

trending_repos_data = []

# Check if the request was successful
if page_response.status_code == 200:
    html_content = page_response.text

    # Parse the HTML
    soup_parser = BeautifulSoup(html_content, 'html.parser')

    # Find all article elements containing repo info
    repository_elements = soup_parser.find_all('article', class_='Box-row')

    for repo_element in repository_elements:
        try:
            # Extract Name
            name_element = repo_element.find('h2', class_='h3')
            full_name = name_element.a.text.strip().replace('\n', '').replace(' ', '')
            repo_name = full_name.split('/')[-1]

            # Extract Star Count
            star_link = repo_element.find('a', href=lambda href: href and '/stargazers' in href)
            star_count_text = star_link.text.strip() if star_link else 'N/A'

            # Extract Link
            link_path = name_element.a['href']
            repo_link = 'https://github.com' + link_path

            repo_info = {
                'name': repo_name,
                'stars': star_count_text,
                'link': repo_link
            }
            trending_repos_data.append(repo_info)

        except AttributeError:
            print("Skipping an element due to missing attributes.")
            continue
else:
    print(f"Failed to retrieve page. Status code: {page_response.status_code}")

# Print the collected data
print(json.dumps(trending_repos_data, indent=2))

Crawling deeper: reading repository READMEs

The trending list is just an entry point. Because we captured a link for each repository, we can crawl further — visiting each one and pulling additional details like the README content.

To do that, iterate over the links, fetch each page, and extract the README. But firing many requests in quick succession from a single IP will run straight into GitHub's rate limits. The simplest polite mitigation is to pause between requests using Python's built-in time module:

import time

Then extend the loop to fetch and parse each repository page:

# Assuming 'trending_repos_data' contains the list of dicts with 'link' keys
for repo in trending_repos_data:
    repo_url = repo['link']
    try:
        print(f"Fetching README for: {repo_url}")
        repo_page_response = requests.get(repo_url, headers=headers)

        if repo_page_response.status_code == 200:
            repo_html = repo_page_response.text
            repo_soup = BeautifulSoup(repo_html, 'html.parser')

            # Find the README section (often an element with id="readme")
            readme_element = repo_soup.find('div', id='readme')

            if readme_element:
                readme_text = readme_element.get_text(separator='\n', strip=True)
                # Store or process the readme_text
                print(f"--- README Start ({repo['name']}) ---")
                print(readme_text[:500] + "...")  # Print first 500 chars
                print(f"--- README End ({repo['name']}) ---\n")
            else:
                print(f"README section not found for {repo['name']}.")
        else:
            print(f"Failed to fetch {repo_url}. Status: {repo_page_response.status_code}")

        # IMPORTANT: Pause to be a good citizen and respect rate limits
        time.sleep(3)  # Wait 3 seconds before the next request

    except requests.exceptions.RequestException as e:
        print(f"Request error for {repo_url}: {e}")
        time.sleep(5)  # Longer pause on error
    except Exception as e:
        print(f"An error occurred while processing {repo['name']}: {e}")
        continue  # Move to the next repo if parsing fails

This visits each repository URL, finds the element that usually holds the README (a div with id="readme"), extracts its text, and prints a snippet. The time.sleep(3) is the important bit: it spaces out requests so you don't hammer the server. The trade-off is speed — a serial crawl with three-second gaps gets slow fast. For anything beyond a handful of repos, rotating IPs across a proxy pool lets you spread the load and keep throughput reasonable.

Scaling up with rotating proxies

A proxy is an intermediary that forwards your request to GitHub through a different IP address. With a rotating pool, each request (or small batch) can leave from a fresh IP, which distributes traffic instead of concentrating it on one address. That's what makes larger public-data collection both faster and more stable — you're staying comfortably within per-IP rate windows rather than pushing one address to its limit.

Why residential proxies? Residential IPs are addresses assigned by ISPs to real consumer connections, so traffic through them looks like ordinary user activity from diverse locations. Compared with datacenter ranges, that tends to mean fewer friction points on consumer-facing sites. Evomi's residential pool is ethically sourced and Swiss-based, and residential plans start at $0.49/GB with a free trial if you want to test your workflow first.

Wiring Evomi proxies into Requests is a small addition. You need your proxy credentials and the endpoint. For Evomi residential proxies the format looks like this:

# Replace with your actual Evomi username, password, and desired settings
# Example using HTTP endpoint for residential proxies
proxy_user = 'your_evomi_username'
proxy_pass = 'your_evomi_password'
proxy_host = 'rp.evomi.com'
proxy_port_http = 1000
proxy_port_https = 1001

evomi_proxies = {
    'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_http}',
    'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_https}',
}

# Optional: If you need SOCKS5 (check your plan/needs)
# proxy_port_socks5 = 1002
# evomi_proxies = {
#     'http': f'socks5h://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_socks5}',
#     'https': f'socks5h://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_socks5}',
# }

Now pass evomi_proxies into each requests.get call in the loop. Because requests are spread across many IPs, you can shorten or drop the fixed delay — though I still recommend a small, randomized pause to stay considerate:

# Loop through repos, fetching READMEs using proxies
for repo in trending_repos_data:
    repo_url = repo['link']
    try:
        print(f"Fetching README for: {repo_url} via proxy")
        # Pass the proxies dictionary to the get request
        repo_page_response = requests.get(
            repo_url, headers=headers, proxies=evomi_proxies, timeout=15
        )

        if repo_page_response.status_code == 200:
            repo_html = repo_page_response.text
            repo_soup = BeautifulSoup(repo_html, 'html.parser')
            readme_element = repo_soup.find('div', id='readme')

            if readme_element:
                readme_text = readme_element.get_text(separator='\n', strip=True)
                print(f"--- README Start ({repo['name']}) ---")
                print(readme_text[:500] + "...")
                print(f"--- README End ({repo['name']}) ---\n")
            else:
                print(f"README section not found for {repo['name']}.")
        else:
            print(f"Failed to fetch {repo_url}. Status: {repo_page_response.status_code}")

    except requests.exceptions.ProxyError as e:
        print(f"Proxy error for {repo_url}: {e}")
    except requests.exceptions.Timeout:
        print(f"Request timed out for {repo_url}")
    except requests.exceptions.RequestException as e:
        print(f"Request error for {repo_url}: {e}")
    except Exception as e:
        print(f"An error occurred while processing {repo['name']}: {e}")
        continue

Notice the explicit handling for ProxyError and Timeout — with a live proxy pool, individual connections occasionally hiccup, and you want those cases logged and skipped rather than fatal.

The full proxy-powered crawler

Here's everything assembled: scrape the trending page, then crawl each repository's README through the proxy pool.

import requests
from bs4 import BeautifulSoup
import json

headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; ResearchScraper/1.0)'
}

# --- Initial Scrape of Trending Page ---
target_url = 'https://github.com/trending/python'
page_response = requests.get(target_url, headers=headers)

trending_repos_data = []

if page_response.status_code == 200:
    html_content = page_response.text
    soup_parser = BeautifulSoup(html_content, 'html.parser')
    repository_elements = soup_parser.find_all('article', class_='Box-row')

    for repo_element in repository_elements:
        try:
            name_element = repo_element.find('h2', class_='h3')
            full_name = name_element.a.text.strip().replace('\n', '').replace(' ', '')
            repo_name = full_name.split('/')[-1]

            star_link = repo_element.find('a', href=lambda href: href and '/stargazers' in href)
            star_count_text = star_link.text.strip() if star_link else 'N/A'

            link_path = name_element.a['href']
            repo_link = 'https://github.com' + link_path

            trending_repos_data.append({
                'name': repo_name,
                'stars': star_count_text,
                'link': repo_link
            })
        except AttributeError:
            print("Skipping an element during initial scrape.")
            continue
else:
    print(f"Failed to retrieve trending page. Status code: {page_response.status_code}")
    exit()  # Exit if the initial scrape failed

# --- Configure Evomi Proxies ---
proxy_user = 'your_evomi_username'  # Replace with your details
proxy_pass = 'your_evomi_password'  # Replace with your details
proxy_host = 'rp.evomi.com'
proxy_port_http = 1000
proxy_port_https = 1001

evomi_proxies = {
    'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_http}',
    'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_https}',
}

# --- Crawl Individual Repo READMEs using Proxies ---
all_readme_data = {}  # Store READMEs keyed by repo name

print(f"\n--- Starting README Crawl for {len(trending_repos_data)} repositories ---")

for repo in trending_repos_data:
    repo_url = repo['link']
    repo_name = repo['name']
    try:
        print(f"Fetching README for: {repo_name} ({repo_url})")
        repo_page_response = requests.get(
            repo_url, headers=headers, proxies=evomi_proxies, timeout=20
        )

        if repo_page_response.status_code == 200:
            repo_soup = BeautifulSoup(repo_page_response.text, 'html.parser')
            readme_element = repo_soup.find('div', id='readme')
            if readme_element:
                all_readme_data[repo_name] = readme_element.get_text(separator='\n', strip=True)
            else:
                print(f"README section not found for {repo_name}.")
        else:
            print(f"Failed to fetch {repo_url}. Status: {repo_page_response.status_code}")

    except requests.exceptions.ProxyError as e:
        print(f"Proxy error for {repo_url}: {e}")
    except requests.exceptions.Timeout:
        print(f"Request timed out for {repo_url}")
    except requests.exceptions.RequestException as e:
        print(f"Request error for {repo_url}: {e}")
    except Exception as e:
        print(f"An error occurred while processing {repo_name}: {e}")
        continue

# Save results
with open('github_readmes.json', 'w', encoding='utf-8') as f:
    json.dump(all_readme_data, f, indent=2, ensure_ascii=False)

print(f"\nDone. Collected {len(all_readme_data)} READMEs.")

Staying on the right side of the rules

A few practices keep a scraper both effective and responsible:

  • Prefer the API for structured data. The GitHub REST API returns repos, stars, and README content cleanly with generous authenticated rate limits. Reach for HTML scraping only when the API doesn't cover your need.

  • Stick to public data. Everything here targets pages any signed-out visitor can see. Don't try to reach private repos or anything behind authentication you don't own. If you do need to work with account-gated pages you legitimately control, our guide on scraping login-only sites with Python covers that responsibly.

  • Respect rate limits and pace yourself. Proxies distribute load; they don't license you to flood a service. Keep concurrency reasonable and add small delays.

  • Cache what you fetch. Re-scraping the same page repeatedly is wasteful. Store results locally and only re-request when the data actually changes.

With Requests, Beautiful Soup, and a reliable rotating proxy pool, you can build a GitHub crawler that pulls public repository insights at a steady, sustainable pace — and you can validate your proxy setup any time with Evomi's free proxy tester before you scale a job up.

GitHub is one of the richest sources of public technical data on the web: repository metadata, star counts, language trends, dependency graphs, and the READMEs that describe millions of open-source projects. If you want to track which Python projects are gaining traction, benchmark documentation quality, or build a research dataset of public repos, a small Python scraper gets you a long way. This guide walks through scraping public GitHub pages with Requests and Beautiful Soup, then shows how rotating proxies keep larger crawls fast and stable — all within GitHub's terms and rate limits.

One thing up front: GitHub publishes a first-class REST API. For most structured data (repos, stars, contributors, README contents), the API is cleaner, faster, and explicitly supported. Scraping HTML makes sense when you need something the API doesn't expose the way you want, such as the exact layout of the public trending page. Use the API where you can, and scrape public pages politely where you can't.

Why GitHub is friendly to scrape

A lot of GitHub renders as plain, server-side HTML. The core information on a repository or trending page isn't hidden behind heavy client-side JavaScript, so you rarely need a full browser automation stack like Selenium or Playwright for common tasks. That means the workflow stays simple:

  • Fetch the raw HTML of a public page with an HTTP client.

  • Parse that HTML to pull out the specific fields you care about.

We're using Python here, but the same two-step pattern applies in almost any language. If you're brand new to this, our Beautiful Soup and proxy guide covers the fundamentals before you dig into GitHub specifically.

Choosing your Python toolkit

For scraping GitHub, the two workhorses are Requests and Beautiful Soup. Requests retrieves a page's HTML source, acting like a stripped-down browser. Beautiful Soup then parses that HTML into a navigable tree so you can find and extract elements by tag, class, or attribute. Learn more about the technique itself on Wikipedia's web scraping overview.

Setting up your environment

Make sure you have Python installed — you can grab it from the official Python website. Then install the two libraries with pip:

Fetching the trending page

Our target is GitHub's trending Python repositories. The first job is to download the page HTML. It's worth sending a realistic User-Agent header — GitHub, like most sites, treats requests with no identifying header differently, and a descriptive agent is simply good manners:

import requests

# Define the target URL
target_url = 'https://github.com/trending/python'

# A descriptive User-Agent is polite and reduces friction
headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; ResearchScraper/1.0)'
}

# Send an HTTP GET request
page_response = requests.get(target_url, headers=headers)

# Check if the request was successful (Status code 200)
if page_response.status_code == 200:
    html_content = page_response.text
    # Proceed with parsing...
else:
    print(f"Failed to retrieve page. Status code: {page_response.status_code}")
    # Handle error appropriately

On a successful request, the full HTML source lives in html_content via the response object's .text attribute.

Parsing with Beautiful Soup

With the HTML in hand, create a Beautiful Soup object to parse it:

from bs4 import BeautifulSoup

# Assuming html_content holds the fetched HTML
soup_parser = BeautifulSoup(html_content, 'html.parser')

Each repository on the trending page is wrapped in an article element with the class Box-row. Grab all of them at once:

# Find all article elements, likely containing repo info
repository_elements = soup_parser.find_all('article', class_='Box-row')

Now loop through those elements and extract the repository name, star count, and a direct link:

trending_repos_data = []

for repo_element in repository_elements:
    try:
        # Extract Name (often in an h2 tag)
        name_element = repo_element.find('h2', class_='h3')
        # The name structure might be 'user / repo_name', so we split and clean
        full_name = name_element.a.text.strip().replace('\n', '').replace(' ', '')
        repo_name = full_name.split('/')[-1]

        # Extract Star Count (look for the star icon and its sibling/parent text)
        star_link = repo_element.find('a', href=lambda href: href and '/stargazers' in href)
        star_count_text = star_link.text.strip() if star_link else 'N/A'

        # Extract Link (get the href from the link in the h2)
        link_path = name_element.a['href']
        repo_link = 'https://github.com' + link_path

        repo_info = {
            'name': repo_name,
            'stars': star_count_text,
            'link': repo_link
        }
        trending_repos_data.append(repo_info)

    except AttributeError:
        # Handle cases where an element might be missing (e.g., sponsored repo without stars)
        print("Skipping an element due to missing attributes.")
        continue

The extraction logic in plain terms:

  • name: finds the h2 holding the repo name, reads the link text inside it, strips whitespace and newlines, and splits on / to keep just the repository name.

  • stars: locates the link whose href contains /stargazers and reads its text, with a fallback for when it isn't present.

  • link: pulls the href from the h2's anchor and prefixes the base GitHub URL.

A word on fragile selectors: GitHub occasionally changes class names and markup. If your scraper suddenly returns empty results, re-inspect the page and update the selectors. Wrapping extraction in try/except (as above) keeps one malformed row from crashing the whole run.

Finally, print the collected data:

import json

# Pretty print the JSON output
print(json.dumps(trending_repos_data, indent=2))

The output resembles a list of dictionaries (actual repos will vary):

[
  {
    "name": "some-cool-project",
    "stars": "12.3k",
    "link": "https://github.com/user/some-cool-project"
  },
  {
    "name": "another-trending-repo",
    "stars": "5,870",
    "link": "https://github.com/another-user/another-trending-repo"
  }
]

Here's the complete basic scraper in one piece:

import requests
from bs4 import BeautifulSoup
import json

# Define the target URL
target_url = 'https://github.com/trending/python'

headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; ResearchScraper/1.0)'
}

# Send an HTTP GET request
page_response = requests.get(target_url, headers=headers)

trending_repos_data = []

# Check if the request was successful
if page_response.status_code == 200:
    html_content = page_response.text

    # Parse the HTML
    soup_parser = BeautifulSoup(html_content, 'html.parser')

    # Find all article elements containing repo info
    repository_elements = soup_parser.find_all('article', class_='Box-row')

    for repo_element in repository_elements:
        try:
            # Extract Name
            name_element = repo_element.find('h2', class_='h3')
            full_name = name_element.a.text.strip().replace('\n', '').replace(' ', '')
            repo_name = full_name.split('/')[-1]

            # Extract Star Count
            star_link = repo_element.find('a', href=lambda href: href and '/stargazers' in href)
            star_count_text = star_link.text.strip() if star_link else 'N/A'

            # Extract Link
            link_path = name_element.a['href']
            repo_link = 'https://github.com' + link_path

            repo_info = {
                'name': repo_name,
                'stars': star_count_text,
                'link': repo_link
            }
            trending_repos_data.append(repo_info)

        except AttributeError:
            print("Skipping an element due to missing attributes.")
            continue
else:
    print(f"Failed to retrieve page. Status code: {page_response.status_code}")

# Print the collected data
print(json.dumps(trending_repos_data, indent=2))

Crawling deeper: reading repository READMEs

The trending list is just an entry point. Because we captured a link for each repository, we can crawl further — visiting each one and pulling additional details like the README content.

To do that, iterate over the links, fetch each page, and extract the README. But firing many requests in quick succession from a single IP will run straight into GitHub's rate limits. The simplest polite mitigation is to pause between requests using Python's built-in time module:

import time

Then extend the loop to fetch and parse each repository page:

# Assuming 'trending_repos_data' contains the list of dicts with 'link' keys
for repo in trending_repos_data:
    repo_url = repo['link']
    try:
        print(f"Fetching README for: {repo_url}")
        repo_page_response = requests.get(repo_url, headers=headers)

        if repo_page_response.status_code == 200:
            repo_html = repo_page_response.text
            repo_soup = BeautifulSoup(repo_html, 'html.parser')

            # Find the README section (often an element with id="readme")
            readme_element = repo_soup.find('div', id='readme')

            if readme_element:
                readme_text = readme_element.get_text(separator='\n', strip=True)
                # Store or process the readme_text
                print(f"--- README Start ({repo['name']}) ---")
                print(readme_text[:500] + "...")  # Print first 500 chars
                print(f"--- README End ({repo['name']}) ---\n")
            else:
                print(f"README section not found for {repo['name']}.")
        else:
            print(f"Failed to fetch {repo_url}. Status: {repo_page_response.status_code}")

        # IMPORTANT: Pause to be a good citizen and respect rate limits
        time.sleep(3)  # Wait 3 seconds before the next request

    except requests.exceptions.RequestException as e:
        print(f"Request error for {repo_url}: {e}")
        time.sleep(5)  # Longer pause on error
    except Exception as e:
        print(f"An error occurred while processing {repo['name']}: {e}")
        continue  # Move to the next repo if parsing fails

This visits each repository URL, finds the element that usually holds the README (a div with id="readme"), extracts its text, and prints a snippet. The time.sleep(3) is the important bit: it spaces out requests so you don't hammer the server. The trade-off is speed — a serial crawl with three-second gaps gets slow fast. For anything beyond a handful of repos, rotating IPs across a proxy pool lets you spread the load and keep throughput reasonable.

Scaling up with rotating proxies

A proxy is an intermediary that forwards your request to GitHub through a different IP address. With a rotating pool, each request (or small batch) can leave from a fresh IP, which distributes traffic instead of concentrating it on one address. That's what makes larger public-data collection both faster and more stable — you're staying comfortably within per-IP rate windows rather than pushing one address to its limit.

Why residential proxies? Residential IPs are addresses assigned by ISPs to real consumer connections, so traffic through them looks like ordinary user activity from diverse locations. Compared with datacenter ranges, that tends to mean fewer friction points on consumer-facing sites. Evomi's residential pool is ethically sourced and Swiss-based, and residential plans start at $0.49/GB with a free trial if you want to test your workflow first.

Wiring Evomi proxies into Requests is a small addition. You need your proxy credentials and the endpoint. For Evomi residential proxies the format looks like this:

# Replace with your actual Evomi username, password, and desired settings
# Example using HTTP endpoint for residential proxies
proxy_user = 'your_evomi_username'
proxy_pass = 'your_evomi_password'
proxy_host = 'rp.evomi.com'
proxy_port_http = 1000
proxy_port_https = 1001

evomi_proxies = {
    'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_http}',
    'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_https}',
}

# Optional: If you need SOCKS5 (check your plan/needs)
# proxy_port_socks5 = 1002
# evomi_proxies = {
#     'http': f'socks5h://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_socks5}',
#     'https': f'socks5h://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_socks5}',
# }

Now pass evomi_proxies into each requests.get call in the loop. Because requests are spread across many IPs, you can shorten or drop the fixed delay — though I still recommend a small, randomized pause to stay considerate:

# Loop through repos, fetching READMEs using proxies
for repo in trending_repos_data:
    repo_url = repo['link']
    try:
        print(f"Fetching README for: {repo_url} via proxy")
        # Pass the proxies dictionary to the get request
        repo_page_response = requests.get(
            repo_url, headers=headers, proxies=evomi_proxies, timeout=15
        )

        if repo_page_response.status_code == 200:
            repo_html = repo_page_response.text
            repo_soup = BeautifulSoup(repo_html, 'html.parser')
            readme_element = repo_soup.find('div', id='readme')

            if readme_element:
                readme_text = readme_element.get_text(separator='\n', strip=True)
                print(f"--- README Start ({repo['name']}) ---")
                print(readme_text[:500] + "...")
                print(f"--- README End ({repo['name']}) ---\n")
            else:
                print(f"README section not found for {repo['name']}.")
        else:
            print(f"Failed to fetch {repo_url}. Status: {repo_page_response.status_code}")

    except requests.exceptions.ProxyError as e:
        print(f"Proxy error for {repo_url}: {e}")
    except requests.exceptions.Timeout:
        print(f"Request timed out for {repo_url}")
    except requests.exceptions.RequestException as e:
        print(f"Request error for {repo_url}: {e}")
    except Exception as e:
        print(f"An error occurred while processing {repo['name']}: {e}")
        continue

Notice the explicit handling for ProxyError and Timeout — with a live proxy pool, individual connections occasionally hiccup, and you want those cases logged and skipped rather than fatal.

The full proxy-powered crawler

Here's everything assembled: scrape the trending page, then crawl each repository's README through the proxy pool.

import requests
from bs4 import BeautifulSoup
import json

headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; ResearchScraper/1.0)'
}

# --- Initial Scrape of Trending Page ---
target_url = 'https://github.com/trending/python'
page_response = requests.get(target_url, headers=headers)

trending_repos_data = []

if page_response.status_code == 200:
    html_content = page_response.text
    soup_parser = BeautifulSoup(html_content, 'html.parser')
    repository_elements = soup_parser.find_all('article', class_='Box-row')

    for repo_element in repository_elements:
        try:
            name_element = repo_element.find('h2', class_='h3')
            full_name = name_element.a.text.strip().replace('\n', '').replace(' ', '')
            repo_name = full_name.split('/')[-1]

            star_link = repo_element.find('a', href=lambda href: href and '/stargazers' in href)
            star_count_text = star_link.text.strip() if star_link else 'N/A'

            link_path = name_element.a['href']
            repo_link = 'https://github.com' + link_path

            trending_repos_data.append({
                'name': repo_name,
                'stars': star_count_text,
                'link': repo_link
            })
        except AttributeError:
            print("Skipping an element during initial scrape.")
            continue
else:
    print(f"Failed to retrieve trending page. Status code: {page_response.status_code}")
    exit()  # Exit if the initial scrape failed

# --- Configure Evomi Proxies ---
proxy_user = 'your_evomi_username'  # Replace with your details
proxy_pass = 'your_evomi_password'  # Replace with your details
proxy_host = 'rp.evomi.com'
proxy_port_http = 1000
proxy_port_https = 1001

evomi_proxies = {
    'http': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_http}',
    'https': f'http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port_https}',
}

# --- Crawl Individual Repo READMEs using Proxies ---
all_readme_data = {}  # Store READMEs keyed by repo name

print(f"\n--- Starting README Crawl for {len(trending_repos_data)} repositories ---")

for repo in trending_repos_data:
    repo_url = repo['link']
    repo_name = repo['name']
    try:
        print(f"Fetching README for: {repo_name} ({repo_url})")
        repo_page_response = requests.get(
            repo_url, headers=headers, proxies=evomi_proxies, timeout=20
        )

        if repo_page_response.status_code == 200:
            repo_soup = BeautifulSoup(repo_page_response.text, 'html.parser')
            readme_element = repo_soup.find('div', id='readme')
            if readme_element:
                all_readme_data[repo_name] = readme_element.get_text(separator='\n', strip=True)
            else:
                print(f"README section not found for {repo_name}.")
        else:
            print(f"Failed to fetch {repo_url}. Status: {repo_page_response.status_code}")

    except requests.exceptions.ProxyError as e:
        print(f"Proxy error for {repo_url}: {e}")
    except requests.exceptions.Timeout:
        print(f"Request timed out for {repo_url}")
    except requests.exceptions.RequestException as e:
        print(f"Request error for {repo_url}: {e}")
    except Exception as e:
        print(f"An error occurred while processing {repo_name}: {e}")
        continue

# Save results
with open('github_readmes.json', 'w', encoding='utf-8') as f:
    json.dump(all_readme_data, f, indent=2, ensure_ascii=False)

print(f"\nDone. Collected {len(all_readme_data)} READMEs.")

Staying on the right side of the rules

A few practices keep a scraper both effective and responsible:

  • Prefer the API for structured data. The GitHub REST API returns repos, stars, and README content cleanly with generous authenticated rate limits. Reach for HTML scraping only when the API doesn't cover your need.

  • Stick to public data. Everything here targets pages any signed-out visitor can see. Don't try to reach private repos or anything behind authentication you don't own. If you do need to work with account-gated pages you legitimately control, our guide on scraping login-only sites with Python covers that responsibly.

  • Respect rate limits and pace yourself. Proxies distribute load; they don't license you to flood a service. Keep concurrency reasonable and add small delays.

  • Cache what you fetch. Re-scraping the same page repeatedly is wasteful. Store results locally and only re-request when the data actually changes.

With Requests, Beautiful Soup, and a reliable rotating proxy pool, you can build a GitHub crawler that pulls public repository insights at a steady, sustainable pace — and you can validate your proxy setup any time with Evomi's free proxy tester before you scale a job up.

Author

Nathan Reynolds

Web Scraping & Automation Specialist

About Author

Nathan specializes in web scraping techniques, automation tools, and data-driven decision-making. He helps businesses extract valuable insights from the web using ethical and efficient scraping methods powered by advanced proxies. His expertise covers overcoming anti-bot mechanisms, optimizing proxy rotation, and ensuring compliance with data privacy regulations.

Like this article? Share it.
You asked, we answer - Users questions:
Should I scrape GitHub or use its API?+
Why did my GitHub scraper suddenly stop returning results?+
Do I really need proxies to scrape GitHub?+
Which Evomi proxy type is best for scraping GitHub?+
How do I add Evomi proxies to a Python Requests script?+
Is scraping public GitHub pages allowed?+

In This Article

Read More Blogs

7 min

Why Is My Proxy So Slow? Diagnosing Latency, Bandwidth & Bottlenecks

Your scraper crawls and every request takes eight seconds, but is it the proxy, the route, or the target? This guide separates latency from throughput from success rate, shows you how to measure each with Python and curl, and walks through the real causes in order of likelihood so you fix the right thing.

7 min

How Residential Proxy Networks Actually Work: Peer-Sourced IPs Explained

A residential proxy is just someone's real home IP, so whose is it, and how did the provider get permission to route your traffic through it? This explainer walks through what makes an IP residential, how providers source the pool, the ethics spectrum from opt-in SDKs to malware, and how to tell a clean network from a sketchy one.

2 min Read

Residential vs. Datacenter Proxies: Best Choice?

In the digital age, online privacy and performance are paramount. But with so many proxy options available, how do you choose between residential and datacenter proxies? Discover which type offers superior anonymity and speed for your specific online needs.

Get Started with Swiss Quality Proxies

Try for Free

View Pricing

Explore Our Proxy Blog

Tips on residential proxies, web scraping & security

Frequently Asked Questions

Read if you have any questions regarding proxies

Read about Ethics

Our Ethical Standards in the residential proxy market

Get Started with Swiss Quality Proxies

Try for Free

View Pricing

Explore Our Proxy Blog

Tips on residential proxies, web scraping & security

Frequently Asked Questions

Read if you have any questions regarding proxies

Read about Ethics

Our Ethical Standards in the residential proxy market

Get Started with Swiss Quality Proxies

Try for Free

View Pricing

Explore Our Proxy Blog

Tips on residential proxies, web scraping & security

Frequently Asked Questions

Read if you have any questions regarding proxies

Read about Ethics

Our Ethical Standards in the residential proxy market