Build a Golang Web Scraper with Playwright & Proxies

Go pairs a clean, readable syntax with genuinely fast execution, which makes it a strong pick for web scraping projects. Often called Golang, it can match or outperform Python, Node.js, or Ruby on comparable scraping workloads. In this guide you'll build a Go scraper step by step using Playwright for Go and route traffic through Evomi's residential proxies so you can collect public data reliably and at scale.

What a Web Scraper Actually Does

A web scraper automates the data-collection tasks a person would otherwise do by hand in a browser. Typical, legitimate uses include tracking publicly listed competitor prices, aggregating news, monitoring public job boards, checking product availability, and analysing public reviews for sentiment research. Beyond reading data, a scraper can also interact with pages: clicking elements, submitting forms, navigating, and capturing screenshots for QA or record-keeping.

Whatever your use case, keep it to publicly accessible information, respect each site's Terms of Service and robots.txt, and avoid collecting personal data you don't have a lawful basis to process.

Why Go Works Well for Scraping

Go's case is simple: speed plus a low learning curve. Benchmarks frequently show it running faster than Python or Ruby for equivalent tasks, and the language stays readable as your codebase grows. Its built-in concurrency model (goroutines) is a natural fit for fetching many pages in parallel. The "best" language always depends on your project and your team's familiarity, but Go's combination of performance and simplicity earns it a place on the shortlist. If you prefer another ecosystem, we have parallel walkthroughs for Rust and Ruby.

Static HTML vs. Full Browser Rendering

The right tool depends on how the target site delivers its content. If the data lives in the initial HTML, a lightweight framework like Gocolly is fast and easy to work with. But many modern sites render content with JavaScript after the page loads, and Gocolly focuses on static HTML — so you'd have to reverse-engineer network requests manually.

That's where a headless browser earns its keep. Playwright for Go is a community-maintained port of Microsoft's Playwright automation library. It gives you a Go API to drive Chromium, Firefox, or WebKit — loading pages fully, running their JavaScript, and letting you read the rendered result exactly as a visitor would see it.

Combining a Headless Browser with Proxies

Scraping publicly available data is generally legal, but site operators still manage automated traffic to protect their infrastructure. A common trigger is volume: sending a large number of requests from a single IP address in a short window looks nothing like normal browsing and can lead to rate limiting or temporary blocks.

A real browser via Playwright already renders JavaScript and sends standard headers, so it behaves like an ordinary client. To keep request volume from any single address reasonable, you distribute traffic across an IP pool. Evomi's residential proxies route requests through addresses assigned to real devices worldwide, so your load is spread rather than concentrated. Evomi's pool is ethically sourced, Swiss-based, and starts at $0.49/GB for residential — a sustainable way to scale public-data collection responsibly.

Setting Up Your Go Environment

First, install Go. On macOS with Homebrew:

On Windows with Chocolatey:

For Linux or other systems, grab the right package from the official Go downloads page. Confirm the install:

To run a Go file (e.g. main.go):

You'll also want an editor. Visual Studio Code with the official Go extension is a solid, free choice.

Initializing Your Go Module

Go projects live inside modules, defined by a go.mod file that tracks dependencies. Move into your workspace (often $HOME/go), create a project directory such as myscraper, change into it, and run:

Swap evomi.com/myscraper for your own module path. This creates the go.mod file.

Installing Playwright for Go

Add the Playwright library:

go get

Playwright also needs browser binaries. Install them along with their dependencies:

go run github.com/playwright-community/playwright-go/cmd/playwright install --with-deps

Taking Your First Screenshot

Create main.go in your myscraper directory and add this code:

package main

import (
    "fmt" // Import fmt for printing output later
    "log"

    "github.com/playwright-community/playwright-go"
)

func main() {
    // Start Playwright
    pw, err := playwright.Run()
    if err != nil {
        log.Fatalf("Could not start Playwright: %v", err)
    }

    // Launch the Chromium browser (headless by default)
    // You could also use pw.Firefox.Launch() or pw.WebKit.Launch()
    browser, err := pw.Chromium.Launch()
    if err != nil {
        log.Fatalf("Could not launch browser: %v", err)
    }

    // Open a new page (tab)
    page, err := browser.NewPage()
    if err != nil {
        log.Fatalf("Could not create page: %v", err)
    }

    // Navigate to a target URL (e.g., Evomi's IP checker)
    targetURL := "https://geo.evomi.com/"
    fmt.Printf("Navigating to %s...\n", targetURL)
    if _, err = page.Goto(targetURL, playwright.PageGotoOptions{
        WaitUntil: playwright.WaitUntilStateNetworkidle, // Wait for network activity to settle
    }); err != nil {
        log.Fatalf("Could not navigate to %s: %v", targetURL, err)
    }
    fmt.Println("Navigation successful!")

    // Take a screenshot
    screenshotPath := "page_screenshot.png"
    fmt.Printf("Taking screenshot and saving to %s...\n", screenshotPath)
    if _, err = page.Screenshot(playwright.PageScreenshotOptions{
        Path: playwright.String(screenshotPath), // Specify the file path
    }); err != nil {
        log.Fatalf("Could not take screenshot: %v", err)
    }
    fmt.Println("Screenshot saved!")

    // Close the browser
    if err = browser.Close(); err != nil {
        log.Fatalf("Could not close browser: %v", err)
    }
    // Stop Playwright
    if err = pw.Stop(); err != nil {
        log.Fatalf("Could not stop Playwright: %v", err)
    }
    fmt.Println("Scraper finished successfully.")
}

What each part does:

• package main declares this as an executable package.

• import (...) pulls in logging, fmt for output, and Playwright.

• func main() is the program's entry point.

• pw, err := playwright.Run() starts Playwright. Go functions typically return a value and an error; := declares and initialises in one step.

• if err != nil { ... } is Go's standard error check after each fallible operation.

• pw.Chromium.Launch() starts a Chromium instance; browser.NewPage() opens a tab.

• page.Goto(...) navigates to the URL. WaitUntilStateNetworkidle is handy for pages that load content dynamically.

• page.Screenshot(...) captures the current view to a file. playwright.String() wraps string options.

• browser.Close() and pw.Stop() shut things down cleanly.

Run it:

On success you'll see console output and a page_screenshot.png file showing the loaded page.

Routing Traffic Through Evomi Proxies

To send your scraper's traffic through proxies, configure them at browser launch. Grab your credentials — endpoint, port, username, password — from your Evomi dashboard. Replace this line:

browser, err := pw.Chromium.Launch()

with the following, inserting your own Evomi details:

// Configure proxy settings (Replace with your actual Evomi credentials)
proxyServer := "rp.evomi.com:1000" // Example: Evomi Residential HTTP endpoint
proxyUsername := "YOUR_EVOMI_USERNAME"
proxyPassword := "YOUR_EVOMI_PASSWORD"

proxySettings := playwright.BrowserTypeLaunchOptionsProxy{
    Server:   playwright.String(proxyServer),
    Username: playwright.String(proxyUsername),
    Password: playwright.String(proxyPassword),
}

// Launch the browser with proxy settings
browser, err := pw.Chromium.Launch(playwright.BrowserTypeLaunchOptions{
    Proxy: &proxySettings, // Pass the proxy config as a pointer
})
if err != nil {
    log.Fatalf("Could not launch browser with proxy: %v", err)
}

Key points:

• The proxySettings struct holds the server address, username, and password. Replace the placeholders with your real values and use the correct endpoint and port for your proxy type (e.g. rp.evomi.com:1000 for Residential HTTP).

• Pass a pointer (&proxySettings) to the Proxy field of playwright.BrowserTypeLaunchOptions when calling Launch().

Run go run main.go again. If your target was an IP-checking page such as https://geo.evomi.com/ or https://ipv4.icanhazip.com/, the new screenshot should show the proxy's IP rather than your own. To sanity-check any proxy before a run, Evomi's free proxy tester is handy. Playwright also exposes many other launch options — user agent, viewport, geolocation — plus screenshot options like full-page and per-element captures.

Extracting Data from Elements

Most scraping comes down to reading text or attributes from specific HTML elements. Playwright selects elements with CSS selectors (or XPath) and lets you pull their properties. Here we grab the main heading from the Playwright docs. Insert this before browser.Close():

// Navigate to Playwright's site for data extraction example
dataURL := "https://playwright.dev/docs/intro"
fmt.Printf("Navigating to %s for data extraction...\n", dataURL)
if _, err = page.Goto(dataURL, playwright.PageGotoOptions{
    WaitUntil: playwright.WaitUntilStateNetworkidle,
}); err != nil {
    log.Fatalf("Could not navigate to %s: %v", dataURL, err)
}

// Select the main heading element using its CSS class
headingSelector := "h1" // Simple selector for the main heading
fmt.Printf("Selecting element with selector: '%s'\n", headingSelector)
headingElement, err := page.QuerySelector(headingSelector)
if err != nil {
    log.Fatalf("Could not find element with selector '%s': %v", headingSelector, err)
}
if headingElement == nil {
    log.Fatalf("Element with selector '%s' not found.", headingSelector)
}

// Extract the text content
textContent, err := headingElement.TextContent()
if err != nil {
    log.Fatalf("Could not get text content: %v", err)
}

// Print the extracted text
fmt.Printf("Extracted Heading Text: %s\n", textContent)

// Remember to add back the browser closing and Playwright stopping code:
// if err = browser.Close(); err != nil { ... }
// if err = pw.Stop(); err != nil { ... }

This navigates to a page, uses page.QuerySelector() to find the first element matching the h1 selector, reads its text with TextContent(), and prints it. For multiple matches, use page.QuerySelectorAll(). From there you can store the value in variables, write it to a file, or push it to a database.

Interacting with Pages: Clicks and Form Input

Playwright can simulate almost any user action — clicking, typing, scrolling, and keyboard input. As with extraction, you select an element and then call a method like Click() or Fill(). This example clicks the search button on the Playwright site and types into the search box. Add it before the browser-closing calls:

// Example: Clicking the search button and typing into the input
// Selector for the search button
searchButtonSelector := ".DocSearch-Button"
fmt.Printf("Clicking element: '%s'\n", searchButtonSelector)
searchButton, err := page.QuerySelector(searchButtonSelector)
if err != nil {
    log.Fatalf("Error finding search button: %v", err)
}
if searchButton == nil {
    log.Fatalf("Search button element '%s' not found", searchButtonSelector)
}

// Click the button
if err = searchButton.Click(); err != nil {
    log.Fatalf("Error clicking search button: %v", err)
}
fmt.Println("Search button clicked.")

// Short pause to allow search modal to appear (adjust as needed)
page.WaitForTimeout(500) // Wait 500 milliseconds

// Selector for the search input field that appears
searchInputSelector := "#docsearch-input"
fmt.Printf("Typing into element: '%s'\n", searchInputSelector)
searchInput, err := page.QuerySelector(searchInputSelector)
if err != nil {
    log.Fatalf("Error finding search input: %v", err)
}
if searchInput == nil {
    log.Fatalf("Search input element '%s' not found", searchInputSelector)
}

// Fill the input field
searchText := "page object model"
if err = searchInput.Fill(searchText); err != nil {
    log.Fatalf("Error filling search input: %v", err)
}
fmt.Printf("Filled search input with: '%s'\n", searchText)

// Take a final screenshot to see the result
finalScreenshotPath := "interaction_screenshot.png"
fmt.Printf("Taking final screenshot: %s\n", finalScreenshotPath)
if _, err = page.Screenshot(playwright.PageScreenshotOptions{
    Path: playwright.String(finalScreenshotPath),
}); err != nil {
    log.Printf("Warning: could not take final screenshot: %v", err)
}

The flow selects the search button, clicks it, waits briefly for the modal to render, fills the input, and captures a screenshot. Adjust the timeout and selectors to match whatever page you're working with — CSS selectors are the main thing that changes from site to site.

Scaling Up Responsibly

With the building blocks above — navigation, data extraction, interaction, and proxy support — you have a complete foundation for a Go scraper. As you grow, a few habits keep your project healthy and courteous:

• Add delays and modest concurrency limits so you don't overload the target server.

• Handle errors and retries gracefully rather than hammering a page that failed once.

• Cache results you've already collected instead of re-fetching them.

• Match the proxy type to the job: datacenter proxies (from $0.30/GB) for speed on tolerant sites, residential for a real-user footprint.

If you'd rather skip managing browser binaries and infrastructure entirely, Evomi's Scraping Browser gives you a managed cloud Chromium endpoint (wss://browser.evomi.com) that's Playwright- and Puppeteer-compatible. For scraping in other stacks, our guides on Puppeteer with Node.js and Rust with Selenium cover similar ground.