C# Web Scraping: HttpClient, HtmlAgilityPack and Playwright

C# is a good fit for web scraping. HttpClient is fast and asynchronous, HtmlAgilityPack and AngleSharp parse HTML well, and Playwright for .NET drives a real browser when a page builds its content with JavaScript. Since .NET 10 you don't even need a project file: one .cs file and dotnet run are enough.

This guide builds the same small scraper three ways, collecting every quote, author and tag from quotes.toscrape.com, a sandbox site made for scraping practice. It then adds a proxy to each version. Everything was run on the .NET 10 SDK (10.0.401) in October 2026. The outputs shown are from those runs.

Which C# scraping library should you use?

Start with plain HTTP and a parser whenever the data is in the HTML the server sends. It's an order of magnitude lighter than a browser. To check, view the page source (not the browser's element inspector) and search for a value you want. If it isn't there, you need a browser. For a Puppeteer-style approach, see our guide to PuppeteerSharp.

Set up a .NET 10 file-based app

Install the .NET 10 SDK and check it with dotnet --version. A file-based app declares its NuGet packages at the top of the file with #:package, and dotnet run restores, builds and runs it:

dotnet --version
# 10.0.401
dotnet run scrape.cs

One trap we hit: don't name the file after a package it references. A file called anglesharp.cs produces a project called anglesharp, which NuGet treats as depending on itself and refuses with error NU1108: Cycle detected. We renamed it to quotes-angle.cs and it ran. On older SDKs, create a console project with dotnet new console and add packages with dotnet add package instead. The code is the same.

Scrape a static page with HttpClient and HtmlAgilityPack

This is the complete scraper. It follows the "Next" link through all ten pages, waits a second between requests and writes a CSV:

#:package HtmlAgilityPack@1.13.0

using System.Globalization;
using System.Net;
using System.Text;
using HtmlAgilityPack;

var baseUri = new Uri("https://quotes.toscrape.com/");
using var http = CreateClient();

var quotes = new List<Quote>();
Uri? next = baseUri;

while (next is not null)
{
    var html = await http.GetStringAsync(next);
    var doc = new HtmlDocument();
    doc.LoadHtml(html);

    foreach (var card in doc.DocumentNode.SelectNodes("//div[@class='quote']") ?? Enumerable.Empty<HtmlNode>())
    {
        quotes.Add(new Quote(
            Text: HtmlEntity.DeEntitize(card.SelectSingleNode(".//span[@class='text']").InnerText.Trim()),
            Author: card.SelectSingleNode(".//small[@class='author']").InnerText.Trim(),
            Tags: string.Join(';', card.SelectNodes(".//a[@class='tag']")?.Select(t => t.InnerText) ?? [])));
    }

    var href = doc.DocumentNode.SelectSingleNode("//li[@class='next']/a")?.GetAttributeValue("href", null);
    next = href is null ? null : new Uri(baseUri, href);

    Console.WriteLine($"{quotes.Count} quotes so far");
    if (next is not null) await Task.Delay(TimeSpan.FromSeconds(1)); // one request a second
}

var csv = new StringBuilder("text,author,tags\n");
foreach (var q in quotes)
    csv.AppendLine(string.Join(',', Csv(q.Text), Csv(q.Author), Csv(q.Tags)));
await File.WriteAllTextAsync("quotes.csv", csv.ToString());
Console.WriteLine($"Saved {quotes.Count} quotes to quotes.csv");

static string Csv(string value) => $"\"{value.Replace("\"", "\"\"")}\"";

static HttpClient CreateClient()
{
    var handler = new SocketsHttpHandler
    {
        AutomaticDecompression = DecompressionMethods.All,
        PooledConnectionLifetime = TimeSpan.FromMinutes(5),
    };

    var server = Environment.GetEnvironmentVariable("PROXY_SERVER");
    if (!string.IsNullOrEmpty(server))
    {
        handler.Proxy = new WebProxy(server)
        {
            Credentials = new NetworkCredential(
                Environment.GetEnvironmentVariable("PROXY_USER"),
                Environment.GetEnvironmentVariable("PROXY_PASS")),
        };
        handler.UseProxy = true;
    }

    var client = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(30) };
    client.DefaultRequestHeaders.UserAgent.ParseAdd("QuotesResearchBot/1.0 (+mailto:you@example.com)");
    return client;
}

record Quote(string Text, string Author, string Tags);

$ dotnet run scrape.cs
10 quotes so far
...
100 quotes so far
Saved 100 quotes to quotes.csv

The parts worth copying into your own scrapers:

• One HttpClient for the whole run. Creating one per request leaves sockets behind and eventually exhausts them. PooledConnectionLifetime lets the long-lived client still pick up DNS changes.
• AutomaticDecompression accepts gzip and Brotli responses, which cuts transfer size without any extra code.
• XPath with SelectNodes and SelectSingleNode. SelectNodes returns null, not an empty list, when nothing matches. Hence the ?? Enumerable.Empty<HtmlNode>().
• HtmlEntity.DeEntitize turns ' and similar entities back into the characters a reader sees. InnerText leaves them encoded.
• Relative links resolved with new Uri(baseUri, href), so pagination works whatever form the href takes.
• An honest User-Agent with a contact address and a fixed delay. Site operators can see who is crawling and how to reach them, and one request a second keeps the load trivial.

The same scraper with AngleSharp

AngleSharp parses HTML the way a browser does and lets you query it with CSS selectors, which many developers find easier to read than XPath. Its default loader fetches the page for you:

#:package AngleSharp@1.8.4

using AngleSharp;

var context = BrowsingContext.New(Configuration.Default.WithDefaultLoader());
var document = await context.OpenAsync("https://quotes.toscrape.com/");

foreach (var card in document.QuerySelectorAll("div.quote"))
{
    var text = card.QuerySelector("span.text")?.TextContent.Trim();
    var author = card.QuerySelector("small.author")?.TextContent.Trim();
    var tags = card.QuerySelectorAll("a.tag").Select(t => t.TextContent);
    Console.WriteLine($"{author}: {text} [{string.Join(", ", tags)}]");
}

HtmlAgilityPack or AngleSharp? Both handle messy real-world HTML. Pick HtmlAgilityPack if you think in XPath or need to walk up the tree (ancestor::, following-sibling::). Pick AngleSharp if you think in CSS selectors or want a DOM that matches the browser's. For anything beyond a quick script, fetch with your own configured HttpClient and pass AngleSharp the HTML. Then timeouts, the User-Agent and proxies stay in one place.

Scrape JavaScript pages with Playwright for .NET

The /js/ version of the same site builds its quotes in the browser. Fetch it with HttpClient and the quote markup isn't there. Playwright runs a real Chromium, waits for the content and reads it:

#:package Microsoft.Playwright@1.63.0
#:property PublishAot=false

using Microsoft.Playwright;

// First run: dotnet run playwright.cs -- install
if (args is ["install"])
    return Microsoft.Playwright.Program.Main(["install", "chromium"]);

using var playwright = await Playwright.CreateAsync();

var launch = new BrowserTypeLaunchOptions { Headless = true };
var server = Environment.GetEnvironmentVariable("PROXY_SERVER");
if (!string.IsNullOrEmpty(server))
{
    launch.Proxy = new Proxy
    {
        Server = server,
        Username = Environment.GetEnvironmentVariable("PROXY_USER"),
        Password = Environment.GetEnvironmentVariable("PROXY_PASS"),
    };
}

await using var browser = await playwright.Chromium.LaunchAsync(launch);
var page = await browser.NewPageAsync();

// This version of the site builds its quotes with JavaScript, so a plain
// HttpClient request sees an empty page.
string? url = "https://quotes.toscrape.com/js/";
var total = 0;

while (url is not null)
{
    await page.GotoAsync(url);
    var cards = page.Locator("div.quote");
    await cards.First.WaitForAsync();

    foreach (var card in await cards.AllAsync())
    {
        var text = await card.Locator("span.text").InnerTextAsync();
        var author = await card.Locator("small.author").InnerTextAsync();
        Console.WriteLine($"{author}: {text}");
        total++;
    }

    var next = page.Locator("li.next a");
    url = await next.CountAsync() > 0
        ? new Uri(new Uri(page.Url), await next.GetAttributeAsync("href")).ToString()
        : null;
    await Task.Delay(1000);
}

Console.WriteLine($"{total} quotes");
return 0;

$ dotnet run playwright.cs -- install
$ dotnet run playwright.cs
...
Dr. Seuss: “A person's a person, no matter how small.”
100 quotes

Two notes on this file:

• #:property PublishAot=false is required. File-based apps default to Native AOT settings, which disable the reflection-based JSON serialization Playwright depends on. Without the line, the first launch fails with "Reflection-based serialization has been disabled for this application". In a regular .csproj project you don't need it.
• cards.First.WaitForAsync() waits until at least one quote exists before reading. Playwright's locators also wait on their own when you call InnerTextAsync, so you rarely need fixed sleeps. The one-second delay here is for politeness, not timing.

For a broader comparison of browser tools, see Playwright vs Puppeteer for web scraping, or our guide to scraping JavaScript sites with Puppeteer and Node.js if you work in both ecosystems.

Use a proxy with HttpClient

A proxy routes your requests through another IP address, typically to collect data as it appears in a particular country or to keep a large job from depending on one connection. In .NET it's a property of the handler, set once when you create the client:

using System.Net;

var proxy = new WebProxy(Environment.GetEnvironmentVariable("PROXY_SERVER"))
{
    Credentials = new NetworkCredential(
        Environment.GetEnvironmentVariable("PROXY_USER"),
        Environment.GetEnvironmentVariable("PROXY_PASS")),
};

using var http = new HttpClient(new SocketsHttpHandler { Proxy = proxy, UseProxy = true });

try
{
    var ip = await http.GetStringAsync(Environment.GetEnvironmentVariable("CHECK_URL") ?? "https://ip.evomi.com/s");
    Console.WriteLine($"Exit IP: {ip.Trim()}");
}
catch (HttpRequestException ex)
{
    Console.WriteLine($"Proxy request failed: {ex.StatusCode} {ex.Message}");
}

With Evomi's residential proxies, the server is http://rp.evomi.com:1000. Targeting options go at the end of the password, so your_password_country-DE asks for an exit in Germany:

export PROXY_SERVER=http://rp.evomi.com:1000
export PROXY_USER=your_username
export PROXY_PASS=your_password_country-DE
dotnet run proxy-check.cs
dotnet run scrape.cs   # the scraper above picks up the same variables

Use the HTTP port even for https:// sites. .NET opens a CONNECT tunnel through the proxy and the TLS session runs end to end between your program and the site. If you do use the HTTPS proxy port (1001), the hostname is rp.evomi-proxy.com, because that is the name its certificate covers.

We tested this code against a local proxy that requires a username and password. With the right credentials, the scraper collected all 100 quotes through it. With a wrong password, HttpClient throws before any page is fetched, and the status code tells you it was the proxy, not the site:

Proxy request failed: ProxyAuthenticationRequired The proxy tunnel request to proxy 'http://127.0.0.1:8899/' failed with status code '407'.

By default each request can leave from a different IP. For a sequence that has to keep one address, such as a login followed by page views, add a session to the password: your_password_session-abc123. To see where your exit is, look it up in the IP geolocation checker.

Use a proxy with Playwright for .NET

The Playwright scraper above already reads the same three variables and passes them to BrowserTypeLaunchOptions.Proxy, so every page in that browser goes through the proxy. To give different contexts different exits, set the proxy on each context instead:

var context = await browser.NewContextAsync(new BrowserNewContextOptions
{
    Proxy = new Proxy
    {
        Server = "http://rp.evomi.com:1000",
        Username = "your_username",
        Password = "your_password_country-US",
    },
});
var page = await context.NewPageAsync();

Through our local test proxy, the Playwright scraper collected all 100 quotes. Wrong credentials behaved differently from HttpClient, and it's worth knowing before you debug: there was no 407 error, only TimeoutException: Timeout 30000ms exceeded while navigating. When a proxied Playwright run times out on its very first page, check the proxy credentials before the selectors.

Use the proxy's HTTP endpoint with Playwright. Chromium can't authenticate to a SOCKS5 proxy, and Playwright refuses the combination at launch with "Browser does not support socks5 proxy authentication".

Scrape responsibly

• Prefer an official API or data export when the site offers one.
• Read the site's terms and its robots.txt before you start. Our guide to respecting robots.txt and crawl-delay shows how to apply it in code.
• Keep the request rate low and fixed, cache what you've already fetched, and stop if the site returns errors.
• Collect public data only, and handle any personal data in line with the privacy law that applies to you.

Troubleshooting

• error NU1108: Cycle detected. The file is named after a package it references. Rename it, e.g. to quotes-angle.cs.
• "Reflection-based serialization has been disabled". File-based apps default to AOT settings that Playwright can't run under. Add #:property PublishAot=false.
• SelectNodes returns null. The content is rendered by JavaScript, or the selector is wrong. Check the page source, and use Playwright if the data isn't there.
• Text shows ' or &. InnerText keeps HTML entities. Wrap it in HtmlEntity.DeEntitize.
• ProxyAuthenticationRequired (407). The proxy username or password is wrong. Check the credentials and any options appended to the password.
• Playwright times out on the first page behind a proxy. Usually proxy authentication. Test the same credentials with proxy-check.cs.
• "Browser does not support socks5 proxy authentication". SOCKS5 with credentials doesn't work in Chromium. Use the HTTP proxy endpoint.

FAQ

Is C# good for web scraping?

Yes. HttpClient handles high request volumes efficiently with async/await, HtmlAgilityPack and AngleSharp are mature parsers, and Playwright for .NET covers JavaScript-heavy sites with the same API as its Python and Node.js versions. Python has more scraping tutorials, but nothing in a typical scraping job is harder in C#.

What is the best C# library for web scraping?

For static pages, HttpClient with HtmlAgilityPack (XPath) or AngleSharp (CSS selectors). For pages that need JavaScript, Playwright for .NET. Most projects use a parser for the bulk of the pages and a browser only where it's needed.

Can C# scrape JavaScript-rendered websites?

Yes, with a browser automation library such as Playwright for .NET, PuppeteerSharp or Selenium. Before reaching for one, check the browser's network tab: many JavaScript pages load their data from a JSON endpoint that HttpClient can request directly.

How do I use a proxy with HttpClient?

Set Proxy on the SocketsHttpHandler (or HttpClientHandler) to a WebProxy with NetworkCredential, and create the HttpClient from that handler once. The proxy applies to every request that client sends, so use separate clients if you need separate proxy settings.

What is the best programming language for web scraping?

The one your team already uses. Python has the largest scraping ecosystem, JavaScript runs Puppeteer and Playwright natively, and C#, Go and Java are strong for large, long-running jobs. Our comparison of key languages for web scraping goes through the trade-offs.

Is web scraping legal?

Collecting publicly available data is lawful in many cases, but it depends on the site's terms, the kind of data (personal data has extra rules) and your jurisdiction. Check the terms, prefer official APIs, and get advice for anything commercial or sensitive.