Parsing NEXT_DATA and JSON-LD: The Clean Way to Extract Structured Data

You wrote a dozen CSS selectors to scrape a product page. Title, price, rating, SKU, availability, breadcrumb. They worked Monday. By Friday the site shipped a redesign and half of them returned None, because someone renamed a class from price__current to price-now. You patch the selectors, and a week later it happens again.

Meanwhile, the entire product object, every field you were scraping by hand, plus a dozen you didn't know existed, is sitting in a <script> tag the site ships on every single page load. You were parsing the painting when the blueprint was taped to the back of the canvas.

This post is about finding and reading that blueprint. It's almost always cleaner, richer, and more stable than scraping rendered HTML.

Why the Data Is Already There

Modern sites are server-rendered or hydrated. The server builds the page, but it also serializes the state the page was built from and embeds it so the client-side JavaScript can pick up where the server left off without re-fetching everything. That serialized state is plain JSON, and you can read it too.

A few common shapes:

• JSON-LD (<script type="application/ld+json">) — structured data in schema.org vocabulary, added for SEO and rich search results. Nearly universal on e-commerce, news, and recipe sites.

• __NEXT_DATA__ — Next.js dumps its page props into a script tag with id="__NEXT_DATA__".

• __NUXT__ — the Nuxt (Vue) equivalent.

• __APOLLO_STATE__, __PRELOADED_STATE__, window.__INITIAL_STATE__ — Apollo GraphQL cache, Redux store, and generic hydration blobs.

The rendered DOM is a projection of this data. The data is the source. Read the source.

JSON-LD: The Easiest Win

JSON-LD is the lowest-effort target because it's standardized. Look for one or more blocks like this in view-source:

<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Product","name":"Widget","sku":"W-100",
 "aggregateRating":{"@type":"AggregateRating","ratingValue":"4.6","reviewCount":"212"},
 "offers":{"@type":"Offer","price":"29.99","priceCurrency":"USD","availability":"https://schema.org/InStock"}}
</script>

Common @type values you'll hit: Product, Offer, Article, BreadcrumbList, Review, AggregateRating, Organization, Recipe.

Two things trip people up. First, there can be several blocks on one page, a Product, a BreadcrumbList, and an Organization, each in its own <script>. Second, sites sometimes wrap everything in a single block using @graph, a list of nodes. Your parser has to handle both, plus the case where the top-level value is itself a list.

Here's a parser that collects every block and flattens it into a flat list of typed nodes:

import json
from bs4 import BeautifulSoup

def extract_jsonld(html):
    soup = BeautifulSoup(html, "html.parser")
    nodes = []
    for tag in soup.find_all("script", type="application/ld+json"):
        if not tag.string:
            continue
        try:
            data = json.loads(tag.string)
        except json.JSONDecodeError:
            continue  # some sites ship malformed JSON-LD; skip it
        # Normalize: a block can be a dict, a list, or use @graph
        if isinstance(data, dict) and "@graph" in data:
            nodes.extend(data["@graph"])
        elif isinstance(data, list):
            nodes.extend(data)
        else:
            nodes.append(data)
    return nodes

def find_type(nodes, wanted):
    for n in nodes:
        t = n.get("@type")
        types = t if isinstance(t, list) else [t]
        if wanted in types:
            return n
    return None

Pulling out price and rating, defensively:

nodes = extract_jsonld(html)

product = find_type(nodes, "Product")
if product:
    offers = product.get("offers", {})
    # offers can be a single Offer or a list of them
    if isinstance(offers, list):
        offers = offers[0] if offers else {}
    rating = product.get("aggregateRating", {})

    print(product.get("name"))
    print(offers.get("price"), offers.get("priceCurrency"))
    print(rating.get("ratingValue"), rating.get("reviewCount"))

Note the @type can be a list of strings, offers can be a single object or an array, and any field can be missing. Code for all three from the start.

NEXT_DATA: The Full Page State

When JSON-LD is thin or absent, __NEXT_DATA__ is the next stop on a Next.js site. It's a single script tag, and unlike a hydration blob it's clean JSON you can hand straight to json.loads:

import json
from bs4 import BeautifulSoup

def extract_next_data(html):
    soup = BeautifulSoup(html, "html.parser")
    tag = soup.find("script", id="__NEXT_DATA__")
    if not tag or not tag.string:
        return None
    return json.loads(tag.string)

The shape is consistent at the top, the page-specific data almost always lives under props.pageProps:

data = extract_next_data(html)
page_props = data.get("props", {}).get("pageProps", {})

Below pageProps it's the wild west: every app names things differently, and the schema changes between deploys. Navigate defensively with .get() and never assume a deep path exists:

product = page_props.get("product") or page_props.get("initialData", {}).get("product")
if product:
    price = product.get("price", {}).get("amount")
    name = product.get("title")
    if price is None:
        # Don't fail silently — log it so you notice the day they rename the field
        print("WARN: price missing; pageProps keys =", list(page_props.keys()))

That WARN line is the whole game. Hardcoded deep paths break invisibly. A log line that fires on a missing field turns a silent data-quality bug into a one-line alert you can act on before your dataset rots.

Spotting the Other Hydration Blobs

Same idea, slightly messier packaging. To find them, open view-source and search for application/json and for the known keys:

• __NUXT__ — usually window.__NUXT__ = {...} or a <script> you have to slice the JSON out of. Look under data or state.

• __APOLLO_STATE__ — a normalized GraphQL cache. Keys look like Product:123 and reference each other by ID, so you may have to resolve references manually. Verbose but complete.

• __PRELOADED_STATE__ / window.__INITIAL_STATE__ — Redux/Vuex stores assigned to a global. Search for __INITIAL_STATE__ and you'll spot the assignment.

When the JSON is assigned to a window.* variable rather than sitting in a clean script tag, slice from the first { to the matching close. A small helper:

import re, json

def extract_window_json(html, var_name):
    # Matches: window.__INITIAL_STATE__ = { ... };
    m = re.search(re.escape(var_name) + r"\s*=\s*", html)
    if not m:
        return None
    start = html.index("{", m.end())
    depth = 0
    for i in range(start, len(html)):
        if html[i] == "{":
            depth += 1
        elif html[i] == "}":
            depth -= 1
            if depth == 0:
                return json.loads(html[start:i + 1])
    return None

state = extract_window_json(html, "window.__INITIAL_STATE__")

Use a regex to locate the assignment, then a brace-counting scan to grab a balanced object, then json.loads the result. Do not try to parse the JSON itself with regex, see the mistakes list.

When the Data Isn't in the HTML at All

Sometimes the page ships nearly empty and fills in via XHR after load. View-source shows a skeleton; the product never appears in the static HTML. In that case, the cleanest path is usually to call the JSON API directly.

Open your browser's Network tab, filter to Fetch/XHR, and reload. You'll see the request that returns the product as JSON, often something like https://example.com/api/v2/products/W-100. Copy it as cURL, translate to Python, and you're talking to the same endpoint the site uses, with no HTML parsing at all:

import requests

r = requests.get(
    "https://example.com/api/v2/products/W-100",
    headers={"Accept": "application/json",
             "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                           "AppleWebKit/537.36 (KHTML, like Gecko) "
                           "Chrome/148.0.0.0 Safari/537.36"},
)
product = r.json()

The caveat: some of these endpoints require an auth token, a session cookie, a CSRF header, or a signed parameter that the page's JavaScript generates. If a bare request returns 401/403, replay the exact headers the browser sent. Some of those values are short-lived, which is when a headless browser to mint a fresh session, and a clean residential pool, like Evomi's, for the requests themselves, starts to earn its keep. As always, respect the site's robots.txt and rate limits; hitting an internal API does not change the rules.

Why This Beats CSS Selectors

• It's structured and typed. A price is a number under a known key, not text you have to strip a currency symbol off of.

• It's stable-ish. Internal data keys change far less often than visual class names. A redesign rewrites the CSS; it rarely renames pageProps.product.price.

• It's often richer. The payload routinely contains inventory counts, variant SKUs, internal IDs, and review data that never get rendered on screen.

It is not magic, though. Schemas do drift between deploys, fields go missing, and a value you relied on can turn null. That's exactly why every snippet above leans on .get() and logs missing fields instead of trusting a deep path. Validate defensively and you'll know the moment something shifts.

Mistakes That Waste Time

• Assuming one JSON-LD block. Pages routinely ship three or four, plus @graph wrappers. Collect them all, then filter by @type.

• Hardcoding deep paths without .get(). data["props"]["pageProps"]["product"]["price"] throws a KeyError the day they restructure. Chain .get() and log misses.

• Ignoring the XHR API. If a field only loads after page render, it's not in the static HTML and no parser will find it. Check the Network tab before assuming the data isn't there.

• Regex-parsing JSON. Use a regex to find the blob's location, never to parse its contents. Always finish with json.loads. Hand-rolled regex JSON parsing breaks on nested quotes, escapes, and unicode.

Wrapping Up

Before you write another CSS selector, view-source the page and search for application/ld+json, __NEXT_DATA__, and application/json. The data you want is usually already there as clean JSON. Parse JSON-LD first (it's standardized), fall back to the hydration blob, and if the page is empty until XHR fires, call that API directly. Then wrap every access in .get() and log the missing fields, that one habit is what keeps the pipeline alive through the next redesign.