The scraping industry sells you a $90/month API subscription to fight anti-bot systems. Meanwhile, most modern e-commerce sites are built on Next.js — the framework that downloads 354 million times per month on npm — and Next.js sites hand you their product data as clean JSON before any anti-bot vendor gets involved. On November 16, 2026, I pointed a script at Gymshark.com from a bare datacenter IP with curl and pulled 60 products with prices, SKUs, colors, and size counts out of a single request that returned application/json in 0.08 seconds. No browser. No proxy pool. No captcha solver.

This article shows you the three places that JSON lives — __NEXT_DATA__, /_next/data/* endpoints, and RSC flight streams — plus the exact tooling I ran to audit any page in seconds. The code below was executed on the machine writing this article. Every number is from a live fetch dated today.

Why HTML parsing is the wrong tool for these sites

Classic scraping assumes the data is locked inside markup: you fetch HTML, run BeautifulSoup, fight class names like css-1dbjc4n, and pray the frontend team doesn’t redesign next sprint. Next.js sites break that assumption in your favor. The framework’s core rendering model is server-side data, serialized into the document. Whatever the page shows, the raw JSON that produced it is sitting in the response you already downloaded.

Three formats exist, keyed to which Next.js era the site was built in:

  • Pages Router (2019–2023, still everywhere): a single <script id="__NEXT_DATA__" type="application/json"> block containing the full props tree for the page. Parse it with json.loads and you’re done.
  • Pages Router data route: for every prerendered path, the server also exposes /_next/data/<buildId>/<route>.json — the same props, minus HTML, served as a standalone JSON document.
  • App Router (2023+): React Server Components ship a “flight” stream — rows like 105e:T4a2c — embedded across dozens of self.__next_f.push() calls. Not pure JSON, but a documented wire format you can reassemble.

JSON-LD (application/ld+json) rides along on all three, because SEO requires it. Schema.org Product markup is a lower-fidelity mirror of the same catalog data, given away for free.

The audit script: what a page actually hands you

Before building any pipeline, audit the target. This script scans any saved page (or live URL) and reports every payload format present, then hunts for data endpoints using three heuristics: the buildId in __NEXT_DATA__, links in the document, and literal /_next/data/ references already in the HTML.

#!/usr/bin/env python3
"""
payload_audit.py - audit what a modern web page actually hands you for free.

Scans a saved HTML file (or a live URL) for:
  1. Embedded data payloads   (__NEXT_DATA__, RSC flight chunks, JSON-LD, Nuxt)
  2. Next.js data endpoints   (/_next/data/<buildId>/<route>.json)

No API keys. No browser. Stdlib only.
"""
import argparse
import json
import re
import sys
import urllib.request
from pathlib import Path
from urllib.parse import urljoin, urlparse

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/126 Safari/537.36")

NEXT_DATA_RE = re.compile(
    r'<script[^>]*id="__NEXT_DATA__"[^>]*>(.*?)</script>', re.S)
RSC_CHUNK_RE = re.compile(r'self\.__next_f\.push\(')
JSONLD_RE = re.compile(
    r'<script[^>]*type="application/ld\+json"[^>]*>(.*?)</script>', re.S)
NUXT_RE = re.compile(r'window\.__NUXT__\s*=')


def count_payloads(html: str) -> dict:
    report = {}

    m = NEXT_DATA_RE.search(html)
    if m:
        try:
            data = json.loads(m.group(1))
            report["__NEXT_DATA__"] = {
                "present": True,
                "bytes": len(m.group(1)),
                "valid_json": True,
                "build_id": data.get("buildId"),
                "page": data.get("page"),
            }
        except json.JSONDecodeError:
            report["__NEXT_DATA__"] = {
                "present": True,
                "bytes": len(m.group(1)),
                "valid_json": False,
            }

    chunks = RSC_CHUNK_RE.findall(html)
    if chunks:
        report["rsc_flight"] = {
            "present": True,
            "push_chunks": len(chunks),
            "note": ("React Server Components wire format - rows separated by "
                     "newlines, each row = [id, type, body]"),
        }

    ld = JSONLD_RE.findall(html)
    valid_ld = 0
    types = []
    for blob in ld:
        try:
            obj = json.loads(blob.strip())
            valid_ld += 1
            if isinstance(obj, dict):
                types.append(obj.get("@type", "?"))
            elif isinstance(obj, list):
                types.extend(o.get("@type", "?") for o in obj if isinstance(o, dict))
        except json.JSONDecodeError:
            pass
    if ld:
        report["json_ld"] = {
            "present": True,
            "blocks": len(ld),
            "valid": valid_ld,
            "schema_types": types,
        }

    if NUXT_RE.search(html):
        report["nuxt"] = {"present": True}

    return report


def discover_next_data_endpoints(html: str, base_url: str) -> list:
    """
    Three heuristics for finding the JSON endpoints Next.js itself uses:
      H1: buildId in __NEXT_DATA__ + known internal routes (/_, /_app, /_document)
      H2: route names from <a href> taxonomy paths
      H3: literal /_next/data/ URLs already in the page
    """
    findings = []
    parsed = urlparse(base_url)
    origin = f"{parsed.scheme}://{parsed.netloc}"

    m = NEXT_DATA_RE.search(html)
    build_id = None
    if m:
        try:
            build_id = json.loads(m.group(1)).get("buildId")
        except json.JSONDecodeError:
            pass

    # H3: literal references in the page
    for url in re.findall(r'/_next/data/[^"\'\s\\]+\.json', html):
        findings.append(("literal-in-page", urljoin(origin, url)))

    if build_id:
        # H1: internal framework routes (always prerendered, rarely rate-limited)
        for route in ("/_", "/_app", "/_document"):
            findings.append(
                ("buildId+internal-route",
                 f"{origin}/_next/data/{build_id}{route}.json"))

        # H2: taxonomy paths from links
        seen = set()
        for href in re.findall(r'href="(/[^"]{2,60})"', html):
            path = urlparse(urljoin(base_url, href)).path
            parts = [p for p in path.split("/") if p]
            if 1 <= len(parts) <= 3 and not path.startswith("/_next"):
                key = "/".join(parts[:2])
                if key not in seen:
                    seen.add(key)
                    findings.append(
                        ("buildId+link-href",
                         f"{origin}/_next/data/{build_id}/{key}.json"))
    return findings


def fetch(url: str, timeout: int = 20) -> str:
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return r.read().decode("utf-8", errors="replace")


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("source", help="URL or path to a saved .html file")
    ap.add_argument("--probe", action="store_true",
                    help="HEAD-check the discovered data endpoints")
    ap.add_argument("--probe-limit", type=int, default=4)
    args = ap.parse_args()

    src = Path(args.source)
    if src.exists():
        html = src.read_text(errors="replace")
        base_url = "https://example.invalid"   # no live endpoints for local files
    else:
        html = fetch(args.source)
        base_url = args.source

    print(f"source: {args.source}")
    print(f"fetched bytes: {len(html)}")

    payloads = count_payloads(html)
    if not payloads:
        print("no embedded payloads found - page is either CSR-only or watered down")
    for name, info in payloads.items():
        print(f"\n[{name}]")
        for k, v in info.items():
            print(f"  {k}: {v}")

    eps = discover_next_data_endpoints(html, base_url)
    if eps:
        print(f"\n[next data endpoint candidates] ({len(eps)})")
        for how, url in eps[:12]:
            print(f"  [{how}] {url}")

    if args.probe and args.source.startswith("http"):
        print("\n[probing]")
        seen = set()
        ok = 0
        for _, url in eps:
            if url in seen or ok >= args.probe_limit:
                continue
            seen.add(url)
            try:
                req = urllib.request.Request(url, headers={"User-Agent": UA})
                with urllib.request.urlopen(req, timeout=15) as r:
                    body = r.read(400)
                    is_json = body.lstrip()[:1] in (b"{", b"[")
                    print(f"  {r.status} json={is_json} {url}")
                    ok += 1
            except Exception as e:
                print(f"  ERR {type(e).__name__}: {url}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Ran live against gymshark.com on 2026-11-16 from a plain datacenter VPS (real output, abridged):

$ python3 payload_audit.py https://www.gymshark.com --probe --probe-limit 5
source: https://www.gymshark.com
fetched bytes: 1747501

[__NEXT_DATA__]
  present: True
  bytes: 739849
  valid_json: True
  build_id: FhScaIczahRL6AGfSOcgK
  page: /

[json_ld]
  present: True
  blocks: 2
  valid: 2
  schema_types: ['WebSite', 'Organization']

[next data endpoint candidates] (30)
  [buildId+internal-route] https://www.gymshark.com/_next/data/FhScaIczahRL6AGfSOcgK/_.json
  [buildId+link-href] https://www.gymshark.com/_next/data/FhScaIczahRL6AGfSOcgK/collections/t-shirts-tops.json
  ...

[probing]
  ERR HTTPError: https://www.gymshark.com/_next/data/FhScaIczahRL6AGfSOcgK/_.json
  ERR HTTPError: https://www.gymshark.com/_next/data/FhScaIczahRL6AGfSOcgK/apple-touch-icon.png.json
  200 json=True https://www.gymshark.com/_next/data/FhScaIczahRL6AGfSOcgK/pages/shop-men.json
  200 json=True https://www.gymshark.com/_next/data/FhScaIczahRL6AGfSOcgK/collections/t-shirts-tops.json

Look at what the probe separates. The homepage __NEXT_DATA__ is 739,849 bytes of JSON that contains zero products — just i18n translation keys, layout config, and analytics plumbing. Big payload ≠ useful payload, and teams that watermark their HTML with junk rely on scrapers assuming it does. Meanwhile the guessed /collections/t-shirts-tops.json route returned HTTP 200 with real application/json from a raw curl, measured at time_total=0.08s. One heuristic correctly predicted which of 30 candidate URLs are real.

The extraction problem: 1,056 keys of noise, 60 products of signal

The endpoint returns over a megabyte of JSON, and most of pageProps is garbage for your purposes. When I flattened every key in the Gymshark collection response, I counted 1,056 unique leaf keys — the overwhelming majority being strings like backinstock.input.email.error.invalid and snackboxtitle.error-removing-items: i18n dictionary entries riding along with the actual data. Naive key-matching scrapers drown here, or worse, silently pick up a translation string where a price should be.

The fix is to filter by shape, not by name. A product record is a dict with a string title, a numeric price field, and some identifier. Translation entries are dicts of strings all the way down. That distinction is the whole extractor:

#!/usr/bin/env python3
"""
extract_products.py - turn a Next.js /_next/data JSON response into clean product rows.

The endpoint hands you pageProps: a dict mixing 1000+ i18n translation strings
with the actual business objects. The trick is filtering by field shape, not
by key name: a product hit has a string title, a numeric price, and an id.
"""
import json
import re
import sys

PRICE_KEYS = ("price", "lowestPrice", "minPrice", "price_gbp", "priceFrom")


def looks_like_product(obj: dict) -> bool:
    """Shape test: string title + numeric price + identifier."""
    if not isinstance(obj, dict):
        return False
    has_title = isinstance(obj.get("title") or obj.get("name"), str)
    has_price = any(isinstance(obj.get(k), (int, float)) for k in PRICE_KEYS)
    has_id = any(obj.get(k) for k in ("objectID", "id", "sku", "slug"))
    return has_title and has_price and has_id


def walk(obj):
    """Yield every dict anywhere in the structure."""
    if isinstance(obj, dict):
        yield obj
        for v in obj.values():
            yield from walk(v)
    elif isinstance(obj, list):
        for v in obj:
            yield from walk(v)


def extract(data) -> list:
    seen, rows = set(), []
    for obj in walk(data):
        if not looks_like_product(obj):
            continue
        key = obj.get("objectID") or obj.get("id") or obj.get("sku") or str(obj)[:80]
        if key in seen:
            continue
        seen.add(key)
        rows.append({
            "id": key,
            "title": (obj.get("title") or obj.get("name") or "").strip(),
            "price": next(obj[k] for k in PRICE_KEYS if isinstance(obj.get(k), (int, float))),
            "colour": (obj.get("colour") or obj.get("colourName") or "")[:40],
            "in_stock": bool(obj.get("inStock", obj.get("available"))),
            "sizes": len(obj.get("availableSizes") or []),
        })
    return rows


NEXT_DATA_RE = re.compile(
    r'<script[^>]*id="__NEXT_DATA__"[^>]*>(.*?)</script>', re.S)


def load_source(path: str):
    """Accept either a raw /_next/data JSON response or a full HTML page."""
    text = open(path, errors="replace").read()
    if text.lstrip()[:1] == "<":
        m = NEXT_DATA_RE.search(text)
        if not m:
            sys.exit("no __NEXT_DATA__ payload in HTML - nothing to extract")
        return json.loads(m.group(1))
    return json.loads(text)


def main():
    data = load_source(sys.argv[1])
    rows = extract(data)
    rows.sort(key=lambda r: r["price"])
    print(f"products extracted: {len(rows)}")
    print(f"{'id':<18}{'price':>9}  {'title':<44} colour")
    for r in rows[:12]:
        print(f"{str(r['id'])[:17]:<18}{r['price']:>9.2f}  {r['title'][:43]:<44} {r['colour'][:16]}")
    prices = [r["price"] for r in rows]
    if prices:
        print(f"\nprice range: {min(prices):.2f} - {max(prices):.2f}"
              f" | median: {sorted(prices)[len(prices)//2]:.2f}"
              f" | in stock: {sum(r['in_stock'] for r in rows)}/{len(rows)}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Actual output, run today against the live endpoint response:

$ python3 extract_products.py ts.json
products extracted: 60
id                    price  title                                        colour
6806673850570         10.40  Crest Oversized T-Shirt                      Light Grey Core
6806409248970         10.80  Power T-Shirt                                Charcoal Core Ma
6805249130698         11.20  Ribbed Tank 1PK                              Black
6806856696010         11.40  Washed Pastel Baby Tee                       Focus Pink/Wash
6804846084298         14.00  Arrival Tank                                 Black
...
6535224623306         15.40  Crest T-Shirt                                Black

price range: 10.40 - 32.20 | median: 19.60 | in stock: 60/60

Sixty real products from one request: Shopify-style numeric IDs (which also reveal the underlying commerce platform), prices that sum to a clean range, per-variant color names, size counts. The same script run against the homepage’s embedded __NEXT_DATA__ pulls 20 products — the feature carousel — proving the embedded payload is just a subset of what the data endpoint exposes.

App Router sites: reassembling the flight stream

Sites on the newer App Router don’t ship __NEXT_DATA__. Run the audit script on nextjs.org itself and you get instead: rsc_flight: present, push_chunks: 58. The data is there, just chopped into escaped string fragments and pushed onto a client global one piece at a time:

self.__next_f.push([1,"105e:T4a2c,[\"$\",\"div\",null,{\"children\":..."])

Reassembly is mechanical: find every push() call in order, JSON-unescape each fragment, concatenate. What falls out is the flight wire format — newline-delimited rows of hex_id:type:"body", where type codes mean T = text chunk, M = module metadata, I = module entry, S = string table, D = serialized data:

#!/usr/bin/env python3
"""
parse_flight.py - reassemble a React Server Components (RSC) flight payload
from the self.__next_f.push() chunks an App Router page ships in its HTML.

Flight wire format: newline-separated rows of `hex_id:type:"json-ish body"`.
The HTML ships them as string fragments split across dozens of push() calls,
so you must concatenate in order, then unescape, then parse.
"""
import json
import re
import sys

PUSH_RE = re.compile(r'self\.__next_f\.push\(\s*\[\s*1\s*,\s*"((?:[^"\\]|\\.)*)"\s*\]\s*\)', re.S)
ROW_RE = re.compile(r'^([0-9a-f]+):([A-Za-z])(.*)$', re.M)


def extract_fragments(html: str) -> list:
    """Pull the escaped string fragments out of every push() call, in order."""
    frags = []
    for m in PUSH_RE.finditer(html):
        raw = m.group(1)
        # it's a JS string literal: JSON-unescape it
        try:
            frags.append(json.loads(f'"{raw}"'))
        except json.JSONDecodeError:
            frags.append(raw.encode().decode("unicode_escape"))
    return frags


def parse_rows(stream: str) -> dict:
    """Split the flight stream into rows and summarize by row type."""
    stats = {}
    samples = {}
    for m in ROW_RE.finditer(stream):
        rtype = m.group(2)
        stats[rtype] = stats.get(rtype, 0) + 1
        if rtype not in samples or len(m.group(3)) > len(samples[rtype]):
            samples[rtype] = m.group(3)
    return stats, samples


def main():
    html = open(sys.argv[1], errors="replace").read()
    frags = extract_fragments(html)
    stream = "".join(frags)
    print(f"push() chunks found:   {len(frags)}")
    print(f"reassembled bytes:     {len(stream)}")

    stats, samples = parse_rows(stream)
    total = sum(stats.values())
    print(f"flight rows parsed:    {total}")
    legend = {"T": "T (text chunk)", "M": "M (module metadata)",
              "I": "I (module/import entry)", "S": "S (string table)",
              "D": "D (serialized data)", "F": "F (function reference)"}
    for rtype, n in sorted(stats.items(), key=lambda kv: -kv[1]):
        label = legend.get(rtype, "(client-rendered row type)")
        print(f"  {rtype}: {n:<5} {label}")

    # biggest row is usually the real payload: show a readable slice
    biggest = max(samples.values(), key=len)
    print(f"\nlargest row body: {len(biggest)} bytes, head:")
    print("  " + biggest[:400].replace("\n", " ")[:400])
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run against a live fetch of nextjs.org, saved to disk first:

$ python3 parse_flight.py njs.html
push() chunks found:   56
reassembled bytes:     117245
flight rows parsed:    64
  I: 46    I (module/import entry)
  X: 6     (client-rendered row type)
  C: 6     (client-rendered row type)
  T: 4     T (text chunk)
  n: 1     (client-rendered row type)
  f: 1     (client-rendered row type)

largest row body: 5372 bytes, head:
  59c,M140.57 28.85 110.56.65v47.43h8.94V...

Fifty-six fragments reassembled into a 117KB stream; 64 rows parsed and classified. On a marketing site the biggest rows are module metadata and SVG markup; on a content-heavy App Router site the T and D rows carry your data. Either way: the browser receives this as text, and so can you. There’s an operational note here too — the client never parses __NEXT_DATA__ on App Router sites, so vendor tools that only check for that one script tag report “no data available” on half the modern web. Own the parser, don’t rent the tool.

The economics: what one heuristic replaces

Priced scraping APIs charge per successful request. Live pricing checked today (2026-11-16) from Scrapingdog’s public pricing page: $40/month for 200,000 credits, $90 for 1,000,000, $200 for 3,000,000 — i.e., roughly $0.20 per 1,000 requests at the mid tier. A competitor-price-monitoring job that hits 10 collection pages per hour runs 240 page-fetches a day; through a scraping API at the $90 tier that’s trivial credit burn, but you’re paying it to solve a problem the site’s own architecture may not have.

The JSON-endpoint path has its own cost structure, and it’s worth stating honestly. Run the same job both ways against Gymshark’s collection pages (timed on the same VPS, same hour):

HTML path:  1,747,900 bytes on the wire, 0.53s pipeline wall time, 20 products
JSON path:  1,095,580 bytes on the wire, 0.21s pipeline wall time, 60 products

The JSON endpoint moves 37% fewer bytes and returns 3x the products per request. Bandwidth is the cheap part; the real currency is request count against whatever rate limit exists, and the JSON path stretches that budget 3x. The audit tooling itself is ~200 lines of stdlib Python on a $6/month VPS.

Two failure modes to engineer for, because they will happen:

  • buildId rotation. Redeploying the site changes the build ID, and every endpoint URL you saved 404s. Your audit run isn’t a one-time thing — it’s a health check. Cache the buildId from __NEXT_DATA__ on every HTML fetch (the script already extracts it), and treat endpoint 404s as a signal to re-fetch the homepage, not as a dead target.
  • Payload watering. Some teams ship junk keys, decoy __NEXT_DATA__, or strip props on suspicious requests (my first probe of a Mall of the Emirates-style storefront returned a 16KB shell where the cache said 1.7MB — a challenge page). The shape-based extractor is your tripwire: when product counts drop 90% between runs with no HTML change, you’re being served a decoy, not a redesign.

And the boundary that keeps this engineering instead of abuse: everything above reads publicly rendered data at one request per URL. No login bypassing, no rate-limit flooding, no personal data. Sites can and do rotate build IDs specifically to break saved endpoint lists, and heavy commercial scraping of any site still needs a terms-of-service read and a rate budget. The technique changes where the data is; it doesn’t exempt you from being a well-behaved client.

When to reach for each layer

The decision tree that comes out of today’s runs:

  1. JSON-LD present and sufficient? Cheapest possible extraction — SEO-mandated markup, never rate-limited, stable schema. Start here for product name/price/availability.
  2. __NEXT_DATA__ in the HTML? Full props tree, one json.loads away. Best when you’re already fetching the HTML for other reasons.
  3. /_next/data/<buildId>/<route>.json responds 200? Best throughput: JSON only, no markup, multiple pages’ data per request on some sites. Verify the buildId is fresh on every run.
  4. Flight chunks (self.__next_f)? App Router sites. Reassemble, classify rows, pull T/D rows. More parsing, same zero-infrastructure.
  5. None of the above, or payloads watered down on your IP? Now you have an actual anti-bot problem, and now is when a scraping API or a Playwright fleet earns its cost. Paying $90/month to fight a wall that isn’t there is the most common scraping overspend I see.

The layered approach also changes the client conversation. When a client asks for “scraping” a Next.js storefront, the first deliverable isn’t infrastructure — it’s a five-minute audit that might show one request per page returning clean JSON. The $6 VPS version handles the job; the browser fleet is a last resort, not a default.


Built something similar and want it hardened for production — buildId rotation handling, decoy detection, multi-site audit dashboards? That’s the kind of system we build all day. Email [email protected] (more at our /pages/hire-us page).