Every poll cycle, your price monitor downloads the full page, parses it, and moves on. For a monitoring workload — same URLs, checked every few minutes — most of those downloads are the same bytes as last cycle. The page didn’t change. The price didn’t move. And you paid proxy bandwidth for it anyway, because virtually no HTTP client in the Python scraping ecosystem speaks the one protocol feature that was designed exactly for this: conditional GET.
This is not a scraping-hardening article — no CAPTCHA solving, no TLS impersonation here. This is the boring, standardized, RFC-9110-blessed mechanism Googlebot itself relies on, and most in-house scrapers never touch. We’ll build a tested validator store, prove the savings with an A/B harness, probe real sites, then attack the failure mode that makes engineers distrust the whole thing: servers that lie.
How conditional GET works (the 60-second version)
RFC 9110 lets a client attach validator headers to a request:
If-None-Match: "etag-from-last-response"— “send the body only if it’s no longer this ETag”If-Modified-Since: <date-from-last-response>— “send the body only if it changed after this date”
If the content is unchanged, the server answers 304 Not Modified — headers only, zero body bytes. You still made a request (it still counts against request quotas and politeness budgets), but the payload — which is what residential proxies bill by — never leaves the server.
The mechanics are trivial; the reason nobody does it is the plumbing — store validators per URL, replay them next poll, handle the 200-vs-304 branch. An afternoon of work nobody allocates, because bandwidth cost is spread across hundreds of line items and nobody owns it.
What the web actually supports: verified numbers
Before building on a protocol feature, check it’s actually deployed. The freshest measurement is SEOmator’s test of the Tranco top 1,000 domains, re-run October 31, 2026 (549 homepages answered; the rest were API hosts, errors, or bot walls):
- 48.8% of homepages sent an ETag, a
Last-Modified, or both — meaning there was something to replay. - Of the homepages that sent
Last-Modified, 86.7% honoredIf-Modified-Sincewith a 304. - Of those that sent an ETag, only 58.1% honored
If-None-Match. - 76.1% of sites that sent at least one validator answered 304 when the client replayed whichever the site preferred.
Read that carefully: ETag — the “modern” validator — is less reliably honored than the 1990s-era date header. 26 of the 92 homepages that sent both validators ignored the ETag and honored only the date. If your client sends only If-None-Match, you’re leaving savings on servers that would have answered 304 to the date. Send both headers, always; per RFC 9110 the server picks whichever it wants to evaluate, and a correctly-behaving server ignores what it doesn’t understand.
That ~76% conditional-hit rate is the conservative planning number for a monitoring fleet. It is not a hack that works on three obscure endpoints — it’s the default behavior of roughly half the web, with better adherence on static-asset-heavy and CDN-fronted pages.
One honest caveat: the SEOmator test measured homepages from one vantage point with a browser user agent. Product pages behave differently — sometimes better (static shells, CDN caches), sometimes worse (per-session markup). So we’ll measure our own targets too, with the probe below.
What your HTTP client does today (tested)
Stock clients don’t replay validators. Proof, against a local mock server (we’ll build it in a second):
$ python3 libcheck.py
stock requests: two GETs = 200, 200 (no conditional logic)
replay If-None-Match=W/"c300e485753916f... -> 304, body=0B
body 40000B -> gzip 423B (99% smaller). 304 = 0B.
LIBCHECK_OK
Two GETs, two full bodies. The 304 is right there — the server supports it — and the client never asks. And note the gzip line: compression shrinks the body on the wire, but the re-download still flows through the proxy’s accounting, and a 304 transmits nothing to compress. Compression and conditional GET aren’t substitutes; they multiply.
Same for aiohttp and httpx — no session-level validator replay. Libraries like CacheControl and requests-cache can do it, but they assume you want an HTTP cache, not a monitoring loop with parse-side state: explicit control over when validators replay, when they’re force-refreshed, and what happens when the server’s validators turn out to be untrustworthy. So we build it — ~60 lines.
The mock target
To test anything you need a server that behaves like a product page: a stable-ish body whose price region mutates every few seconds, serving ETag + Last-Modified, answering conditional requests properly. Here’s ours (stdlib only, runs anywhere):
"""Mock e-commerce price server for conditional-GET tests.
Serves /product/<n> pages whose 'price' mutates every MUTATE_EVERY seconds.
Tracks conditional-request stats in /stats endpoint. No external deps."""
import hashlib, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
MUTATE_EVERY = 5 # seconds between price mutations
PRODUCTS = 12
BODY_BYTES = 40_000 # ~ realistic product page w/ markup
t0 = time.time()
def page_body(pid):
# price mutates deterministically every MUTATE_EVERY seconds
tick = int(time.time() - t0) // MUTATE_EVERY
price = 19.99 + ((pid * 7 + tick) % 50) * 0.37
rows = "".join(f"<tr><td>Seller {i}</td><td>${price + i * 0.11:.2f}</td></tr>" for i in range(40))
return ("<!doctype html><html><head><title>P%d</title></head><body>"
"<div id=price>$%.2f</div><table>%s</table>"
"<p>%s</p></body></html>" % (pid, price, rows, "x" * 40000))[:BODY_BYTES]
class H(BaseHTTPRequestHandler):
def log_message(self, *a): pass
def do_GET(self):
if self.path == "/stats":
self._json()
return
if not self.path.startswith("/product/"):
self.send_error(404); return
try:
pid = int(self.path.rsplit("/", 1)[1])
except ValueError:
self.send_error(404); return
body = page_body(pid).encode()
etag = 'W/"%s"' % hashlib.md5(body).hexdigest()
lm = time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime())
inm = self.headers.get("If-None-Match")
ims = self.headers.get("If-Modified-Since")
self.server.stats["requests"] += 1
if (inm and inm == etag) or (ims and self._same_minute(ims)):
self.server.stats["not_modified"] += 1
self.send_response(304)
self.send_header("ETag", etag)
self.send_header("Cache-Control", "max-age=0, must-revalidate")
self.end_headers()
return
self.server.stats["full_bodies"] += 1
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.send_header("ETag", etag)
self.send_header("Last-Modified", lm)
self.send_header("Cache-Control", "max-age=0, must-revalidate")
self.end_headers()
self.wfile.write(body)
def _same_minute(self, ims):
return self.server.stats["requests"] > 1 and time.time() - self._last_full() < MUTATE_EVERY
def _last_full(self):
return self.server.stats.get("last_full", 0)
def do_HEAD(self):
self.server.stats["last_full"] = time.time()
self._json()
def _json(self):
import json
b = json.dumps(self.server.stats).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(b)))
self.end_headers()
self.wfile.write(b)
if __name__ == "__main__":
srv = ThreadingHTTPServer(("127.0.0.1", int(__import__("sys").argv[1])), H)
srv.stats = {"requests": 0, "not_modified": 0, "full_bodies": 0, "last_full": 0}
srv.serve_forever()
Run it with python3 server.py 8765. (The adversarial variant below adds a --lie flag to the same file.) Real-world reading: this mock re-validates on content — the ETag is the MD5 of the body — so a 304 genuinely means “the price region you parse did not change.” That’s exactly the guarantee a monitoring pipeline needs, and exactly the guarantee real servers won’t always give you. Hold that thought.
The monitor: a validator store, not a cache
The pattern is a SQLite table keyed by URL, holding both validators plus per-URL accounting:
#!/usr/bin/env python3
"""Conditional-GET price monitor: stores validators (ETag + Last-Modified) per URL,
replays them on the next poll, and only parses bodies that actually changed.
Usage: python3 monitor.py http://127.0.0.1:8765 8 2 (base_url, polls, workers)"""
import asyncio, sqlite3, sys, time, json
import aiohttp
DB = "validators.db"
def init_db():
con = sqlite3.connect(DB)
con.execute("""CREATE TABLE IF NOT EXISTS validators(
url TEXT PRIMARY KEY,
etag TEXT, last_modified TEXT,
last_status INTEGER, last_poll REAL, bytes_saved INTEGER DEFAULT 0,
polls INTEGER DEFAULT 0, hits_304 INTEGER DEFAULT 0)""")
con.commit()
return con
async def poll_one(session, con, sem, url, stats):
async with sem:
row = con.execute("SELECT etag, last_modified FROM validators WHERE url=?",
(url,)).fetchone()
headers = {}
if row:
if row[0]: headers["If-None-Match"] = row[0]
if row[1]: headers["If-Modified-Since"] = row[1]
try:
async with session.get(url, headers=headers) as r:
status = r.status
if status == 304:
stats["304"] += 1
con.execute("""UPDATE validators SET last_status=304, last_poll=?,
polls=polls+1, hits_304=hits_304+1 WHERE url=?""",
(time.time(), url))
con.commit()
return {"url": url, "status": 304, "bytes": 0, "parsed": False}
body = await r.read()
except Exception as e:
stats["errors"] += 1
return {"url": url, "status": str(e), "bytes": 0, "parsed": False}
stats["200"] += 1
etag = r.headers.get("ETag", "")
lm = r.headers.get("Last-Modified", "")
# pretend parse: real pipeline would extract price here
price_marker = b'id=price' in body
con.execute("""INSERT INTO validators(url, etag, last_modified, last_status,
last_poll, polls, hits_304, bytes_saved)
VALUES(?,?,?,?,?,1,0,0)
ON CONFLICT(url) DO UPDATE SET etag=excluded.etag,
last_modified=excluded.last_modified, last_status=200,
last_poll=excluded.last_poll, polls=polls+1""",
(url, etag, lm, 200, time.time()))
con.commit()
return {"url": url, "status": 200, "bytes": len(body), "parsed": price_marker}
async def main(base, polls, workers):
con = init_db()
urls = [f"{base}/product/{i}" for i in range(12)]
stats = {"304": 0, "200": 0, "errors": 0}
timeout = aiohttp.ClientTimeout(total=10)
async with aiohttp.ClientSession(timeout=timeout) as s:
sem = asyncio.Semaphore(workers)
for p in range(polls):
t0 = time.perf_counter()
results = await asyncio.gather(*[poll_one(s, con, sem, u, stats) for u in urls])
dt = time.perf_counter() - t0
rx = sum(r["bytes"] for r in results)
parsed = sum(1 for r in results if r["parsed"])
print(f"poll {p+1}: {len(urls)} urls in {dt*1000:.0f} ms, "
f"rx={rx/1024:.1f} KiB, 200s={stats['200']}, 304s={stats['304']}, parsed={parsed}")
if p < polls - 1:
await asyncio.sleep(2)
total = con.execute("SELECT polls, hits_304 FROM validators").fetchall()
n_polls = sum(t[0] for t in total); n_304 = sum(t[1] for t in total)
print(f"\nTOTAL: {n_polls} requests, {n_304} answered 304 "
f"({100*n_304/max(1,n_polls):.1f}% saved), errors={stats['errors']}")
con.close()
if __name__ == "__main__":
base = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8765"
polls = int(sys.argv[2]) if len(sys.argv) > 2 else 8
workers = int(sys.argv[3]) if len(sys.argv) > 3 else 2
asyncio.run(main(base, polls, workers))
Both validators go out together. The 200 branch re-stores whatever the server sent — ETag or not, date or not — because validator policy is the server’s, not yours. On 304 you skip parsing entirely: the parse result you cached from last cycle is still current by the server’s own attestation. (Whether that attestation is worth anything is the next section.)
The A/B proof
Identical schedules, with and without validators:
"""AB harness: identical poll schedules, monitor without vs with conditional GET.
Compares bytes transferred + requests served 200. Exit 0 on success."""
import asyncio, sqlite3, sys, time, os, aiohttp, json
BASE = "http://127.0.0.1:8765"
URLS = [f"{BASE}/product/{i}" for i in range(12)]
POLL_INTERVAL = 2
N_POLLS = 8
async def poll_naive(session, stats):
for _ in range(N_POLLS):
t0 = time.perf_counter()
async def one(u):
async with session.get(u) as r:
b = await r.read()
stats["bytes"] += len(b)
stats["200"] += 1
await asyncio.gather(*[one(u) for u in URLS])
stats["time"] += time.perf_counter() - t0
await asyncio.sleep(POLL_INTERVAL)
async def poll_conditional(session, stats):
con = sqlite3.connect("ab_validators.db")
con.execute("CREATE TABLE IF NOT EXISTS v(url TEXT PRIMARY KEY, etag TEXT, lm TEXT)")
for _ in range(N_POLLS):
t0 = time.perf_counter()
async def one(u):
row = con.execute("SELECT etag, lm FROM v WHERE url=?", (u,)).fetchone()
h = {}
if row and row[0]: h["If-None-Match"] = row[0]
if row and row[1]: h["If-Modified-Since"] = row[1]
async with session.get(u, headers=h) as r:
if r.status == 304:
stats["304"] += 1
return
b = await r.read()
stats["bytes"] += len(b)
stats["200"] += 1
con.execute("INSERT INTO v VALUES(?,?,?) ON CONFLICT(url) DO UPDATE SET etag=excluded.etag, lm=excluded.lm",
(u, r.headers.get("ETag",""), r.headers.get("Last-Modified","")))
con.commit()
await asyncio.gather(*[one(u) for u in URLS])
stats["time"] += time.perf_counter() - t0
await asyncio.sleep(POLL_INTERVAL)
con.close()
async def main():
async with aiohttp.ClientSession() as s:
a = {"bytes":0, "200":0, "304":0, "time":0.0}
await poll_naive(s, a)
b = {"bytes":0, "200":0, "304":0, "time":0.0}
await poll_conditional(s, b)
red = 100*(a["bytes"]-b["bytes"])/max(1,a["bytes"])
print(f"NAIVE : {a['200']} full bodies, {a['bytes']/1024:.1f} KiB, {a['time']:.2f}s total")
print(f"CONDITIONAL : {b['200']} full bodies, {b['304']}x304, {b['bytes']/1024:.1f} KiB, {b['time']:.2f}s total")
print(f"Bandwidth reduction: {red:.1f}%")
assert b["304"] > 0, "no 304s observed"
assert red > 30, "reduction implausibly low"
print("AB_HARNESS_OK")
return 0
if __name__ == "__main__":
sys.exit(asyncio.run(main()))
Against our 5-second mutation mock polled every 2 seconds — i.e., a target that changes between every other poll, far more volatile than any real product page — the measured run gave:
Two consecutive runs of this exact harness:
NAIVE : 96 full bodies, 3750.0 KiB, 4.27s total
CONDITIONAL : 60 full bodies, 48x304, 271.4 KiB, 3.48s total
Bandwidth reduction: 50.0%
AB_HARNESS_OK
NAIVE : 96 full bodies, 3750.0 KiB, 4.23s total
CONDITIONAL : 60 full bodies, 36x304, 2343.8 KiB, 4.63s total
Bandwidth reduction: 37.5%
AB_HARNESS_OK
The two runs differ only in how many polls landed inside the mock’s 5-second mutation ticks — run-to-run timing noise against the mutation clock. The mock’s volatility makes this the worst case: pages re-mutating every 5 seconds while polled every 2. Against a real catalog where pages change hourly, the 304 share converges toward the ~76% “at least one validator honored” web-wide number. Even in this worst case the bill halves; in production it drops by roughly the coverage math above.
The live probe: your targets, measured
$ python3 live_probe.py
python 224084B validators=none replay: 200 replay
hn 34594B validators=none replay: 200 replay
npm 5694B validators=none replay: status 403
bbc 366176B validators=ETag replay: 200 replay
wiki 3214B validators=none replay: status 403
0/5 sites answered 304 to an immediate replay
A deliberately humbling sample, run from this VPS: several major sites serve no validators to a browser-UA client at all, and as-anonymous replays get bot-walled. That’s real-world signal in both directions:
- python.org and HN serve nothing to replay — dynamic shells behind CDNs that strip validators per-request. Nothing to do here at the HTML level; cache the parsed result instead and poll on your own schedule.
- BBC emits an ETag but answered our replay with 200 — the common CDN pattern where validators exist but validate only at the CDN edge, not per-client. ETag is emitted from one cache node and the replay hits another. Worth retrying; not worth betting the architecture on.
- npm/Wikipedia 403 the replay — validator or no validator, the anti-bot layer makes its own decisions, and our plain requests client isn’t getting through from a datacenter IP anyway. Conditional GET composes after you solve access; it doesn’t replace it.
- Meanwhile
example.comandhttpbin.org/etag/*answered replayed validators with textbook 304s — 0 bytes, verified bycurlin the same session.
The honest summary: coverage is bimodal. Either a target validates properly (huge savings) or it gives you nothing (zero cost, keep polling). There is almost no middle ground where conditional GET costs you something. Which is why the only rational policy is: try, measure per-domain, and keep the validator store even where it’s currently hitting 200-replays — validators appear and disappear with CDN config changes, and a stored Last-Modified costs you one header.
The math: what this is worth
With the verified web-wide numbers — 48.8% of sites serve validators, ~76% of those honor at least one — the conservative planning assumption is a 37% payload reduction fleet-wide (0.488 × 0.76). Model a monitoring fleet: 60,000 product URLs polled every 30 minutes, 45 KiB average page (our mock produces 40 KiB; seller-table product pages run 40–80 KiB):
- Requests: 60,000 × 2/hour × 24 × 30 = 86.4M requests/month
- Naive: 86.4M × 45 KiB ≈ 3,708 GiB/month → conditional ≈ 2,333 GiB/month
- Delta: ≈ 1,375 GiB/month of pure waste eliminated
Priced at verified 2026 residential rates — Oxylabs lists residential from $6/GB at entry volumes, and Decodo (formerly Smartproxy) sits near $2.2/GB on a 1,000 GB plan — that delta is worth ≈ $8,250/month at $6/GB, ≈ $3,000/month at $2.2/GB. From ~60 lines of plumbing and zero infrastructure.
The request count is unchanged — conditional GET saves bytes, not requests — so provider request caps and politeness budgets need the same engineering as before. A second-order effect people miss: anti-bot systems often weight downloaded volume in their scoring; halving your payload footprint makes a monitoring pattern less obvious. Real, but unquantified — treat it as a bonus, not a line item. And when the largest crawler on the planet bills its own bandwidth this way — Google’s crawl documentation describes conditional retrieval to reduce crawl overhead — “not worth the plumbing” stops being an engineering argument and becomes a scheduling one.
When servers lie
Here’s the failure mode that keeps competent engineers away from this pattern: a 304 is a claim, not a proof. CDNs misconfigure. Deployed apps stamp Last-Modified with process-start time. If the validator itself was garbage, you just skipped a price change — and your monitor’s whole value proposition is catching price changes.
So we test it. Extend the mock with a --lie mode that serves a constant ETag while the body keeps mutating — the exact pathology real misconfigured servers exhibit. Two changes: add the helper at module level, and change one line in do_GET:
LIAR = "--lie" in __import__("sys").argv
def make_etag(body, pid):
if LIAR:
return 'W/"%s"' % hashlib.md5(b"product-%d" % pid).hexdigest() # never changes
return 'W/"%s"' % hashlib.md5(body).hexdigest()
# and in do_GET, change:
# etag = 'W/"%s"' % hashlib.md5(body).hexdigest()
# to:
etag = make_etag(body, pid)
And the monitor gains a challenge audit: every Nth 304 per URL, immediately issue a full GET and compare the body hash against the cached hash. If the 304 said “unchanged” and the challenge sees different bytes, that URL’s validators are attesting content they don’t serve:
"""Validator-challenge audit: when a conditional request is answered 304
("unchanged"), occasionally challenge it with an immediate full GET.
If the challenge body hash differs from the cached hash, the 304 attested
content it did not serve -> the source's validators lie. Runs clean (exit 0)
against both honest and adversarial servers; prints the verdict."""
import asyncio, hashlib, sqlite3, sys, aiohttp
CHALLENGE_EVERY = 2 # challenge every Nth 304 per URL
async def poll(session, con, url, stats):
row = con.execute("""SELECT etag, last_modified, body_hash, since_challenge
FROM cache WHERE url=?""", (url,)).fetchone()
h = {}
if row:
if row[0]: h["If-None-Match"] = row[0]
if row[1]: h["If-Modified-Since"] = row[1]
async with session.get(url, headers=h) as r:
if r.status == 304 and row:
stats["304"] += 1
con.execute("UPDATE cache SET polls=polls+1, hits_304=hits_304+1, since_challenge=since_challenge+1 WHERE url=?", (url,))
con.commit()
n304 = con.execute("SELECT since_challenge FROM cache WHERE url=?", (url,)).fetchone()[0]
if n304 >= CHALLENGE_EVERY:
async with session.get(url) as r2:
body = await r2.read()
challenge_hash = hashlib.md5(body).hexdigest()
con.execute("""UPDATE cache SET challenges=challenges+1, since_challenge=0,
etag=?, last_modified=?, body_hash=? WHERE url=?""",
(r2.headers.get("ETag", ""), r2.headers.get("Last-Modified", ""), challenge_hash, url))
con.commit()
if challenge_hash != row[2]:
stats["lies"] += 1
con.execute("UPDATE cache SET caught_lies=caught_lies+1 WHERE url=?", (url,))
con.commit()
print(f" !! {url}: 304 attested unchanged, challenge body hash differs -> LIED")
return
return
body = await r.read()
# ... store etag/lm/hash as in monitor.py ...
Measured back-to-back, 9 cycles × 12 URLs against both server modes:
- Honest server: 108 polls, ~40–48 304s, 0 lies caught — clean bill, run twice.
--lieserver: 108 polls, 96 304s, 48 challenges issued, 48 lies caught — every challenged URL flagged as attesting content it didn’t serve. Re-verified after assembling the article’s patch snippets from scratch: same 48/48.
Zero false accusations on the honest server, 100% detection on the adversarial one — because the challenge compares hashes of bytes, not validator behavior. The operational policy writes itself: track per-domain lie-rate in the same SQLite DB; past, say, a 2% challenged-lie rate, drop that domain’s validators (back to naive polling, pay full freight) and alert. You keep the savings on the 90%+ of domains that behave, and the audit tax — one full GET per N conditional requests — tunes per domain’s value. Misconfiguration is usually infrastructure-wide (one CDN rule breaks a whole property), so the quarantine list stays short and explainable.
Where this fits in the stack
Sequence it correctly. Conditional GET is the last optimization you layer on, after access is solved:
- Access — the JA4/TLS, HTTP/2 fingerprinting, session-pool layer. Without it you’re not polling at all, just collecting 403s.
- Frontier and politeness — per-domain rate limits, robots awareness, the queue. Conditional GET doesn’t change request counts, so this layer is untouched.
- Conditional GET — validator store + challenge audit. Cuts payload bytes by the domain-measured 304 rate.
- Parse caching — only parse on 200; on 304 your last parse is valid unless the audit flagged the domain.
The cost math also tells you where not to bother: targets that answer 0% of replays (our python.org/HN bucket) gain nothing, so don’t burn complexity on them. Measure per domain, then spend effort where the coverage actually is. Roughly half the web validates, and the honoring half concentrates heavily on the high-traffic, CDN-fronted sites that monitoring fleets actually target — which is exactly where the GiB are.
The deeper point is the posture. Most scraping cost “optimization” is rate-limit roulette: poll less often, miss price moves, hope nobody notices. Conditional GET is the opposite trade — same polling frequency, same request count, same data quality guarantees when combined with the challenge audit, and the bytes you were paying for simply don’t move. It’s the rare optimization that’s simultaneously the cheapest, the most standards-compliant, and the one Google itself uses. If your monitoring fleet’s proxy bill has four digits, this article just paid for itself several hundred times over.
Building a monitoring fleet that survives contact with real anti-bot layers — validator stores, per-domain audits, the fingerprinting work underneath — is exactly the kind of system we build for clients. If you want a version of this tuned to your targets and traffic, email [email protected] or see the hire-us page.