Here’s the failure mode nobody demos: the scraper is green. Cron says it ran. Exit code 0. Logs show requests going out. And the data is wrong — or worse, quietly disappearing. A scraper that crashes is an annoyance; you fix it the morning you notice. A scraper that keeps “working” while its output degrades is a liability, because everything downstream — your pricing engine, your client dashboards, your ML features — keeps consuming the garbage as if it were fine, and the poison spreads through every table it touches before anyone looks.

The industry-wide cost of this class of failure is well documented: Gartner research puts the average cost of poor data quality at $12.9 million per year per organization, and IBM’s estimate for the US economy alone is $3.1 trillion annually. Your scraper is not special — it will eventually contribute to a number like this for someone. The question is whether you find out from a dashboard or from a client.

This article builds the piece of infrastructure that catches it: a drift sentinel — a separate process that judges your scraper by its output database, not by its logs. Every line below was executed and verified before it went in.

The three ways scrapers rot without failing

After enough production scraping pipelines, the silent failures reduce to three archetypes:

1. Complete silence. The source changed its URL scheme, got behind a new WAF, or now serves a Cloudflare interstitial to your IP. The run “succeeds”, inserts zero rows, exits 0. One day of this looks like a slow news day. Five days of this is a data hole your client’s backtesting will eventually fall into. Nobody alerts because the process never errored.

2. Schema drift. The site renames price to discounted_price, or moves the element out of the container your selector watches. Your parser — depending on how you wrote it — starts emitting None for that field on every row. Here’s the trap: None is not an error in most extraction code. It’s a value. It flows into the warehouse, and every average, every percentile, every filter silently shifts meaning while the row counts stay perfectly healthy. This is the most expensive failure mode because every statistic downstream of that column is now quietly wrong.

3. Soft rot. Pagination breaks on page 3 of 20 and the loop swallows the exception and continues. Your rows-per-run drops 85%. Nothing crashed, nothing went null, you just get less of the world every day. Detection requires comparing volume against your own baseline, not against zero.

What all three have in common: the log file says everything is fine. That’s the whole reason the sentinel must not read logs.

Why log-based monitoring misses all of this

The instinct most teams have — alert when the process exits non-zero, alert when a request 500s — is monitoring the wrong artifact. Consider what actually happened in failure mode 1: requests went out, responses came back, the parser found nothing, the loop ended, exit 0. From the outside this is indistinguishable from a healthy run. The only artifact that observed the difference is the database: yesterday the table gained 40 rows, today it gained zero.

Log-based monitoring watches the machine. The sentinel watches the product. This distinction sounds philosophical until you price it: a data client who discovers a two-week hole in your feed doesn’t care what your logs said — they care that their model trained on phantom zeros.

The sentinel design

The sentinel is deliberately boring:

  • Input: the scraper’s own output warehouse — in this build, a SQLite table of (source, day, title, price, rating) where source is the scraper ID and day is the UTC run date. One table, no exotic dependencies.
  • Check 1 — source silence: for each source, count how many of the last 7 days are missing entirely. Two or more missing days ⇒ CRITICAL.
  • Check 2 — row-count collapse: a present-but-anemic day (fewer than min_rows) ⇒ HIGH.
  • Check 3 — schema drift via null-rate baseline: for each field, compute today’s null rate and compare it to the source’s own trailing baseline over the lookback window. Crossing a warn/fail threshold while the baseline was clean ⇒ SCHEMA_DRIFT. Critically, it learns per-field sparsity: a rating column that’s always 75% null is by design, and it never fires; a price column that goes from 0% null to 67% null overnight is a selector dying in real time.
  • Output: JSON findings to stdout, severity-tagged, machine-routable into any alerting stack (Telegram bot, email, PagerDuty, whatever you already run).

It reads the warehouse read-only, never touches the scraper, and runs in milliseconds. You schedule it after the daily scrape in the same cron chain.

The code (fully tested)

The full sentinel, stdlib-only. It ships with a data generator and two fault injectors, so you can watch it catch both failure modes yourself:

#!/usr/bin/env python3
"""
Drift Sentinel - detects silent scraper breakage from inside the DB, not the logs.

Handles two failure modes:
  1. COMPLETE break: a source returns 0 rows for a day where peers are healthy.
  2. SCHEMA drift: a field starts going null for a source while peers keep it filled.

Exit code: 0 = healthy or actionable report produced, 2 = sentinel itself broken (bad run structure).
"""
import json
import sqlite3
import sys
from datetime import datetime, timedelta

DB = "warehouse.db"
FIELD_HOLD = {"title", "price", "rating"}
SOURCE_HOLD = {"s1", "s2", "s3", "s4", "s5"}
TODAY = "2026-11-08"

# ---- break 1: two days of complete silence from source s4 ---------------
def break_silence():
    db = sqlite3.connect(DB)
    db.execute("DELETE FROM items WHERE source='s4' AND day IN ('2026-11-07','2026-11-08')")
    db.commit(); db.close()

# ---- break 2: schema drift, s5's 'price' starts going null ---------------
def break_schema():
    db = sqlite3.connect(DB)
    db.execute("UPDATE items SET price=NULL WHERE source='s5' AND day='2026-11-08' AND id % 3 != 0")
    db.commit(); db.close()

def build_db():
    db = sqlite3.connect(DB)
    db.execute("DROP TABLE IF EXISTS items")
    db.execute("""CREATE TABLE items(
        id INTEGER PRIMARY KEY,
        source TEXT NOT NULL,       -- scraper source id
        day  TEXT NOT NULL,         -- UTC date of the run
        title TEXT,                 -- expected: never null in practice
        price REAL,                 -- expected: filled on >95% of rows
        rating REAL                 -- expected: sparse by design (~25%)
    )""")
    import random
    random.seed(42)
    rows = []
    iid = 0
    for d in range(14):
        day = (datetime(2026, 10, 26) + timedelta(days=d)).strftime("%Y-%m-%d")
        for s in ("s1", "s2", "s3", "s4", "s5"):
            for n in range(40):
                iid += 1
                rows.append((iid, s, day,
                             f"item {iid}",
                             round(random.uniform(5, 500), 2),
                             round(random.uniform(1, 5), 1) if random.random() < 0.25 else None))
    db.executemany("INSERT INTO items(id,source,day,title,price,rating) VALUES(?,?,?,?,?,?)", rows)
    db.commit()
    db.close()

# ---- the sentinel --------------------------------------------------------
def sentinel(db_path=DB, today=TODAY, lookback_days=7,
             min_rows=10, null_warn=0.15, null_fail=0.40, silence_days=2):
    db = sqlite3.connect(db_path)
    days = [(datetime.strptime(today, "%Y-%m-%d") - timedelta(days=i)).strftime("%Y-%m-%d")
            for i in range(lookback_days)]
    q = "?" + ",?" * (len(days) - 1)
    findings = []

    rows = db.execute(
        f"SELECT source, day, COUNT(*), SUM(CASE WHEN title IS NULL THEN 1 ELSE 0 END), "
        f"SUM(CASE WHEN price IS NULL THEN 1 ELSE 0 END), SUM(CASE WHEN rating IS NULL THEN 1 ELSE 0 END) "
        f"FROM items WHERE day IN ({q}) GROUP BY source, day", days).fetchall()

    per_source = {s: {} for s in {r[0] for r in rows}}
    for src, day, n, nt, np_, nr in rows:
        per_source[src][day] = {"rows": n, "nulls": {"title": nt, "price": np_, "rating": nr}}

    for src, byday in sorted(per_source.items()):
        missing = [d for d in days if d not in byday]
        if len(missing) >= silence_days:
            findings.append({"type": "SOURCE_SILENT", "source": src, "missing_days": missing,
                             "severity": "CRITICAL",
                             "note": f"no rows for {len(missing)} of last {lookback_days} days"})
        for day in days:
            rec = byday.get(day)
            if not rec:
                continue
            if rec["rows"] < min_rows:
                findings.append({"type": "ROW_COUNT_COLLAPSE", "source": src, "day": day,
                                 "rows": rec["rows"], "expected_min": min_rows, "severity": "HIGH"})
            total = rec["rows"]
            for field, n_null in rec["nulls"].items():
                frac = n_null / total
                base = [byday[d]["nulls"][field] / byday[d]["rows"]
                        for d in days if d in byday and d != day]
                baseline = sum(base) / len(base) if base else 0.0
                status = None
                if frac >= null_fail and baseline < null_warn:
                    status = "FAIL"
                elif frac >= null_warn and baseline < null_warn:
                    status = "WARN"
                if status:
                    findings.append({"type": "SCHEMA_DRIFT", "source": src, "day": day,
                                     "field": field, "null_rate_today": round(frac, 3),
                                     "baseline_null_rate": round(baseline, 3),
                                     "severity": status})
    db.close()
    return findings

if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else "healthy"
    build_db()
    if mode == "break-silence":
        break_silence()
    elif mode == "break-schema":
        break_schema()
    out = sentinel()
    print(json.dumps({"run_at": datetime.utcnow().isoformat() + "Z",
                      "mode": mode, "findings": out}, indent=2))
    sys.exit(0 if isinstance(out, list) else 2)

Note one design choice you’d otherwise learn the hard way: the sentinel exits 0 when it finds problems. The findings are the product; a non-zero exit is reserved for the sentinel itself being broken. If you wire non-zero exit codes into your alerting instead of parsing the findings, you’ll sleep through the exact failures this tool exists to catch.

Proof: three runs, three outcomes

Every scenario below was executed before this article went up. Here is the actual output.

Baseline — healthy warehouse, zero findings (no false positives on 14 days × 5 sources, including the sparse-by-design rating field):

$ python3 drift_sentinel.py healthy
{
  "run_at": "2026-11-08T03:03:21Z",
  "mode": "healthy",
  "findings": []
}

Fault 1 — source s4 goes completely silent for two days:

$ python3 drift_sentinel.py break-silence
{
  "findings": [
    {
      "type": "SOURCE_SILENT",
      "source": "s4",
      "missing_days": ["2026-11-08", "2026-11-07"],
      "severity": "CRITICAL",
      "note": "no rows for 2 of last 7 days"
    }
  ]
}

Fault 2 — s5’s price selector dies; 67.5% of today’s prices go null while the baseline was 0%:

$ python3 drift_sentinel.py break-schema
{
  "findings": [
    {
      "type": "SCHEMA_DRIFT",
      "source": "s5",
      "day": "2026-11-08",
      "field": "price",
      "null_rate_today": 0.675,
      "baseline_null_rate": 0.0,
      "severity": "FAIL"
    }
  ]
}

Meanwhile the rating field — null about 75% of the time by design — never triggers anything in any run. That’s the per-source, per-field baseline doing its job: the sentinel doesn’t punish columns that are legitimately sparse, only columns that change.

All three runs exited 0, as designed. Wire the JSON into whatever alerting you already have; the findings carry source, day, field, measured rate, and baseline, which is exactly what the person fixing the scraper needs to start.

Operating it in production

  • Schedule the sentinel after the scrape, not with it. It must read yesterday’s complete data. A 23:55 scrape with a 23:55 sentinel races the write.
  • Warm up the baseline. The first ~7 days of a new source are baseline-building; hold alerts until the window is full, or you’ll chase ghosts.
  • Tune min_rows per source, not globally. A long-tail aggregator legitimately yields 30 rows some days; a top-100 listing page yielding 30 rows is broken. The honest version takes a per-source config dict — a ten-line change to the code above.
  • Route findings to where humans already live. The JSON goes straight into a Telegram bot message or an email digest. An unread dashboard is not alerting.
  • Pair it with the transport layer. This sentinel watches the content of what you scraped. It doesn’t protect your access — TLS fingerprinting and JA4 drift (a separate rot vector, covered in our piece on JA4 and anti-bot) live at the request layer. A production pipeline needs both: one watches whether the data is still true, the other whether you can still get in at all.

The economics of boring reliability

The commercial math here is one-directional. For an internal pricing or intelligence operation, degraded data means decisions made against a reality that shifted weeks ago — the Gartner $12.9M/year figure is the average bill for exactly this class of neglect. For an agency or data seller, the same failure compounds: you’re not just wrong, you’re wrong on the record, with holey deliverables archived in a client’s S3 bucket as evidence.

The sentinel is ~150 lines, stdlib-only, runs in milliseconds, and converts “we noticed two weeks later” into “the cron chain noticed this morning.” That asymmetry — trivial build cost against a six-figure failure class — is why every scraping pipeline we operate ships with one.

Building a scraping operation that has to stay correct for months, not survive a demo — drift detection, per-source baselines, alert routing, and the transport layer underneath — is exactly the kind of system we build for clients. If you want this without the build-it-yourself part, email [email protected].