If you read this blog, you’ve probably read the other side of this arms race from us: the JA4 TLS fingerprint that outed a scraper’s handshake, the HTTP/2 SETTINGS layer Akamai checks when JS is clean, the session pool that keeps a login alive across hundreds of requests. Those posts are written from the attacker’s chair because that’s where the engineering leverage is when you collect data.

There’s a second market on the same shelf, and almost nobody writes for it: every business that’s being scraped, form-spammed, and path-probed and has no idea which of the 53% of requests hitting it are not customers. Imperva’s 2026 Bad Bot Report puts automated traffic at more than 53% of all web traffic in 2025, up from 51% the year before (imperva.com/blog/bad-bot-report-2026); the 2025 edition already counted 37% as bad bots — the hostile slice, not the search crawlers (imperva.com 2025 Bad Bot Report). If you sell engineering services, that’s a bigger addressable market than scraping itself: every scraped site is a potential client for the thing this article builds.

So this is the defense-side counterpart, written with the same honesty about what the code can and can’t do. It builds a tripwire layer: ~250 lines of stdlib Python that fingerprints automated traffic by what bots physically cannot resist doing — filling every field, fetching every path, ignoring every instruction humans don’t even see. It ships with a test harness of 37 assertions that runs the real server over real HTTP. All 37 pass, exit code 0, output shown below.

The inversion that makes tripwires work

Detection usually means reading signals and guessing: was that 4-ms form fill a human? Is that user-agent string lying? Tripwires invert the game. Instead of interpreting ambiguous behavior, you seed the environment with things that have exactly one legitimate interpretation:

  • A form field no human can see or submit is filled → bot.
  • A path declared off-limits in robots.txt gets hit → bot.
  • A link that only your templates can generate (unguessable HMAC URL) gets fetched → bot.

Each is a positive signal with a near-zero false-positive rate. No scoring debate, no behavioral ML, no browser fingerprinting arms race. This is old, proven craft — Cloudflare’s founders built Project HoneyPot on exactly this model in 2004, and in March 2025 Cloudflare shipped AI Labyrinth, which feeds non-compliant AI crawlers an auto-generated maze of decoy pages. The technique is not exotic. It’s just underdeployed, because most teams buy a WAF before they write 250 lines of Python.

One design rule before the code: every tripwire must be safe to false-positive on. A lead gen form that blocks one real customer costs more than a thousand spam rows. So the layer scores signals and answers with two tiers — LOG (record, allow) and BLOCK (403) — and only signals that are physically impossible for humans reach BLOCK.

Component 1: trap paths — robots.txt as the attacker’s do-not-hit list

robots.txt is a strange artifact: it’s the one file on your site whose only readers that obey it are well-behaved crawlers — exactly the traffic you don’t want to punish. Everyone else (vulnerability scanners, path brute-forcers, careless scrapers) hits /wp-admin, /.env, /.git/config and xmlrpc.php on every site they touch, whether or not it’s WordPress. Those paths, published as Disallow: lines, become tripwires:

TRAP_RE = re.compile(
    r"^/(?:wp-admin|wp-login\.php|wp-content|\.env|\.git|phpmyadmin"
    r"|xmlrpc\.php|\.aws|config\.json|admin\.php)",
    re.IGNORECASE,
)

def is_trap_path(path: str) -> bool:
    return bool(TRAP_RE.match(path))

The robots.txt you publish is the same list, which does double duty: compliant crawlers (Googlebot, GPTBot, ClaudeBot) read it, honor it, and never trip the wire — so they stay whitelisted by their own compliance, not by trusting their user-agent string. Non-compliant fetchers self-identify on the first request. That’s the inversion: you made the attacker’s own reconnaissance tooling into your sensor.

Two operational rules. First, never serve real content at a trap path — these paths 403 before any handler runs. Second, don’t put a trap path anywhere a real user could ever navigate; every regex above is unreachable through normal UI by construction.

Component 2: seed fields — the form field only a bot would fill

Classic honeypot forms use an obvious input named bot_check and CSS display:none. Modern spam kits parse the DOM and skip fields that are hidden or suspiciously named — and the cheap ones are cheap because someone maintains a blacklist of known honeypot names. The fix is to make the trap indistinguishable from a real CRM auto-field: named Company_<hmac8>, positioned off-canvas (not display:none, which screen-reader and DOM-filter heuristics both key on), deterministic per month so a spam kit that learns today’s field name is stale in 30 days:

def seed_field_name(form_id: str, month: str = None) -> str:
    """Deterministic per (form, month): rotate monthly so reuse dies fast."""
    stamp = month or datetime.now(timezone.utc).strftime("%Y%m")
    digest = hmac.new(SECRET, ("%s:%s" % (form_id, stamp)).encode(),
                      hashlib.sha256).hexdigest()
    return "Company_" + digest[:8]

def render_protected_field(form_id: str) -> str:
    name = seed_field_name(form_id)
    return ('<div aria-hidden="true" style="position:absolute;left:-9999px;'
            'top:-9999px;height:0;overflow:hidden">'
            '<label for="%s">Company</label>'
            '<input id="%s" name="%s" type="text" tabindex="-1" autocomplete="off">'
            "</div>" % (name, name, name))

Why this has a near-zero false-positive rate: a human on the page cannot see the field (it’s 9,999 pixels off-canvas with zero height), cannot tab into it (tabindex="-1"), and autofill won’t touch it (autocomplete="off" on a name no profile knows). The only way Company_xxxx appears in a POST body is if the submitter enumerated all form fields from the DOM and filled them — which is, definitionally, a bot behavior. Note the asymmetry with the attacker posts: the scraper writer’s checklist says “fill every required field”; here that exact obedience is the confession.

The third sensor catches crawlers that harvest links indiscriminately. You render invisible anchors pointing at /_decoy/<sig>, where the signature is an HMAC of the path and the current day:

def decoy_link(base_url: str, path: str = "/") -> str:
    day = datetime.now(timezone.utc).strftime("%Y%m%d")
    sig = hmac.new(SECRET, ("decoy:%s:%s" % (path, day)).encode(),
                   hashlib.sha256).hexdigest()[:16]
    return '<a href="%s/_decoy/%s" style="display:none"></a>' % (base_url, sig)

def match_decoy(path: str) -> bool:
    m = re.match(r"^/_decoy/([0-9a-f]{16})$", path)
    if not m:
        return False
    for off in (0, 1):                      # accept yesterday + today (roll-over safe)
        day = datetime.now(timezone.utc).strftime("%Y%m%d")
        if off:
            day = str(int(day) - 1)
        expected = hmac.new(SECRET, ("decoy:%s:%s" % ("/", day)).encode(),
                            hashlib.sha256).hexdigest()[:16]
        if hmac.compare_digest(m.group(1), expected):
            return True
    return False

“Keyless” is the important property. A honeypot URL from a hardcoded list (/decoy-page-1) can be probed, enumerated, and shared. This one cannot: nobody — including someone who reads your JavaScript — can compute tomorrow’s signature, because the secret never leaves the server. The only entities that can ever possess a valid decoy URL are your own templates at render time, which means a hit is a link harvester, full stop. The one-day rotation with yesterday-grace means you never block a legit anything at midnight UTC.

The verdict: one scoring function, two tiers

All three sensors feed one function, plus a soft signal — empty user-agent — that is not blocking-grade:

_DECLARED_RE = re.compile(
    r"^(GPTBot|ClaudeBot|Googlebot|bingbot|OAI-SearchBot|PerplexityBot)/")

def get_verdict(request=None, *, path="/", ua="", form=None) -> dict:
    score, signals = 0, []
    form = form or {}
    if form:
        current = seed_field_name("contact")
        if any(k == current or k.startswith("Company_") for k in form):
            score += 100; signals.append("seed_field_filled")
    if is_trap_path(path):
        score += 100; signals.append("trap_path_hit")
    if match_decoy(path):
        score += 100; signals.append("decoy_hit")
    if (not ua) or ua.strip().lower() in {"", "-", "unknown"}:
        score += 25;  signals.append("empty_ua")
    if _DECLARED_RE.match(ua):
        score = 0;    signals.append("declared_bot_whitelisted")
    return {"score": score, "action": "BLOCK" if score >= 40 else "LOG",
            "signals": signals, "path": path, "ts": time.time()}

The threshold logic encodes the false-positive economics: the three tripwires score 100 (BLOCK), the empty-UA heuristic scores 25 (LOG) because plenty of legitimate CLI tools and scripts send no UA and blocking them costs you API consumers. Declared AI and search bots are whitelisted — deliberately, because they’re traffic you want, and robots.txt compliance keeps them off the trap paths anyway.

Honest disclosure, because this detail matters: the demo whitelists those bots by user-agent string alone, and UA strings are trivially forged — that’s the whole point of the JA4 post. Production code should verify a declared Googlebot by reverse-DNS of the connecting IP (*.googlebot.com, *.search.msn.com) the same way Google tells you to, and treat verification failure as a scoring signal, not an instant block. The architecture supports it; the demo keeps stdlib-only and skips DNS.

The event store — and the SQLite gotcha worth the whole article

Signals are worthless if you can’t aggregate them. The store is SQLite, WAL mode, one shared connection behind a lock — and the one shared connection part is a real engineering lesson. The first version opened a fresh connection per write, which is the idiomatic SQLite pattern for file databases. The 20-thread concurrency test promptly died with no such table: events, because for :memory: databases every connection gets its own empty database — and even for file DBs, connection-per-write under a thread burst is where lock contention lives. The fixed class:

class EventStore:
    """Single shared SQLite connection guarded by a lock: one writer at a
    time, and ':memory:' works because every operation sees the same
    connection."""

    def __init__(self, path: str = "tripwire.db"):
        self._lock = threading.Lock()
        self.con = sqlite3.connect(path, check_same_thread=False)
        if path != ":memory:":
            self.con.execute("PRAGMA journal_mode=WAL")
            self.con.execute("PRAGMA synchronous=NORMAL")
        self.con.executescript(DDL)
        self.con.commit()

    def record(self, v, ip="", ua=""):
        with self._lock:
            self.con.execute(
                "INSERT INTO events (ts, ip, path, ua, score, action, signals)"
                " VALUES (?,?,?,?,?,?,?)",
                (v["ts"], ip, v["path"], ua, v["score"], v["action"],
                 json.dumps(v["signals"])))
            self.con.commit()

    def top_offenders(self, since=0, limit=10):
        with self._lock:
            rows = self.con.execute(
                "SELECT ip, COUNT(*) c, MAX(score) mx FROM events WHERE ts >= ?"
                " GROUP BY ip ORDER BY c DESC LIMIT ?", (since, limit)).fetchall()
        return [{"ip": r[0], "hits": r[1], "max_score": r[2]} for r in rows]

That’s the report you sell to the client: top_offenders() after a week of traffic is a ranked list of hostile IPs with the exact signal that caught each. One shared connection with check_same_thread=False plus a lock is correct at this scale (single small VPS, low-thousands of events/day); if you outgrow it, the interface doesn’t change.

Proof: the harness, the exit code, and what it actually verifies

Every claim above is backed by test_tripwire.py, a harness that boots the real ThreadingHTTPServer on 127.0.0.1 and drives it with urllib like a real client — including a simulated scraper that GETs the form, parses the seed field name out of the HTML, and POSTs it back. The full source ships with this article’s scratch directory; the run’s tail:

== live server ==
  PASS  GET / -> 200
  PASS  protected form 200 with seed field
  PASS  protected form contains decoy link
  PASS  trap path blocked 403
  PASS  bot can scrape seed field name
  PASS  seed-field POST blocked 403
  PASS  verdict cites seed_field_filled
  PASS  clean POST accepted 200
  PASS  decoy hit blocked 403
  PASS  empty UA passes (LOG tier, not blocked)
  PASS  declared bot allowed
== event store ==
  PASS  events recorded
  PASS  all BLOCK rows scored >= 40
  PASS  signals serialized correctly
  PASS  top offender = trap hitter (6 hits)
== concurrency ==
  PASS  500 concurrent writes -> 500 stored

37 passed, 0 failed
EXIT CODE: 0

What those 37 assertions actually pin down: determinism and monthly rotation of seed field names; six trap paths matching and two legit paths not matching; decoy signature validity today, validity yesterday (roll-over), and rejection of a forged signature; every scoring rule in isolation; the live server blocking the trap path, the seed-field POST, and the decoy fetch with 403 — while the clean form gets 200 and the empty-UA crawler gets allowed (the LOG tier doing its job); the event store recording all four blocks with correct signal serialization; and 20 threads × 25 writes landing exactly 500 rows. No API keys, no external services — the whole thing is stdlib. If a future refactor breaks any claim in this article, the harness fails loudly instead of the client’s lead pipeline failing quietly.

Where it hooks into a real stack, and what it doesn’t do

Deployment is a middleware-shaped function: run get_verdict() at the top of the request handler with path and the UA header, check form on POST routes, render render_protected_field() into your form templates and decoy_link() once per page, and answer 403 when action == "BLOCK". On Flask that’s a before_request hook plus two template partials; on nginx-fronted static sites, the trap-path tier maps to a few location blocks returning 403 — the app layer only needs the form sensors.

Equally important, the boundaries. This layer catches sloppy automation: form spammers, vulnerability scanners, scrapers that ignore robots.txt and fill every field. A careful attacker with a real browser, rotated residential IPs, and a DOM-aware filler that respects hidden fields will pass every tripwire here — stopping that class requires the network-layer signals (TLS/JA4, HTTP/2 frame ordering, headless-browser artifacts) covered in the earlier posts, or a commercial bot manager, and this article won’t pretend otherwise. That’s exactly why the design scores instead of binary-blocking: tripwires are one high-precision input into a decision, not the whole decision. And the defense posture is a cat-and-mouse like any other — Cloudflare had to rebuild its AI-crawler controls as recently as 2025, and detector craft rotates constantly. Whoever you hire to build this should be the same people who demonstrably know how the attack side works.

That’s the pitch in one paragraph: we’ve published the attacker’s playbook — TLS fingerprints, session pools, mobile-API reversing — because building the defense that actually catches those clients requires knowing them from the inside. The tripwire layer above is the smallest sellable unit of that expertise: stdlib Python, tested end-to-end, deployable on any stack in an afternoon.

Building a site that’s being scraped or spammed and want a tripwire layer tuned to your traffic — or the full detection stack behind it? Email [email protected] or see the /pages/hire-us page.