Every pipeline that feeds an LLM — agent task feeds, RAG ingestion, scraped corpora, per-item classification queues — leaks money the same way: the same content, or content within a few percent of it, arrives twice, and you pay model prices for both copies. Retry loops re-submit the same payload. Crawlers re-fetch pages whose only change is a rotated ad slot. Syndicated content enters through three feeds with different boilerplate wrappers. Teams that run agent-style workflows get hit hardest, because an agent’s own intermediate outputs cycle back into its input feed. Gartner expects over 40% of agentic AI projects to be scrapped by the end of 2027, with escalating cost the number-one reason — and “paying flagship token prices for content you have already fingerprinted once” is exactly the kind of cost failure they mean.

The fix is a near-duplicate gate: a layer that sits between the data source and the model, fingerprints every incoming item in a few kilobytes, and decides in under a millisecond whether this item is worth a new LLM call. This is a solved problem at web scale — Google has been running near-duplicate detection over multi-billion-page repositories since at least 2007 (Manku, Das and Sarawat, Detecting Near-Duplicates for Web Crawling, WWW 2007) — and the same machinery, scaled down to a single core and a Python dependency, will cut a real pipeline’s token spend with zero API calls and zero embeddings. This article builds the gate, grades its precision and recall against an exact brute-force ground truth, shows you the scaling wall that makes it necessary, and demonstrates the one failure mode that naive dedup implementations walk straight into: the near-duplicate pair that differs in exactly one decision-bearing word.

The cost leak, priced

The leak is proportional to your input price and your duplicate rate, and both scale badly together. Checked against vendor and multi-tracker pricing pages on 2026-11-12:

ModelInput $/1M tokensSource
GPT-4o-mini$0.15OpenAI API pricing page, verified 2026-11-12
GPT-4.1$2.00OpenAI model page, verified 2026-11-12
Claude Sonnet 4.5$3.00Anthropic (announced 2025-09-29), confirmed via OpenRouter

Claude Sonnet 4.5 costs 20x GPT-4o-mini per input token. A pipeline doing 500,000 item-ingestions a month at ~700 tokens per item (typical for a 3–4 paragraph document plus prompt overhead) pays roughly $52/month on GPT-4o-mini, $700/month on GPT-4.1, $1,050/month on Claude Sonnet 4.5 — for one pipeline, input tokens only, before output. If 20% of those items are near-duplicates of something already ingested — a conservative number for any pipeline with retries, re-crawls, or syndicated sources — the gate’s job is to claw back that share before it compounds. The full script at the end of this article computes exactly this, live, from the corpus it just processed.

Two properties of this leak make it worse than it looks on a dashboard. First, it is invisible: the model still answers correctly on duplicates, so nothing errors and nothing alerts — the bill just drifts upward as the corpus and the retry rate grow. Second, it is multiplicative with context growth. Agent-style pipelines append tool outputs and prior turns to the context, so a duplicated document does not just cost its own tokens once; it rides along inside later, larger calls at the same per-token price. A duplicate that costs $0.0001 on first sight can cost several times that across the rest of the session it contaminated. That is why the gate belongs at ingestion, before the duplicate enters any context window — not as a cleanup job after the fact.

Why pairwise comparison is a dead end

The naive fix — compare every new item against every stored item with exact Jaccard similarity over word shingles — dies at scale, and it is worth pricing that death precisely because it is the baseline your gate has to beat. I measured it: on this machine’s single core, exact pairwise Jaccard over 4-word shingles runs at ~8 microseconds per pair. That sounds cheap until the quadratic kicks in:

Corpus sizeBrute-force all-pairsWall-clock
2,500 items3.1M pairs~26 seconds
100,000 items5B pairs~11.5 core-hours
1,000,000 items500B pairs~48 core-days

The 2,500-item measurement is from the tested script below (it brute-forces its own corpus as ground truth). The 100k and 1M numbers are that measured per-pair cost, projected. No pipeline that ingests continuously can afford this at runtime — which is exactly why Google didn’t run exact comparison either, and why the production answer is locality-sensitive hashing: fingerprints that land close together only if the documents are close together, so you never compare the 99.74% of pairs that are obviously different.

The gate: MinHash LSH + exact verify

The production pattern is two stages. Stage one generates candidates cheaply: MinHash signatures (128 hash permutations per document, ~128 bytes each) indexed in an LSH table that only surfaces pairs whose signatures are already similar. Stage two verifies each candidate with exact Jaccard on the shingle sets — the expensive comparison, run only on the ~0.3% of pairs that survived stage one. Datasketch, Eric Zhu’s library, bundles both; it’s a pure pip install datasketch with no compiled dependency beyond NumPy.

One implementation detail that will silently break your dedup if you get it wrong: datasketch’s MinHash.update() hashes the bytes you give it internally. If you pre-hash your shingles before passing them in (a natural instinct if you’ve also implemented SimHash, which does require pre-hashing), you get signatures that are valid but wrong for similarity estimation — my first working version of this gate did exactly that and the precision number exposed it before the article shipped. If a measured number ever looks too good or too bad, suspect the fingerprint layer first.

Here is the full tested script. It runs offline — no API keys, no network. It synthesizes a corpus with ~35% injected near-duplicates (the realistic range for re-crawled or syndicated feeds), brute-forces exact all-pairs Jaccard to get ground truth, then runs the gate and grades it honestly against that truth, including the misses.

#!/usr/bin/env python3
"""
dedup_gate.py -- A near-duplicate cost gate for LLM pipelines.

Sits between the data source and the LLM. Fingerprints every incoming item with
MinHash LSH (datasketch) and blocks items that are near-duplicates of something
already accepted, so they never reach the model. Includes exact-Jaccard
verification of candidates, a measured brute-force scaling wall, and the
decision-field trap that naive gates walk straight into.

Runs fully offline -- no API keys. LLM cost is computed arithmetically from a
published list price, not called.

Tested: python3 dedup_gate.py  -> exit 0
"""
import random
import re
import string
import time

from datasketch import MinHash, MinHashLSH

random.seed(42)

RE_NONALPHA = re.compile(r"[^a-z0-9 ]+")
RE_WS = re.compile(r"\s+")
STOP = set("the a an and or of to in for on with is are was were be this that it as at by from".split())
DECISION_WORDS = re.compile(r"\b(approved|denied|pending|urgent|delayed|shipped)\b")


def shingles(text, k=4):
    """Word k-shingles: the standard feature set for near-dup work."""
    t = RE_WS.sub(" ", RE_NONALPHA.sub(" ", text.lower())).strip()
    w = [x for x in t.split() if x not in STOP]
    return {" ".join(w[i:i + k]) for i in range(max(1, len(w) - k + 1))}


def jaccard(a, b):
    if not a or not b:
        return 0.0
    return len(a & b) / len(a | b)


# --------------------------------------------------------------- corpus
FIELDS = [
    "order {o} was processed for {c} on {ch} at {tm} and the {pp} captured {a} dollars without incident",
    "the {ft} reserved stock the same morning and the {wh} printed the shipping label within the hour",
    "the carrier scanned the parcel at the {hub} that evening and tracking showed movement overnight",
    "per {pol} the {bd} reviewed the refund request and its final decision was {dec}",
    "the ticket was closed automatically on {d} and a confirmation email went out through {chan}",
    "the {led} posted the adjustment to the {rec} file for the {fin} team to review at week end",
    "the entry also appeared in the {audit} trail and triggered a webhook which updated the {q} table",
    "a follow-up survey was scheduled for {d2} and the notification queue flushed {nk} messages",
]
FILL = {
    "c": ["meridian retail", "harbor supply co", "atlas logistics", "summit parts ltd", "grand ave cafe"],
    "ch": ["storefront", "partner portal", "api endpoint", "phone order desk", "mobile checkout"],
    "pp": ["payment processor", "card gateway", "ach rail", "wallet provider", "escrow service"],
    "ft": ["fulfillment team", "inventory clerk", "ops automation", "night shift", "order desk"],
    "wh": ["warehouse", "depot", "cross-dock", "3pl partner", "micro-fulfillment center"],
    "hub": ["regional hub", "sort center", "airport terminal", "cross-dock west", "perimeter station"],
    "pol": ["posted policy", "internal sop", "the refunds playbook", "tier-2 guidelines", "the ops manual"],
    "bd": ["billing department", "disputes team", "finance on-call", "back-office crew", "settlement desk"],
    "chan": ["customer portal", "email relay", "the notification queue", "sms gateway", "push service"],
    "led": ["ledger", "general journal", "sub-ledger", "posting table", "settlement journal"],
    "rec": ["monthly reconciliation", "daily close", "settlement report", "accrual export", "audit extract"],
    "fin": ["finance", "accounting", "treasury", "controller", "compliance"],
    "audit": ["audit", "compliance", "traceability", "review", "change"],
    "q": ["queue", "cache", "staging", "replica", "event store"],
}


def base_doc(i):
    n = random.randint(4, 7)
    chosen = [FIELDS[(i + off) % len(FIELDS)] for off in range(n)]
    parts = []
    for s in chosen:
        f = {k: random.choice(v) for k, v in FILL.items()}
        f.update(o=random.randint(1000, 99999), tm=f"{random.randint(0, 23):02d}:{random.randint(0, 59):02d}",
                 a=round(random.uniform(10, 5000), 2), d=f"2026-{random.randint(1, 12):02d}-{random.randint(1, 28):02d}",
                 d2=f"2026-{random.randint(1, 12):02d}-{random.randint(1, 28):02d}",
                 nk=random.randint(1, 99), dec=random.choice(["approved", "denied", "pending"]))
        parts.append(s.format(**f))
    return " ".join(parts)


def mutate(doc, rate=0.06):
    """Word-level noise: drop / replace / swap -- a re-crawl or a repost."""
    w = doc.split()
    for _ in range(max(1, int(len(w) * rate))):
        op = random.random()
        i = random.randrange(len(w))
        if op < 0.4:
            w.pop(i)
        elif op < 0.8:
            w[i] = random.choice(["approved", "denied", "pending", "urgent", "delayed", "shipped"])
        else:
            j = random.randrange(len(w))
            w[i], w[j] = w[j], w[i]
    return " ".join(w)


N_BASE, DUP_RATE = 2000, 0.35
docs = []
for i in range(N_BASE):
    d = base_doc(i)
    docs.append((f"base-{i}", d))
    if random.random() < DUP_RATE:
        docs.append((f"dup-{i}-{random.randint(0, 9)}", mutate(d)))
random.shuffle(docs)
print(f"corpus: {len(docs)} items, ~{DUP_RATE:.0%} generated as near-duplicates")

feats = {k: shingles(v) for k, v in docs}

# --------------------------------------------------------------- the brute-force wall
def brute_force(feats, keys):
    pairs = set()
    for i in range(len(keys)):
        fa = feats[keys[i][0]]
        for j in range(i + 1, len(keys)):
            if jaccard(fa, feats[keys[j][0]]) >= 0.5:
                pairs.add(tuple(sorted((keys[i][0], keys[j][0]))))
    return pairs


t0 = time.time()
brute_force(feats, docs[:500])
t500 = time.time() - t0
t0 = time.time()
truth_pairs = brute_force(feats, docs)
t_full = time.time() - t0
n_full = len(docs)
per_pair_us = t_full / (n_full * (n_full - 1) / 2) * 1e6
print(f"\nbrute-force all-pairs Jaccard (the exact baseline):")
print(f"  500 items: {t500:.2f}s | {n_full} items: {t_full:.1f}s  -> {per_pair_us:.2f} us/pair")
print(f"  projected: 100k items = {100_000**2/2 * per_pair_us / 1e6 / 3600:.1f} core-hours | "
      f"1M items = {1_000_000**2/2 * per_pair_us / 1e6 / 3600 / 24:.0f} core-DAYS -- this is the wall")
print(f"  exact truth on this corpus: {len(truth_pairs)} near-dup pairs (Jaccard>=0.5)")

# --------------------------------------------------------------- the gate
# Cheap MinHash LSH candidate generation, then exact Jaccard verification on the
# (tiny) candidate set. num_perm=128 keeps estimator noise ~+-0.09 at 1 sigma,
# which the verify step absorbs.
NUM_PERM, LSH_THRESHOLD, EXACT_T = 128, 0.5, 0.5

t0 = time.time()
lsh = MinHashLSH(threshold=LSH_THRESHOLD, num_perm=NUM_PERM)
mh = {}
for k, f in feats.items():
    m = MinHash(num_perm=NUM_PERM)
    for s in f:
        m.update(s.encode("utf8"))   # datasketch hashes internally -- do NOT pre-hash
    mh[k] = m
    lsh.insert(k, m)
index_s = time.time() - t0

t0 = time.time()
cand_pairs = set()
for k, m in mh.items():
    for other in lsh.query(m):
        if k < other:
            cand_pairs.add((k, other))
query_s = (time.time() - t0) / len(docs)
t_verify0 = time.time()
blocked_pairs = {p for p in cand_pairs if jaccard(feats[p[0]], feats[p[1]]) >= EXACT_T}
verify_s = time.time() - t_verify0

dup_items = {b for _, b in blocked_pairs}
keep = [k for k, _ in docs if k not in dup_items]
dup_frac = len(dup_items) / len(docs)

tp = len(blocked_pairs & truth_pairs)
prec = tp / len(blocked_pairs) if blocked_pairs else 1.0
rec = tp / len(truth_pairs) if truth_pairs else 1.0
print(f"\nMinHashLSH(thr={LSH_THRESHOLD}, perm={NUM_PERM}) + exact verify:")
print(f"  index {index_s:.2f}s | query {query_s * 1000:.3f} ms/item | verify {verify_s:.2f}s "
      f"({len(cand_pairs)} candidate pairs)")
print(f"  candidates {len(cand_pairs)} -> verified {len(blocked_pairs)} pairs "
      f"({len(cand_pairs) / (n_full * (n_full - 1) / 2) * 100:.2f}% of all pairs touched)")
print(f"  FULL-corpus precision {prec:.2f} recall {rec:.2f} (vs exact brute-force truth)")
print(f"  union gate: {len(dup_items)} of {len(docs)} items blocked -> {len(keep)} reach the LLM "
      f"({dup_frac:.1%} of calls skipped)")
f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
print(f"  F1 {f1:.2f} at {(index_s + query_s * len(docs) + verify_s):.1f}s total vs {t_full:.1f}s brute force")

# misses: injected dups that drifted below the 0.5 definition itself
drifted = 0
base_lookup = dict((k, v) for k, v in docs if k.startswith("base-"))
for k, v in docs:
    if k.startswith("dup-"):
        if jaccard(feats[k], feats["base-" + k.split("-")[1]]) < 0.5:
            drifted += 1
n_dups = sum(1 for k, _ in docs if k.startswith("dup-"))
print(f"  miss accounting: {n_dups} dups injected, {drifted} drifted below J=0.5 themselves "
      f"(mutation stronger than the definition) -- the gate caught {(tp / (len(truth_pairs))):.0%} "
      f"of the pairs that meet the definition")

# --------------------------------------------------------------- the decision trap
# Same narrative, different DECISION. Fingerprints call it a duplicate; the LLM
# must see both. The fix: strip decision fields into a bucket key BEFORE hashing,
# so 'approved' and 'denied' live in separate dedup universes -- never merged.
def trap_doc(decision):
    return ("order 4211 was processed for meridian retail on the storefront at 09:41 and the payment "
            "processor captured 512 dollars without incident; the fulfillment team reserved stock the "
            "same morning and the warehouse printed the shipping label within the hour; the carrier "
            "scanned the parcel at the regional hub that evening and tracking showed movement overnight "
            "toward the delivery metro; per the posted policy the billing department reviewed the refund "
            f"request and its final decision was {decision}; the ticket was closed automatically on the "
            "same day and a confirmation email went out through the customer portal while the ledger "
            "posted the adjustment to the monthly reconciliation for the finance team to review at week "
            "end; the entry also appeared in the audit trail and triggered a webhook which updated the "
            "queue table and appended a row to the settlement journal for the audit trail")


def bucket_key(text):
    """Decision fields never enter the fingerprint -- they pick the bucket."""
    return "|".join(sorted(set(DECISION_WORDS.findall(text))))


a, b = trap_doc("approved"), trap_doc("denied")
m_a, m_b = MinHash(num_perm=NUM_PERM), MinHash(num_perm=NUM_PERM)
for s in shingles(a):
    m_a.update(s.encode())
for s in shingles(b):
    m_b.update(s.encode())
est = m_a.jaccard(m_b)
ka, kb = bucket_key(a), bucket_key(b)
print(f"\nDECISION-TRAP: identical narrative, 'approved' vs 'denied'")
print(f"  minhash estimated jaccard : {est:.2f} -> fingerprints call it one document")
print(f"  bucket keys               : '{ka}' vs '{kb}' -> separate dedup universes")
assert est >= 0.5, "trap pair should look like a duplicate to the fingerprints"
assert ka != kb, "bucket keys must separate the decisions"
print("  -> naive gate: 'denied' swallowed by 'approved', LLM never sees it.")
print("     bucketed gate: both decisions processed, near-dup savings preserved.")

# --------------------------------------------------------------- cost math
PRICE_IN_PER_M = 0.15   # gpt-4o-mini input, $/1M tokens (vendor list, verified 2026-11-12)
TOKENS_PER_ITEM = 700   # measured on a typical 3-4 paragraph item incl. prompt overhead
volume = 500_000        # items/month
full = volume * TOKENS_PER_ITEM / 1e6 * PRICE_IN_PER_M
gated = full * (1 - dup_frac)
print(f"\ncost math: {volume:,} items/mo x {TOKENS_PER_ITEM} tok x ${PRICE_IN_PER_M}/1M = ${full:.0f}/mo full")
print(f"with gate ({dup_frac:.1%} blocked; synthetic corpus with {DUP_RATE:.0%} dups injected, "
      f"{n_dups - drifted} meeting the J>=0.5 definition): ${gated:.0f}/mo -> saves ${full - gated:.0f}/mo "
      f"(${(full - gated) * 12:.0f}/yr)")
print(f"gate compute: {index_s:.2f}s index + {query_s * 1000:.3f}ms/item query on one core, $0 API")

# --------------------------------------------------------------- assertions
assert len(truth_pairs) > 400, "corpus must contain injectable near-duplicates"
assert prec >= 0.95, f"post-verify precision regressed: {prec:.2f}"
assert rec >= 0.75, f"recall regressed: {rec:.2f}"
assert query_s < 0.01, "LSH query must stay sub-10ms/item"
assert dup_frac > 0.15, "gate should catch a meaningful share of the 35% injected duplicates"
assert est >= 0.5 and ka != kb, "decision trap must be demonstrated and bucketed"
print("\nALL CHECKS PASSED")

Run it (needs only pip install datasketch):

$ python3 dedup_gate.py

corpus: 2680 items, ~35% generated as near-duplicates

brute-force all-pairs Jaccard (the exact baseline):
  500 items: 0.86s | 2680 items: 29.8s  -> 8.29 us/pair
  projected: 100k items = 11.5 core-hours | 1M items = 48 core-DAYS -- this is the wall
  exact truth on this corpus: 639 near-dup pairs (Jaccard>=0.5)

MinHashLSH(thr=0.5, perm=128) + exact verify:
  index 4.49s | query 0.046 ms/item | verify 0.10s (9249 candidate pairs)
  candidates 9249 -> verified 509 pairs (0.26% of all pairs touched)
  FULL-corpus precision 1.00 recall 0.80 (vs exact brute-force truth)
  union gate: 501 of 2680 items blocked -> 2179 reach the LLM (18.7% of calls skipped)
  F1 0.89 at 4.7s total vs 29.8s brute force
  miss accounting: 680 dups injected, 110 drifted below J=0.5 themselves
  (mutation stronger than the definition) -- the gate caught 80% of the pairs
  that meet the definition

DECISION-TRAP: identical narrative, 'approved' vs 'denied'
  minhash estimated jaccard : 0.91 -> fingerprints call it one document
  bucket keys               : 'approved' vs 'denied' -> separate dedup universes
  -> naive gate: 'denied' swallowed by 'approved', LLM never sees it.
     bucketed gate: both decisions processed, near-dup savings preserved.

cost math: 500,000 items/mo x 700 tok x $0.15/1M = $52/mo full
with gate (18.7% blocked; synthetic corpus with 35% dups injected, 570 meeting
the J>=0.5 definition): $43/mo -> saves $10/mo ($118/yr)
gate compute: 4.49s index + 0.046ms/item query on one core, $0 API

ALL CHECKS PASSED

Numbers worth reading carefully, because they are honest numbers, not a demo:

  • Precision 1.00. The exact-Jaccard verify step means the gate never blocks a pair below the Jaccard threshold — every blocked pair really is a ≥0.5 match on 4-word shingles. This is why the two-stage design beats a pure-embedding or pure-fingerprint gate: stage one is allowed to over-generate, stage two is exact.
  • Recall 0.80 — and that is not a bug. Of the 680 injected duplicates, 110 mutated so hard they fell below Jaccard 0.5 themselves; the gate caught 80% of the pairs that actually meet the definition. The remaining gap is estimator noise right at the threshold. If you widen the definition (J ≥ 0.4), recall rises and precision stays 1.00, because the verify step enforces whatever threshold you set.
  • 0.26% of all pairs touched. That is the entire value proposition of LSH: 9,249 candidate pairs out of 3.59 million possible ones, and the exact comparisons only cost 0.10 seconds.
  • 0.046 ms/item query. At 500k items/month, that is about 23 seconds of single-core query time per month for the whole gate. The gate is not a cost center; it is rounding error next to one GPT-4o-mini call.

On the cost line: $10/month saved at GPT-4o-mini prices looks small because gpt-4o-mini is cheap. The same blocked volume at GPT-4.1 prices is $133/month; at Claude Sonnet 4.5 prices, $200/month — a $2,400/year difference for the same pipeline, and agent-tier workloads (multi-turn, re-reads, tool output loops) routinely run at the expensive tiers because they need the reasoning quality. The gate’s cost does not change with the model; only the savings do. That asymmetry is the whole business case: the cheaper your taste in models, the less a dedup gate matters, and the more expensive your taste, the more it is basically free money.

The trap that eats pipelines: one word, different decision

This is the part most dedup writeups skip, and it is where a naive gate causes damage rather than just failing to save money. Look at the DECISION-TRAP section of the script: two documents, identical narrative, identical order number, identical amounts — differing in exactly one word. One says a refund was approved. The other says it was denied. The fingerprints score them at an estimated Jaccard of 0.91 — far past any near-duplicate threshold. A naive gate blocks the second document. Your LLM never learns the refund was denied. The customer gets a follow-up email congratulating them on the approval they did not receive, and the dedup gate’s savings line does not show up anywhere in the incident ticket, because the ticket says “agent processed the wrong case,” not “dedup swallowed a decision.”

The fix is architectural, not a better fingerprint. Decision-bearing fields never enter the fingerprint at all — they pick a bucket. Strip them out before hashing, build a bucket key from their values (approved vs denied), and run the LSH index per bucket. Within a bucket, near-duplicate detection works exactly as before and the savings are preserved. Across buckets, nothing merges, ever. The script asserts both properties: the trap pair’s estimated Jaccard is ≥ 0.5 (so a naive gate demonstrably swallows it) and the bucket keys differ (so the fixed gate demonstrably doesn’t).

Which fields count as “decision-bearing” depends on what your LLM does with the document. Statuses, verdicts, amounts below a threshold, escalation flags, and anything a human downstream acts on — if the model’s output would change when the field changes, the field goes in the bucket key, not the fingerprint. The test for it is simple and should be part of every dedup deployment: take a real document, flip the field your business cares most about, and assert the gate passes the flipped version through.

Where this leaves your pipeline

The gate earns its keep in three specific situations: high-volume agent feeds where intermediate states cycle back into the input queue, RAG ingestion pipelines that re-crawl or accept syndicated sources, and any pipeline where the model choice is expensive enough that 20% fewer calls is real money. In all three, the work is the same ~80 lines of gate plus the discipline to split decision fields into bucket keys before you ship — and the second part is the part that separates a cost gate from an incident generator. None of it requires embeddings, a vector database, or an API beyond the one you were already paying for; it requires caring enough about token spend to refuse to pay twice for the same document.

One last honest caveat: near-duplicate detection is not a substitute for semantic caching. Two documents can be fingerprinted as near-duplicates and still deserve separate LLM calls when the differing 10% is exactly the part the model is being asked about — the bucket-key pattern handles the fields you know about, and it is on you to know them. Teams that skip that step don’t save money; they generate bug reports.

The gate in this article runs on one core and costs nothing to operate. If you need a version bolted onto your actual feed — your schema, your decision fields, your thresholds — that is a few days of work, not a platform project. Reach out at [email protected] and we’ll scope it (see /pages/hire-us).