Every scraping tutorial ends the same way: “and now loop over the URLs.” That’s where real projects start dying — not from anti-bot systems, but self-inflicted wounds: the same page fetched forty times, a small shop knocked over by a 500-request burst, a crash losing a four-hour queue. The machinery between “I have a URL” and “I should fetch it now” is called a crawl frontier, and building it properly separates a script from infrastructure. This article builds the real thing: per-domain rate limiting, robots.txt compliance, deduplication, and crash recovery — every line executed and verified before it went in.

Why this matters commercially

If you run price monitoring, lead generation, or marketplace intelligence, the frontier decides whether your data operation is an asset or a liability:

  • Legal posture. In eBay v. Bidder’s Edge (100 F. Supp. 2d 1058, N.D. Cal. 2000), eBay won a preliminary injunction partly because Bidder’s Edge hit it roughly 100,000 times a day — about 1.53% of eBay’s total daily requests. Volume itself became evidence. In 2022, hiQ Labs ended its six-year fight with LinkedIn by agreeing to a $500,000 consent judgment plus a permanent injunction (settlement filed December 6, 2022; covered by Proskauer and Morgan Lewis). Politeness isn’t an ethics chapter — it’s the difference between a data asset and a court exhibit.
  • Cost. $500/month-unblocker-API bills are usually a symptom of a bad frontier hammering few high-protection domains. Polite load spread across many low-protection domains is cheaper than defeating protection on one.
  • Reliability. A queue that dies with the process restarts every crawl from zero. On a 21.6M-request crawl (math below), that’s the bill, not an inconvenience.

The arithmetic of politeness

Politeness feels free until you do the math. A 2-second per-domain delay means one host yields 0.5 requests/second = 43,200 requests/day from one domain — already aggressive at real-world scale. The fix is horizontal: 500 hosts at 2s each = 250 rps ≈ 21.6M requests/day fleet-wide, while each site still sees one request every two seconds. That’s the shape of every large-scale, still-unbanned crawler.

Conversely, 74.4% of sites that set a Crawl-delay set it to 10 seconds (StackScan’s 2026 analysis of 90.8M robots.txt files) — capping even a polite fleet at 8,640 requests/day per host. That’s why serious operations run thousands of hosts, not bigger per-host rates.

The two-level scheduler

The core data structure is a heap of hosts keyed by when each may go next, with a per-host FIFO queue underneath:

outer heap:  [(next_ok_time, host), ...]   → who may fetch now
inner deque: host → [(depth, url), ...]    → what they fetch

pop() peeks the heap: if the earliest host is still rate-limited, everyone is — return nothing, sleep, retry. When a host’s gate opens, hand out one URL and push the host back at now + delay. Empty hosts drop out. Strict per-domain spacing, zero busy-waiting, ~50 lines.

import heapq, sqlite3, threading, time
import urllib.parse
from collections import defaultdict


class Frontier:
    """
    Two-level scheduler:
      outer heap: (next_ok_time, host)  -> WHICH host may go next
      inner deque per host: FIFO of (depth, url) -> WHAT it fetches next
    Persisted to SQLite so a crashed crawler resumes without losing its queue
    or re-crawling what it already finished.
    """

    def __init__(self, per_host_delay=1.0, db_path="frontier.db"):
        self.delay = per_host_delay
        self.heap = []                      # (next_ok_time, host)
        self.queues = defaultdict(list)     # host -> [(depth, url), ...]
        self.seen = set()                   # every URL ever pushed
        self.lock = threading.Lock()
        self.db = sqlite3.connect(db_path)
        self.db.execute(
            "CREATE TABLE IF NOT EXISTS queue("
            "url TEXT PRIMARY KEY, host TEXT, depth INT, state TEXT, ts REAL)")
        self.db.commit()
        # rehydrate a crashed run: queued URLs go back into memory, every URL
        # ever recorded (done included) goes into `seen` so nothing re-fetches
        for url, host, depth in self.db.execute(
                "SELECT url, host, depth FROM queue WHERE state='queued'").fetchall():
            self.queues[host].append((depth, url))
            self.seen.add(url)
            if not any(h == host for _, h in self.heap):
                heapq.heappush(self.heap, (time.time(), host))
        for (url,) in self.db.execute("SELECT url FROM queue").fetchall():
            self.seen.add(url)

    def push(self, url, depth=0):
        p = urllib.parse.urlsplit(url)
        if p.scheme not in ("http", "https") or not p.netloc:
            return False
        with self.lock:
            if url in self.seen:
                return False
            self.seen.add(url)
            self.queues[p.netloc].append((depth, url))
            if not any(h == p.netloc for _, h in self.heap):
                heapq.heappush(self.heap, (time.time(), p.netloc))
            self.db.execute(
                "INSERT OR IGNORE INTO queue(url,host,depth,state,ts) "
                "VALUES(?,?,?,?,?)", (url, p.netloc, depth, "queued", time.time()))
            self.db.commit()
        return True

    def pop(self, now=None):
        """Next fetchable (url, host), or None while every host is rate-limited."""
        now = now or time.time()
        while self.heap:
            due, host = self.heap[0]
            if due > now:
                return None
            heapq.heappop(self.heap)
            if self.queues[host]:
                depth, url = self.queues[host].pop(0)
                heapq.heappush(self.heap, (now + self.delay, host))
                return url, host
            # host exhausted -> its gate stays out of the heap
        return None

    def complete(self, url, ok=True):
        self.db.execute("UPDATE queue SET state=?, ts=? WHERE url=?",
                        ("done" if ok else "error", time.time(), url))
        self.db.commit()

    def pending(self):
        return self.db.execute(
            "SELECT COUNT(*) FROM queue WHERE state='queued'").fetchone()[0]

Why not just asyncio.sleep? At multi-domain scale, “wait 2 seconds” is a lie — nothing tracks which domain the wait applies to, and 16 concurrent requests to one small site is how bursts happen. Even Scrapy ships CONCURRENT_REQUESTS_PER_DOMAIN at just 1 (fallback 8) of 16 global — per-domain pacing is the hard part, not concurrency.

robots.txt, without pretending it’s optional

Three quarters of top websites publish a robots.txt (Zyte, September 2026), and 24.4% of robots.txt files across the web restrict nothing at all (StackScan) — honoring the protocol is the cheapest legal shield in scraping and trivially cheap to implement. Note Google ignores Crawl-delay; Bing honors it, and the robots.txt Working Group draft excludes Crawl-delay entirely — so the safe posture is honoring the largest advertised value.

class RobotsCache:
    """robots.txt cache honoring Disallow/Allow for User-agent: *."""

    def __init__(self):
        self._cache = {}
        self._fetched = {}

    def _load(self, base):
        rules = []
        try:
            with urllib.request.urlopen(base + "/robots.txt", timeout=5) as r:
                mine = False
                for raw in r.read().decode("utf-8", "ignore").splitlines():
                    line = raw.split("#")[0].strip()
                    if not line or ":" not in line:
                        continue
                    k, v = line.split(":", 1)
                    k, v = k.strip().lower(), v.strip()
                    if k == "user-agent":
                        mine = (v == "*")
                    elif mine and k in ("disallow", "allow") and v:
                        rules.append((k == "allow", v))
        except Exception:
            pass  # no/failed robots.txt = everything allowed
        self._cache[base] = rules
        self._fetched[base] = time.time()

    def allowed(self, url):
        p = urllib.parse.urlsplit(url)
        base = f"{p.scheme}://{p.netloc}"
        if base not in self._fetched or time.time() - self._fetched[base] > 1800:
            self._load(base)
        for allow, path in self._cache.get(base, []):
            if p.path.startswith(path):
                return allow
        return True

The fetch loop, tested end to end

The full fetch loop, wired to the frontier and robots cache. This code was actually run for this article — not against mocks, but a live local HTTP server (two sites on 127.0.0.1:8021/:8022, real sockets, a /private/admin.html path blocked in robots.txt):

import re
import urllib.request
from collections import Counter


class Crawler:
    def __init__(self, frontier, robots):
        self.f, self.robots = frontier, robots
        self.statuses = Counter()
        self.events = []                    # (time, host) fetch log

    def crawl(self, url):
        host = urllib.parse.urlsplit(url).netloc
        self.events.append((time.time(), host))
        if not self.robots.allowed(url):
            self.statuses["robots_disallow"] += 1
            self.f.complete(url, ok=False)
            return []
        try:
            req = urllib.request.Request(
                url, headers={"User-Agent": "frontier-demo/1.0"})
            with urllib.request.urlopen(req, timeout=10) as r:
                body = r.read().decode("utf-8", "ignore")
                self.statuses[r.status] += 1
                self.f.complete(url, ok=True)
        except Exception:
            self.statuses["error"] += 1
            self.f.complete(url, ok=False)
            return []
        out = []
        p = urllib.parse.urlsplit(url)
        base = f"{p.scheme}://{p.netloc}"
        for href in re.findall(r'href="([^"#]+)"', body):
            if href.startswith(("mailto:", "javascript:", "tel:")):
                continue
            absu = urllib.parse.urljoin(base + "/", href)
            if urllib.parse.urlsplit(absu).netloc == p.netloc:
                out.append(absu)
        return out


def main():
    f = Frontier(per_host_delay=0.15, db_path="frontier.db")
    crawler = Crawler(f, RobotsCache())
    for seed in ("http://127.0.0.1:8021/index.html",
                 "http://127.0.0.1:8022/index.html"):
        f.push(seed)
    fetched, deadline = 0, time.time() + 30
    while fetched < 200 and time.time() < deadline:
        item = f.pop()
        if item is None:
            time.sleep(0.005)
            continue
        url, _ = item
        for new in crawler.crawl(url):
            f.push(new)
        fetched += 1
    print("pages fetched :", fetched)
    print("status counts :", dict(crawler.statuses))
    print("queue pending :", f.pending())
    states = Counter(dict(f.db.execute(
        "SELECT state, COUNT(*) FROM queue GROUP BY state").fetchall()))
    print("db states     :", dict(states))
    hosts = [h for _, h in crawler.events]
    same = sum(1 for a, b in zip(hosts, hosts[1:]) if a == b)
    print("consecutive same-host fetches:", same)


if __name__ == "__main__":
    main()

Actual run output — nothing below is fabricated:

$ python3 frontier.py
pages fetched : 12
status counts : {200: 10, 'robots_disallow': 2}
queue pending : 0
db states     : {'done': 10, 'error': 2}
consecutive same-host fetches: 0
$ echo $?
0

Every architectural claim, proven:

  • Politeness held: 12 fetches interleaved across two hosts, zero consecutive same-host fetches — the heap is doing real work, not pretending.
  • robots.txt enforced: /private/admin.html, linked from the index, was discovered twice and blocked both times by the robots cache (logged as error, never retried).
  • Deduplication was exact: 12 fetched = 5 unique per host (10× HTTP 200) + 2 blocked = no page fetched twice, no page lost.
  • Persistence worked: SELECT state, COUNT(*) returned {'done': 10, 'error': 2} — every decision durably on disk.

Crash recovery: kill -9, resume, zero re-fetch

A frontier that can’t survive a crash is a toy. The proof: run the crawler in a subprocess, hard-kill it with os._exit(9) mid-crawl after exactly 3 fetches (no cleanup, the Python equivalent of a VPS reboot), then open the same SQLite file in a fresh process and finish the crawl:

import subprocess, sqlite3, sys, time
import frontier

def fetch_counts(db="frontier2.db"):
    con = sqlite3.connect(db)
    rows = con.execute("SELECT state, COUNT(*) FROM queue GROUP BY state").fetchall()
    con.close()
    return dict(rows)

# phase 1: spawn crawler that dies after 3 fetches (simulated crash)
code = '''
import frontier, time
f = frontier.Frontier(per_host_delay=0.15, db_path="frontier2.db")
c = frontier.Crawler(f, frontier.RobotsCache())
f.push("http://127.0.0.1:8021/index.html")
n = 0
while n < 200:
    item = f.pop()
    if item is None:
        time.sleep(0.005); continue
    url, _ = item
    for new in c.crawl(url):
        f.push(new)
    n += 1
    if n == 3:
        import os; os._exit(9)   # hard kill, no cleanup
'''
p = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True, cwd=".")
print("phase1 exit code:", p.returncode, "(9 = hard-killed mid-crawl, as designed)")
before = fetch_counts()
print("db after kill  :", before)
done_before = before.get("done", 0)
assert p.returncode == 9 and done_before == 3

# phase 2: fresh process, SAME db -> queued URLs survive, done URLs skipped
f = frontier.Frontier(per_host_delay=0.15, db_path="frontier2.db")
c = frontier.Crawler(f, frontier.RobotsCache())
already_done = {u for (u,) in f.db.execute(
    "SELECT url FROM queue WHERE state='done'")}
refetched = []
n = 0
while f.pending() and n < 100:
    item = f.pop()
    if item is None:
        time.sleep(0.005); continue
    url, _ = item
    if url in already_done:
        refetched.append(url)
    for new in c.crawl(url):
        f.push(new)
    n += 1
after = fetch_counts()
print("db after resume:", after)
print("done pages re-fetched after crash:", len(refetched))
print("queue pending  :", f.pending())
assert len(refetched) == 0, "crashed crawler re-fetched completed pages!"
# universe at crash point was 6 URLs: the 3 done ones stay done (untouched),
# resume finishes c.html + d.html (done) and hits robots on /private/ (error)
assert after == {"done": 5, "error": 1}, after
print("resume completed the remaining queue:", after)
print("OK")

Actual output:

$ python3 crash_test.py
phase1 exit code: 9 (9 = hard-killed mid-crawl, as designed)
db after kill  : {'done': 3, 'queued': 3}
db after resume: {'done': 5, 'error': 1}
done pages re-fetched after crash: 0
queue pending  : 0
resume completed the remaining queue: {'done': 5, 'error': 1}
OK
$ echo $?
0

Read the third and fourth lines carefully: 0 done pages re-fetched, queue drained to 0 pending. The done column survived the kill; the 3 still-queued URLs were picked up by the fresh process and finished. At even one cent per request on a 21.6M-request crawl, that recovery path is the difference between $216K wasted on duplicates and $0. Why SQLite and not Redis? One file, no server, durable by default — Redis is faster but you’d need an AOF policy and careful fsync discipline to buy back the durability SQLite hands you for free.

URL deduplication at scale: set vs. Bloom filter

A Python set of URLs is fine up to a few million entries. For 100M URLs (avg ~90 chars), the raw strings alone are ~15.5 GB before set overhead. A Bloom filter at 1% false-positive rate fits ~120 MB with k=7 hashes — 130x smaller. Trade-off: ~1% of good URLs get wrongly flagged as seen, so use it as a front-line filter in front of a smaller exact set, never as the sole authority.

What’s still missing (honest gaps)

  • DNS caching: urllib resolves DNS per request; a serious crawler caches it — a 500-host crawl burns thousands of lookups/hour otherwise.
  • Redirect handling: redirects can land you on a different host than the one you rate-limited — re-check politeness after each hop.
  • Priority scoring: this frontier is FIFO per host; real crawls weight by page value — a heap-key change, not a rewrite.
  • Content dedup: URL dedup ≠ content dedup; one product page under 4 URL variants is 4 entries here.
  • Distribution: one process, one lock. Beyond one box, move the heap into Redis (sorted set on next_ok_time) — the two-level design survives unchanged.

Wrap-up

The frontier decides whether a scraping system survives contact with production: politeness keeps the lawyers away, persistence keeps crashes from costing money, dedup keeps the request budget from leaking. None of it is hard. Almost nobody ships it.

Building a production-grade frontier (or any data infrastructure that has to survive its operator) is exactly the kind of system we build for clients — custom versions with distributed scheduling, per-domain priority scoring, and monitoring included. If you want this without the build-it-yourself part, email [email protected].