Most scrapers don’t fail because they get blocked. They fail because they retry their way into a ban. The developer adds exponential backoff with jitter — the AWS-approved pattern — and the scraper still ends up with a permanently 403’d identity two hours into the run. The backoff formula isn’t the problem. The problem is that backoff is a per-request reflex, while bans are a session-level judgment. Two different control loops, and most codebases only implement the first one.

This article builds the second loop: a client-side AIMD controller (additive increase, multiplicative decrease — the same feedback algorithm TCP has used since 1988) plus a circuit breaker, both driven entirely by server signals. No guessing a “safe rate.” The target teaches you its threshold. We’ll test it against a simulated rate-limiting edge that escalates from 429 to hard 403 exactly the way Cloudflare-style rules do, and show the numbers: naive pacing gets the identity banned in 200 requests, every seed; the controller takes zero hard blocks across the same runs.

All code ran clean before publication — full sources at the end, exit code 0, no external dependencies.

Why exponential backoff alone loses

Exponential backoff answers the question “this one request failed — when do I try it again?” It has no memory of the session. Three failure modes fall out of that:

1. Backoff treats 429 and 403 as the same event. They aren’t. A 429 Too Many Requests is protocol-level feedback — RFC 6585 defines it as “the user has sent too many requests in a given amount of time,” and the server MAY attach a Retry-After header telling you exactly how long to wait. It’s a speeding ticket. A 403 under anti-bot is different: per RFC 9110 the server “understands the request but refuses to fulfill it,” and in practice it means your identity — IP, TLS fingerprint, cookie session — is now suspect. That’s a criminal record. The right response to a ticket is slowing down; the right response to a record is realizing the session may be done, and not burning 50 more requests proving it.

2. Backoff recovers too fast. Standard backoff waits, retries, and on success resumes the old rate immediately. But rate-limit windows on real edges last 10 seconds to 10 minutes — Cloudflare’s own best-practice examples use rules like 4 failed login attempts per 1 minute and 10 per 10 minutes, counted on http.response.code in {401 403}. If your penalty decays in 2 seconds, you’re re-entering the window that just tripped.

3. Backoff retries on a budget that doesn’t exist. The AWS SDKs are the reference implementation here, and their internals are instructive: standard mode uses exponential backoff with jitter (50 ms base for transient errors, 1,000 ms for throttling, formula random(0,1) × min(20,000 ms, base × 2^retry)), plus a retry quota — a 500-token bucket where each retry burns 5 tokens (throttling) or 14 (transient), successes refund 1, and once the bucket drains, the SDK stops retrying and fails fast. The quota starts materially draining when more than roughly 22% of requests are failing transiently, or 32% for throttling errors. AWS engineered a session-level brake on top of per-request backoff — because they measured that backoff alone retry-storms during incidents. Scrapers face strictly worse conditions than AWS APIs, and most of them have no equivalent brake at all.

And the environment is hostile at a level that makes this non-optional. The 2026 Thales/Imperva Bad Bot Report found bots now account for the majority of global web traffic, and edge providers respond by scoring every session — Cloudflare assigns each request a bot score from 1 to 99, where 1–29 is treated as automated. Your pacing behavior is one of the few signals you fully control.

The design: two loops and a fuse

The controller has three components:

  1. Adaptive pacing (AIMD). One variable: delay, the seconds to wait between requests. After N consecutive clean responses, multiply the delay by ~0.9 (additive increase in log space — careful probing upward). On a 429 without Retry-After, multiply by 3. On a 403, multiply by 4. On a timeout or 5xx, multiply by 1.3 (could be you, could be them — mild). This is multiplicative decrease, and it’s deliberately asymmetric: increase slowly, collapse fast. AIMD is the proven scheme precisely because of this shape — it’s what converges in TCP, and the classical result (Chiu & Jain, 1989; standardized in RFC 5681’s lineage) is that additive-increase/multiplicative-decrease flows converge to stable sharing, while MIMD and AIAD variants don’t.

  2. Server-authored waits. When a 429 arrives with Retry-After, believe it — set the delay to that value (capped at 15 minutes so a hostile Retry-After: 86400 can’t freeze your job). RFC 9110 defines the field as exactly this contract: a server-indicated number of seconds (or HTTP-date) to wait.

  3. Circuit breaker. Keep a ring buffer of the last 30 signals. If more than 35% of them are 429s or 403s, the session is hot — stop all traffic for a cooldown (120 s, doubling per repeated trip, capped at an hour). This is the AWS retry-quota idea generalized: a session-level budget that fails fast instead of grinding.

The signal classification is where the engineering judgment lives:

class Signal(Enum):
    OK = "ok"
    RATE_LIMITED = "rate_limited"   # 429, or Retry-After on 503
    BLOCKED = "blocked"             # 403 / 406 / challenge page
    ERROR = "error"                 # network timeout, 5xx

def to_signal(status, headers=None):
    headers = headers or {}
    ra = headers.get("Retry-After")
    retry_after = None
    if ra:
        try:
            retry_after = float(ra)
        except ValueError:
            retry_after = None  # HTTP-date form: fall back to factors

    if status == 429:
        return (Signal.RATE_LIMITED, retry_after)
    if status in (403, 406):
        return (Signal.BLOCKED, None)
    if status in (401, 429, 503) and retry_after:
        return (Signal.RATE_LIMITED, retry_after)
    if 500 <= status < 600:
        return (Signal.ERROR, None)
    return (Signal.OK, None)

And the core of the state machine — every mutation of delay happens here, nowhere else:

def report(self, signal: Signal, retry_after: Optional[float] = None) -> None:
    self.sent += 1
    self._signals.append(signal)
    if len(self._signals) > self.cfg.breaker_window:
        self._signals.pop(0)

    if signal is Signal.OK:
        self.oks += 1
        self._ok_streak += 1
        # Additive increase: after N clean responses, shave the delay a bit.
        if self._ok_streak >= self.cfg.increase_successes:
            self._ok_streak = 0
            self.delay = max(self.cfg.delay_floor, self.delay * self.cfg.increase_step)
        return

    self._ok_streak = 0
    if signal is Signal.RATE_LIMITED:
        self.rate_limited += 1
        if retry_after:
            # Server told us exactly how long. Believe it (capped).
            self.delay = min(max(self.delay, retry_after), self.cfg.retry_after_cap)
        else:
            self.delay = min(self.cfg.delay_ceiling, self.delay * self.cfg.rate_limited_factor)
    elif signal is Signal.BLOCKED:
        self.blocked += 1
        self.delay = min(self.cfg.delay_ceiling, self.delay * self.cfg.blocked_factor)
    else:
        self.errors += 1
        self.delay = min(self.cfg.delay_ceiling, self.delay * self.cfg.error_factor)

    self._maybe_trip()

Integration is three lines around any fetch function: ctrl.wait() → request → ctrl.report(*to_signal(resp.status_code, dict(resp.headers))).

The adversary: a target that escalates like a real edge

Testing against your own code’s happy path proves nothing. The mock target enforces a rate limit of 60 requests/minute per identity, issues 429 + Retry-After on violations, counts strikes with decay (a clean request forgives one strike — the way scoring systems reward sustained good behavior without forgetting bursts), and after the strike limit silently switches to hard 403s. That last behavior isn’t invented: Cloudflare’s rate limiting rules are configured by default to return 429 when a rule trips, but site owners set the block response code anywhere in 400–499, and ban-style rules counting http.response.code eq 403 from the origin are a documented first-class pattern. Translation for scraper operators: the 429 is the warning shot; the 403 is the sentence.

class MockTarget:
    def request(self, t: float) -> tuple[int, dict]:
        if self.hard_ban:
            self.log.append((t, 403))
            return 403, {}

        if self._last is not None:
            gap = t - self._last
            if gap < self.min_interval:
                self.ignored_429s += 1
                if self.ignored_429s >= self.block_after_ignored:
                    self.hard_ban = True   # escalation: ignore us, get banned
                    self.log.append((t, 403))
                    return 403, {}
                self.log.append((t, 429))
                return 429, {"Retry-After": str(self.retry_after)}  # no credit decay

        if self.rng.random() < 0.03:
            self.log.append((t, 503))
            return 503, {}

        self._last = t
        # strike DECAY, not reset: one clean request forgives one strike.
        self.ignored_429s = max(0, self.ignored_429s - 1)
        self.log.append((t, 200))
        return 200, {}

One design note worth its weight: the strike counter uses decay, not reset. Our first version reset strikes on every clean request — and the naive baseline then survived forever, because a scraper at 3× the limit interleaves just enough successes to keep its record clean. That was a broken adversary, and it would have produced a beautifully green test and a dead real-world scraper. Fixing the adversary before praising the controller is the difference between testing and theater.

The numbers

We replay the request loop in virtual time (deterministic, 400 requests per run, 3% background 503s), comparing three strategies. The naive baseline runs a constant 0.333 s delay — 3× the target’s limit, with the circuit breaker disabled — which is exactly what “just add retries” code does in production when someone eyeballed a delay.

SeedNaive 3× overspeedAIMD from same hot startAIMD from conservative start (2 s)
1banned, 178×4030×403, 362/400 OK—
7banned at req #66, 135×4030×403, 260/400 OK0×403, 376/400 OK
42banned, 194×4030×403, 253/400 OK—
99banned, 140×4030×403, 248/400 OK—
2026banned, 135×4030×403, 246/400 OK—

The hot-start AIMD runs still eat 120–150 early 429s — that’s the honest cost of starting 3× over the limit and discovering the ceiling by touching it. The circuit breaker fires (1–9 trips) and saves the identity, but throughput suffers while the delay ratchets down. Start conservatively instead — 2 s base delay against an unknown target — and the picture changes completely: 94% OK rate, 9 total 429s, the first violation only at request #71, zero breaker trips, while the controller still converges upward to find the real ceiling. The lesson is boring and important: AIMD protects your identity, but every 429 you cause is throughput you paid for. Probing is for finding ceilings, not for starting against them.

The full test suite (test_controller.py) also verifies: signal mapping per the RFCs (429+Retry-After: 7 → delay ≥ 8 s honored; 403 → BLOCKED; 5xx → ERROR), breaker trips on a full bad window with escalating cooldowns, live real-time pacing of 12 requests at the commanded rate, and config validation rejecting nonsense like a zero base delay. Final line: ALL TESTS PASSED, exit 0.

What this buys you in money terms

Run the arithmetic against real proxy costs. Entry-level rotating residential access is cheap to start and brutally per-GB at scale — Bright Data’s live pricing page currently lists residential proxies from $2.5/GB (down from $5 with their current 50% discount), datacenter from $0.90/IP, ISP from $1.30/IP; Webshare’s storefront lists rotating residential plans from $3.50/month across “80 million+” IPs. If a ban costs you a residential identity mid-job — plus the re-warmup time, plus the retry traffic itself which you pay for either way — then every 403 storm has a direct line item. A controller that converts 135 hard blocks per 200 requests into zero per 400 doesn’t just “improve reliability”; it changes the effective per-row cost of the entire operation, which is the only scrape metric that shows up in accounting.

There’s a second, less obvious payoff. Sustained low-violation pacing reads as credible traffic to scoring systems. A session that has sent 10,000 requests with a 99.7% success rate is a very different statistical object from one that tripped 429 rules six times in its first hour — even if both end the job with the same data. Edge models score histories, not just requests.

Where this breaks (read before deploying)

Honesty about failure modes is the difference between an engineer and a content farm, so:

  • Distributed targets need shared state. The controller above is per-process. If you run 20 workers behind 20 residential IPs, each worker converges independently — which is fine — but if they share one IP, their combined rate is what the target sees, and you need coordination (Redis-backed shared delay, or one dispatcher). The failure is subtle: each worker sees a healthy 80% OK rate while the identity as a whole is melting down.
  • Silent soft-blocks bypass the classifier. Some targets never send 403; they serve a degraded page (empty results, a login wall, a “verify you’re human” interstitial) with status 200. No signal, no adaptation. This is why serious pipelines pair the controller with content-level tripwires — seed fields you expect on every page, trap paths that only a parser following bad links would hit — which we’ve covered separately. Response body validation is a signal source just like status codes.
  • Retry-After can lie. Malicious or misconfigured servers can send huge values to freeze you, or absurdly small ones to bait retries. Hence the cap (15 min) and the breaker as an independent safety net.
  • TLS/IP-level bans don’t care about your pacing. If the edge flags your JA3 fingerprint or the IP subnet itself, slowing down doesn’t help — the fix is identity rotation, which is a different subsystem. The controller’s breaker at least stops you from burning requests into a wall.
  • This is a reliability tool, not a permission slip. Respect robots.txt, terms of service, and applicable law for your targets. The educational point of this piece is engineering, not evasion — the same state machine keeps legitimate integrations (partner APIs, monitoring) alive when the other side throttles.

Takeaways

  1. Backoff is per-request memory; bans are session-level judgment. You need both loops.
  2. Classify before reacting: 429 is a ticket, 403 is a record, 5xx is noise. Different multipliers: ×3, ×4, ×1.3.
  3. Retry-After is the server handing you the answer. Take it (with a cap).
  4. Circuit-break at the session level — the AWS retry quota (500 tokens, fails fast past ~22–32% sustained failure) exists because even AWS measured that backoff alone isn’t enough.
  5. Start below the suspected limit. Probing upward costs 429s; starting hot costs identities.
  6. Test against an adversary that escalates — and make sure the adversary is actually adversarial before you celebrate green output.

The full controller (~250 lines, stdlib only) and test harness are Python 3.10+ clean, deterministic, and dependency-free — drop controller.py into any scraper and wrap your fetch call.

Building a scraping or automation pipeline that has to survive real anti-bot infrastructure — rate limits, fingerprinting, session scoring? We build these systems for clients every week, tuned to your targets and budgets. Need a custom version of this controller wired into your stack? Email [email protected].