Every guide about Playwright at scale talks about anti-bot detection. Almost none talk about what actually kills long-running browser automation on a $5 VPS: the fleet rotting. Contexts that never quite get closed. Browsers that survive their own crashes as half-orphaned process trees eating half a gigabyte. Tasks that hang forever because the sync API has no cancellation. This article builds the piece that production setups have and tutorials skip: a fleet supervisor — and every number in it comes from a live run on this exact class of box (4 GB RAM, 2 vCPU).

All code ran clean before publication (exit code 0, full sources at the end). Stack: Python 3.14, Playwright 1.63, Chromium 153 (headless shell), psutil. No API keys, nothing mocked.

First, measure the leak everyone ships

The classic long-running scraper pattern is: launch once, loop forever, new_context() per task. Here is that exact rig, run for 20 iterations against data-URL pages, with contexts intentionally never closed — because that is what happens on every machine where an early return or an exception path skips the cleanup:

import subprocess, psutil
from playwright.sync_api import sync_playwright

def chrome_rss_mb():                      # total RSS of every chromium process
    out = subprocess.run(["bash", "-c", "pgrep -f 'headless_shel[l]'"],
                         capture_output=True, text=True).stdout.split()
    tot = 0
    for pid in out:
        try:
            tot += psutil.Process(int(pid)).memory_info().rss
        except psutil.Error:
            pass
    return tot / 1048576

with sync_playwright() as p:
    b = p.chromium.launch()
    samples = []
    for i in range(20):
        ctx = b.new_context()
        pg = ctx.new_page()
        pg.goto("data:text/html,<html><body><p>leak-%d</p></body></html>" % i)
        _ = pg.content()                  # context intentionally NEVER closed
        samples.append(round(chrome_rss_mb(), 1))
    b.close()

Real output from this box (Playwright 1.63, Chromium 153):

leaked-context RSS series (MB): [377.8, 470.3, 562.6, 654.8, 747.1, 839.9,
932.4, 1025.0, 1117.8, 1210.3, 1303.0, 1395.7, 1488.3, 1581.0, 1673.6,
1766.4, 1859.2, 1951.9, 2044.4, 2137.2]
leak per context: ~93 MB

93 MB of RSS per task, forever. It is linear and it does not plateau. Two details matter for the business math:

  1. The driver’s own RSS stays flat — the leak is inside the chromium process tree, so a naive “monitor the Python process” alert never fires. You find out when the OOM killer starts shooting processes at random.
  2. On a 4 GB box running one 300 MB chromium, you cross into swap pressure after roughly 35 leaked contexts. At a modest 2,000 tasks/day with a 1% leak rate, you reboot or OOM-crash daily.

The fix for the leak itself is one line — ctx.close() in a finally block. In the matched control run, 40 tasks with proper closes drifted 2.3 MB total. But “remember to close” is not an architecture, and the other failure modes below don’t have a one-line fix. So build the supervisor.

The four failure modes nobody demos

F1 — Silent leak. Covered above: any exception path that skips ctx.close() compounds at 93 MB/task. You need enforcement, not convention.

F2 — Browser death mid-task. The chromium process can be OOM-killed by the kernel at any moment. The sync API surfaces this as TargetClosedError — but so do page timeouts on a crashed tab, so matching exception types is how you accidentally rebuild healthy browsers and leave dead ones alone.

F3 — Driver death mid-task. Playwright runs a Node.js driver per client. If the driver dies (it’s a 100+ MB Node process on a memory-starved box, prime OOM-killer bait), the Python side can hang indefinitely on a call that will never return — there is no timeout because nothing raised one.

F4 — Stuck tasks. A page that never fires load, an infinite wait_for_selector, a bad sleep in a task: the sync API has no cancellation mechanism. A task that hangs holds its slot forever.

There is also a maintenance failure mode people discover the hard way: when chromium is killed, it is re-parented to a subreaper — not pid 1. We verified this live: after killing the playwright driver with SIGKILL, the orphaned headless_shell tree’s parent was a long-lived python3 daemon that had been running on this box for weeks. Half the zombie-reaper scripts on GitHub check ppid == 1 and would never fire here. The reaper in our supervisor therefore matches by ownership (which pids belong to a live worker’s tree) rather than by parentage, and kills anything headless_shell-shaped that nobody claims.

A second process-tree gotcha bit us during development, and it is worth your first debugging hour: identifying your own chromium root by “any new headless_shell pid after launch” is wrong, because chromium’s children also match. Grab a child and your ownership set misses the real root — the reaper then kills your own live browser, and the fleet enters a crash-recovery loop it never exits. Hence the parent-not-new rule in the next section.

The design: five rules, each one a measured response

The supervisor (browserfleet.py, full source at the end) enforces:

  • R1 — Context-per-task, closed in finally. The 93 MB leak rate is why this is code, not style advice.
  • R2 — Browser recycling. ctx.close() recovered 96+ MB/task in our control run but is not perfect: the browser is recycled after max_tasks_per_browser (default 200), max_browser_age_s (900), or an RSS ceiling breach. Recycling is cheap — a fresh launch measured 0.2 s on this box.
  • R3 — Recovery by liveness probing, never by exception type. After any task exception, the worker asks: is the driver pid alive? Is the chromium root pid alive? Only if one of them is dead does it rebuild. Healthy browser + exception = a task bug; report it and move on.
  • R4 — Deadline kills with slot respawn. Every task gets a deadline. The supervisor kills an over-deadline worker’s driver tree — which unblocks a stuck playwright call — and abandons the thread. For a task stuck in pure Python, no signal can free it, so the unit of recovery is the slot: the thread is abandoned, the task is reported failed, and a fresh worker takes over the queue.
  • R5 — A reaper that doesn’t trust ppid. Every 10 s it kills headless_shell trees that belong to no registered worker. Orphans are found by ownership, not parentage — which is the only approach that works when a subreaper adopts your corpses.

Two implementation details that each cost a real debugging session:

Finding your own chromium root needs the parent rule. As described above, the root is the new pid whose parent is not itself new (its parent is the playwright driver). Launches are also serialized behind a lock, and the reaper stands down while a launch is mid-flight — both guards exist because two workers launching at once turned the pid diff into a lottery:

before = set(_all_chrome_pids())
self.pw = sync_playwright().start()
self.browser = self.pw.chromium.launch(args=list(self.cfg.launch_args))
new_pids = set(_all_chrome_pids()) - before
self.browser_pid = next(
    (p for p in new_pids
     if (pp := _safe_ppid(p)) is not None and pp not in new_pids),
    None)
self.driver_pid = _safe_ppid(self.browser_pid)

The sync API refuses threads with an asyncio loop. If your supervisor harness runs one (many do), sync_playwright().start() throws “It looks like you are using Playwright Sync API inside the asyncio loop.” Browsers must be started inside the worker threads, which are loop-free — not in the main thread’s fleet.start().

Proof: the four scenarios, live

Run as demo.py on the box; output below is verbatim from the run:

=== SCENARIO 1: crash recovery (chromium SIGKILLed mid-fleet) ===
s1: 10/10 tasks OK before the crash
s1: recovered in 1.1s, result=post-crash, crashes_recovered=1

=== SCENARIO 2: stuck task rescued by the deadline supervisor ===
s2: stuck task reported as DeadlineExceeded: task over 30s, slot rebuilt
    after 30.8s (deadline was 30s)
s2: slot works again: slot-rebuilt

=== SCENARIO 3: throughput on a 4GB-class box (2 slots) ===
s3: 40/40 tasks in 6.2s = 6.4 tasks/s, rss during load: 589MB total chromium

=== SCENARIO 4: leak containment (the whole point) ===
s4: metrics: {'tasks_done': 52, 'tasks_failed': 0, 'crashes_recovered': 2,
              'deadline_kills': 1, 'browser_recycles': 0,
              'zombies_killed': 0, 'uptime_s': 47.3}
s4: chromium processes left after shutdown: 0

Read that last line again: zero chromium processes after shutdown, including after an abandoned-thread deadline kill — the case where a stuck thread’s browser would otherwise leak forever. That took a specific mechanism: when a thread is abandoned, its tree root is recorded and the reaper (or shutdown, whichever fires first) kills the whole tree by ownership, because the dead thread can no longer be relied on to close anything.

The full integration suite (12 checks: happy path, task-failure non-recovery, external chromium kill, external driver kill, deadline rescue, post-kill slot health, RSS recycle with fresh pid, RSS bounding, clean shutdown) exits 0 in ~27 s on this box.

What this is worth in money

A supervised fleet and a naive loop differ in revenue-hours, not vibes:

  • A crashed slot rebuilds in ~1.1 s (measured). The naive alternative is a cron-polled process that a human notices. If your scraper bills per-records-scraped or your price-monitoring client pays for freshness, every crashed hour is unbilled.
  • No daily OOM reboot on 4 GB boxes. The $4.29/month Hetzner CX23 (2 vCPU / 4 GB — current pricing as of this writing) becomes a viable browser node instead of a crash loop; the alternative is sizing up to 8 GB or paying a managed browser service (Browserless’s entry paid tier is $25/month for 10 concurrent browsers, billed annually).
  • Abandoned-thread trees get reaped, so a bad deploy doesn’t leave five 300 MB chromium zombies pinned to a supervisor daemon overnight — the pattern that turns “my VPS is slow” into a 2 a.m. incident.

If you run browser automation for clients, this layer is also a product feature: “our scrapers self-heal” is a line you can defend in a demo, and the metrics dict (tasks_done, crashes_recovered, deadline_kills, zombies_killed) is a monitoring page that writes itself.

The layer also changes where it is cheap to run things. Once recycling is enforced, a fleet’s steady-state memory is a function of slots x baseline plus a small drift, not of how long the box has been up — which is what makes a $4.29/month 4 GB VPS a planning unit you can multiply. Need twelve scraped verticals? That is roughly twelve CX23 instances at ~€51/month total, or two boxes with denser slots and an RSS ceiling per worker — the config that decides it is three numbers in FleetConfig, and the same code runs unchanged on both shapes. This is also the quiet argument against jumping straight to a managed browser cloud: Browserless’s Prototyping tier runs $25/month for 10 concurrent browsers (billed annually, ~20k units), which is the right call when you need their proxy and captcha stack, and the wrong call when your problem is just that your own browsers keep dying. Fix the dying first — it’s ~380 lines — then pay for someone else’s browsers only when you actually need their network position.

Honest limitations

  • This is the sync API. If your stack is asyncio, the same rules apply but every mechanism needs the async variants (and the slot-respawn logic gets simpler, because asyncio.wait_for actually cancels).
  • The reaper’s 20 s grace period on fresh trees trades responsiveness for safety. A process can leak for up to ~30 s before the reaper catches it. Fine for maintenance; don’t use the reaper as your primary cleanup.
  • A deadline kill counts the task as failed. If a task legitimately needs 10 minutes, raise task_deadline_s for that fleet — the deadline is a liveness assumption, not a performance target.
  • Kill-based recovery loses the browser’s session state (cookies, storage). For login-walled scraping, pair this with a session-pool layer that rehydrates storage state per context — the browser is disposable, the session must not be.
  • Measured on Chromium 153 / Playwright 1.63 / Linux 6.8 on a Tencent Cloud 2 vCPU / 4 GB instance. The leak rate will differ across versions and page complexity (heavier pages leak more per context); the architecture won’t.
  • TargetClosedError handling here is defensive only — recovery decisions come from the pid probes, exactly because exception types proved unreliable during testing (a driver kill surfaced as a plain Exception from the transport, not a typed error, in one of our runs).

The code

browserfleet.py (~380 lines, stdlib + playwright + psutil) and the test suite (test_fleet.py, 12 assertions) are reproduced below in full. Tested: Python 3.14.7, Playwright 1.63.0, Chromium 153.0.8010.12, psutil 7.2.2, Linux 6.8.

# browserfleet.py — a fleet supervisor for long-running Playwright workers
# on small VPS boxes (stdlib + playwright + psutil only).
from __future__ import annotations

import queue
import threading
import time
from dataclasses import dataclass, field

import psutil
from playwright.sync_api import sync_playwright
from playwright._impl._errors import TargetClosedError  # not re-exported (1.63)

CHROME_PROC_RE = "ms-playwright"  # narrows to Playwright-managed chromium builds


def _all_chrome_pids() -> list[int]:
    out = []
    for proc in psutil.process_iter(["pid", "cmdline"]):
        try:
            cmd = " ".join(proc.info["cmdline"] or [])
            if CHROME_PROC_RE in cmd and "headless_shell" in cmd:
                out.append(proc.info["pid"])
        except (psutil.Error, TypeError):
            continue
    return out


def _rss_tree_mb(pid: int) -> float:
    """RSS of pid + all descendants, MB. 0.0 if pid is gone."""
    try:
        proc = psutil.Process(pid)
        total = proc.memory_info().rss
        kids = proc.children(recursive=True)
    except psutil.Error:
        return 0.0
    for child in kids:
        try:
            total += child.memory_info().rss
        except psutil.Error:
            pass
    return total / 1048576


def _safe_ppid(pid):
    if not pid:
        return None
    try:
        return psutil.Process(pid).ppid()
    except psutil.Error:
        return None


def _kill_tree(pid):
    """SIGKILL pid and descendants; ignore races."""
    if not pid:
        return
    try:
        proc = psutil.Process(pid)
        kids = proc.children(recursive=True)
    except psutil.Error:
        return
    for child in kids:
        try:
            child.kill()
        except psutil.Error:
            pass
    try:
        proc.kill()
    except psutil.Error:
        pass


@dataclass
class FleetConfig:
    slots: int = 2
    max_tasks_per_browser: int = 200   # R2
    max_browser_age_s: float = 900.0   # R2
    rss_ceiling_mb: float = 1400.0     # R2: per-worker chromium-tree ceiling
    task_retry_on_crash: int = 1       # R3
    task_deadline_s: float = 60.0      # R4
    launch_args: tuple = (
        "--disable-gpu",
        "--disable-dev-shm-usage",
        "--no-sandbox",
        "--disable-extensions",
        "--disable-background-networking",
        "--disable-component-update",
        "--mute-audio",
    )


@dataclass
class Metrics:
    tasks_done: int = 0
    tasks_failed: int = 0
    crashes_recovered: int = 0
    deadline_kills: int = 0
    browser_recycles: int = 0
    zombies_killed: int = 0
    started_at: float = field(default_factory=time.time)

    def summary(self) -> dict:
        return {
            "tasks_done": self.tasks_done,
            "tasks_failed": self.tasks_failed,
            "crashes_recovered": self.crashes_recovered,
            "deadline_kills": self.deadline_kills,
            "browser_recycles": self.browser_recycles,
            "zombies_killed": self.zombies_killed,
            "uptime_s": round(time.time() - self.started_at, 1),
        }


class _Worker:
    def __init__(self, name, cfg: FleetConfig, metrics: Metrics, log,
                 launch_lock: threading.Lock):
        self.name, self.cfg, self.metrics, self.log = name, cfg, metrics, log
        self.launch_lock = launch_lock
        self.pw = None
        self.browser = None
        self.browser_pid = None
        self.driver_pid = None
        self.orphan_tree_root = None  # set when a thread is abandoned alive
        self.tasks_on_browser = 0
        self.browser_started = 0.0
        self.rss_flag = False
        self.deadline = None  # set while a task runs
        self.abandoned = False      # supervisor gave up on this thread
        self.current_payload = None

    # ---- lifecycle ---------------------------------------------------------
    def start_browser(self):
        self.launch_lock.acquire()
        try:
            before = set(_all_chrome_pids())
            self.pw = sync_playwright().start()
            self.browser = self.pw.chromium.launch(
                args=list(self.cfg.launch_args))
            new_pids = set(_all_chrome_pids()) - before
            # R6: the root is the new pid whose parent is not itself new
            self.browser_pid = next(
                (p for p in new_pids
                 if (pp := _safe_ppid(p)) is not None and pp not in new_pids),
                None)
            self.driver_pid = _safe_ppid(self.browser_pid)
            self.tasks_on_browser = 0
            self.browser_started = time.time()
            self.rss_flag = False
            self.log(f"[{self.name}] browser up chrome={self.browser_pid} "
                     f"driver={self.driver_pid}")
        finally:
            self.launch_lock.release()

    def stop_browser(self, graceful=True):
        if graceful and self.browser is not None:
            timer = threading.Timer(5.0, _kill_tree, args=(self.driver_pid,))
            timer.daemon = True
            timer.start()
            try:
                self.browser.close()
            except Exception:
                pass
            timer.cancel()
        try:
            if self.pw is not None:
                self.pw.stop()
        except Exception:
            pass
        # hard fallback: nothing of this worker may survive
        _kill_tree(self.driver_pid)
        _kill_tree(self.browser_pid)
        self.pw = self.browser = self.browser_pid = self.driver_pid = None
        self.deadline = None
        self.current_payload = None

    def liveness(self) -> str:
        """R3: probe instead of guessing from exception types."""
        driver_ok = bool(self.driver_pid and psutil.pid_exists(self.driver_pid))
        chrome_ok = bool(self.browser_pid and psutil.pid_exists(self.browser_pid))
        if driver_ok and chrome_ok:
            return "ok"
        return "chrome-dead" if driver_ok else "driver-dead"

    def needs_recycle(self) -> bool:
        if self.rss_flag:
            return True
        if self.tasks_on_browser >= self.cfg.max_tasks_per_browser:
            return True
        return time.time() - self.browser_started >= self.cfg.max_browser_age_s

    def recycle(self):
        reason = ("rss-ceiling" if self.rss_flag
                  else "task-count"
                  if self.tasks_on_browser >= self.cfg.max_tasks_per_browser
                  else "age")
        self.stop_browser()
        self.start_browser()
        self.metrics.browser_recycles += 1
        self.log(f"[{self.name}] recycled browser ({reason})")

    def abandon(self):
        """Supervisor-side: this worker's thread is stuck and its slot is
        being respawned. Record the tree root so the reaper can finish the
        job, and drop object references the dead thread still 'owns'."""
        self.orphan_tree_root = self.browser_pid
        self.pw = self.browser = self.browser_pid = self.driver_pid = None
        self.deadline = None
        self.current_payload = None
        self.abandoned = True

    def chrome_rss_mb(self) -> float:
        return _rss_tree_mb(self.browser_pid) if self.browser_pid else 0.0

    # ---- task execution ------------------------------------------------------
    def run_task(self, fn, payload):
        attempts = self.cfg.task_retry_on_crash + 1
        last_err = None
        for attempt in range(attempts):
            ctx = None
            try:
                if self.abandoned:
                    raise RuntimeError("worker abandoned by supervisor")
                self.current_payload = payload
                self.deadline = time.time() + self.cfg.task_deadline_s
                ctx = self.browser.new_context()
                result = fn(ctx, payload)
                self.metrics.tasks_done += 1
                return result
            except Exception as e:  # deliberate broad catch
                last_err = e
                if self.abandoned:
                    raise  # supervisor already reported this task
                state = self.liveness()
                if state == "ok":
                    # browser healthy -> task-level failure, no rebuild
                    self.metrics.tasks_failed += 1
                    self.log(f"[{self.name}] task failed: "
                             f"{type(e).__name__}: {str(e)[:120]}")
                    break
                # infrastructure died underneath us -> rebuild and retry
                self.metrics.crashes_recovered += 1
                self.log(f"[{self.name}] infra down ({state}), recovering "
                         f"(attempt {attempt + 1}/{attempts}): "
                         f"{type(e).__name__}")
                self.stop_browser(graceful=False)
                self.start_browser()
            finally:
                self.deadline = None
                if ctx is not None:
                    try:
                        ctx.close()
                    except Exception:
                        pass
                self.tasks_on_browser += 1
        raise last_err


class BrowserFleet:
    def __init__(self, cfg=None, log=print):
        self.cfg = cfg or FleetConfig()
        self.metrics = Metrics()
        self.log = log
        self.task_q = queue.Queue()
        self.result_q = queue.Queue()
        self.workers = []
        self._stop = threading.Event()
        self._threads = []
        self._launch_lock = threading.Lock()
        self._pending = 0                 # submitted-but-unreported tasks
        self._pending_lock = threading.Lock()

    # ---- public API ----------------------------------------------------------
    def submit(self, fn, payload):
        with self._pending_lock:
            self._pending += 1
        self.task_q.put((fn, payload))

    def _emit(self, payload, res, err):
        self.result_q.put((payload, res, err))
        with self._pending_lock:
            self._pending -= 1

    def drain(self):
        """Yield (payload, result, error) as tasks complete; returns once
        every submitted task has been reported. Accounting is fleet-level
        (_pending): an abandoned thread never calls task_done, so the
        queue's own counter would hang the drain forever."""
        while True:
            try:
                yield self.result_q.get(timeout=0.25)
            except queue.Empty:
                pass
            if self._pending <= 0 and self.result_q.empty():
                return

    def start(self):
        threading.Thread(target=self._supervisor_loop, daemon=True).start()
        threading.Thread(target=self._reaper_loop, daemon=True).start()
        for i in range(self.cfg.slots):
            self._spawn_worker(f"w{i}")

    def shutdown(self):
        self._stop.set()
        for w in self.workers:
            w.abandoned = True   # unblock stale sleeps via supervisor stop
        for t in self._threads:
            t.join(timeout=5)
        for w in self.workers:
            w.stop_browser()
        # abandoned-alive threads may leave a chromium tree behind; the
        # reaper is stopped now, so purge those roots directly
        for w in self.workers:
            if w.orphan_tree_root:
                _kill_tree(w.orphan_tree_root)
        self.log(f"[fleet] shutdown metrics={self.metrics.summary()}")

    # ---- internals -------------------------------------------------------------
    def _spawn_worker(self, name):
        w = _Worker(name, self.cfg, self.metrics, self.log, self._launch_lock)
        self.workers.append(w)
        # the browser starts inside the worker thread: the sync API refuses
        # to attach to any thread already running an asyncio loop
        t = threading.Thread(target=self._worker_loop, args=(w,), daemon=True)
        t.start()
        self._threads.append(t)

    def _worker_loop(self, w):
        w.start_browser()
        while not self._stop.is_set():
            try:
                fn, payload = self.task_q.get(timeout=0.25)
            except queue.Empty:
                # idle: honour a pending recycle flag immediately
                if w.needs_recycle() and self.task_q.empty():
                    w.recycle()
                continue
            try:
                if w.needs_recycle():
                    w.recycle()
                res = w.run_task(fn, payload)
                self._emit(payload, res, None)
            except Exception as e:
                if not w.abandoned:
                    self._emit(payload, None, f"{type(e).__name__}: {e}")
            finally:
                try:
                    self.task_q.task_done()  # may double-fire; harmless
                except ValueError:
                    pass

    def _supervisor_loop(self):
        """RSS ceiling flagging + task deadline enforcement (R2/R4).

        The sync Playwright API has no cancellation. A task stuck in a
        playwright call is freed by killing the driver tree; a task stuck
        in pure Python never frees. Either way the thread may be a loss,
        so the unit of recovery is the SLOT: report the task as failed,
        abandon the thread, and spawn a replacement worker."""
        while not self._stop.is_set():
            for w in list(self.workers):
                if (w.deadline is not None and time.time() > w.deadline
                        and w.driver_pid
                        and psutil.pid_exists(w.driver_pid)):
                    self.metrics.deadline_kills += 1
                    self.log(f"[supervisor] {w.name} task over "
                             f"{self.cfg.task_deadline_s:.0f}s -> killing "
                             f"driver tree, abandoning thread, respawning slot")
                    _kill_tree(w.driver_pid)
                    w.abandon()
                    self._emit(
                        w.current_payload, None,
                        f"DeadlineExceeded: task over "
                        f"{self.cfg.task_deadline_s:.0f}s, slot rebuilt")
                    self.workers.remove(w)
                    self._spawn_worker(f"{w.name}+")
                    continue
                if w.browser_pid and not w.rss_flag:
                    rss = w.chrome_rss_mb()
                    if rss > self.cfg.rss_ceiling_mb:
                        w.rss_flag = True
                        self.log(f"[supervisor] {w.name} rss={rss:.0f}MB > "
                                 f"{self.cfg.rss_ceiling_mb:.0f}MB -> recycle "
                                 f"after current task")
            time.sleep(0.5)

    def _reaper_loop(self):
        """R5/R6: kill headless_shell trees owned by nobody."""
        while not self._stop.is_set():
            if self._launch_lock.locked():
                time.sleep(2)   # never reap a mid-flight launch
                continue
            owned = set()
            orphan_roots = set()
            for w in self.workers:
                if w.orphan_tree_root:
                    orphan_roots.add(w.orphan_tree_root)
                if w.browser_pid:
                    owned.add(w.browser_pid)
                    try:
                        proc = psutil.Process(w.browser_pid)
                        owned.update(c.pid
                                     for c in proc.children(recursive=True))
                    except psutil.Error:
                        pass
            now = time.time()
            for proc in psutil.process_iter(
                    ["pid", "cmdline", "create_time"]):
                try:
                    cmd = " ".join(proc.info["cmdline"] or [])
                    if CHROME_PROC_RE not in cmd or "headless_shell" not in cmd:
                        continue
                    if now - proc.info["create_time"] < 20:
                        continue  # grace: fresh trees may still register
                    pid = proc.info["pid"]
                    if pid in owned:
                        continue
                    if pid in orphan_roots:
                        _kill_tree(pid)
                        self.metrics.zombies_killed += 1
                        self.log(f"[reaper] SIGKILL abandoned worker tree "
                                 f"pid={pid}")
                        continue
                    _kill_tree(pid)
                    self.metrics.zombies_killed += 1
                    self.log(f"[reaper] SIGKILL orphan chromium pid={pid}")
                except (psutil.Error, ProcessLookupError, TypeError):
                    continue
            time.sleep(10)

Usage — the whole point is that task code stays boring:

from browserfleet import BrowserFleet, FleetConfig

def render(ctx, payload):
    page = ctx.new_page()
    page.goto(payload["url"], timeout=30000)
    return {"title": page.title(), "len": len(page.content())}

fleet = BrowserFleet(FleetConfig(slots=2, task_deadline_s=60.0))
fleet.start()
for url in urls:
    fleet.submit(render, {"url": url})
for payload, result, err in fleet.drain():
    if err:
        print("failed:", payload, err)
    else:
        save(result)
fleet.shutdown()

Drop-in, one dependency you already have (psutil), and it turns “my scraper needs babysitting” into “my scraper files a report about itself.” That difference is the entire margin between a $500 automation gig and a support-ticket hobby.

Building something that has to survive contact with production — scraper fleets, AI pipelines, automation infra? That’s literally our day job. Email [email protected] for a custom version.