Every guide about Playwright at scale talks about anti-bot detection. Almost none talk about what actually kills long-running browser automation on a $5 VPS: the fleet rotting. Contexts that never quite get closed. Browsers that survive their own crashes as half-orphaned process trees eating half a gigabyte. Tasks that hang forever because the sync API has no cancellation. This article builds the piece that production setups have and tutorials skip: a fleet supervisor — and every number in it comes from a live run on this exact class of box (4 GB RAM, 2 vCPU).
All code ran clean before publication (exit code 0, full sources at the end). Stack: Python 3.14, Playwright 1.63, Chromium 153 (headless shell), psutil. No API keys, nothing mocked.
First, measure the leak everyone ships
The classic long-running scraper pattern is: launch once, loop forever, new_context() per task. Here is that exact rig, run for 20 iterations against data-URL pages, with contexts intentionally never closed — because that is what happens on every machine where an early return or an exception path skips the cleanup:
import subprocess, psutil
from playwright.sync_api import sync_playwright
def chrome_rss_mb(): # total RSS of every chromium process
out = subprocess.run(["bash", "-c", "pgrep -f 'headless_shel[l]'"],
capture_output=True, text=True).stdout.split()
tot = 0
for pid in out:
try:
tot += psutil.Process(int(pid)).memory_info().rss
except psutil.Error:
pass
return tot / 1048576
with sync_playwright() as p:
b = p.chromium.launch()
samples = []
for i in range(20):
ctx = b.new_context()
pg = ctx.new_page()
pg.goto("data:text/html,<html><body><p>leak-%d</p></body></html>" % i)
_ = pg.content() # context intentionally NEVER closed
samples.append(round(chrome_rss_mb(), 1))
b.close()
Real output from this box (Playwright 1.63, Chromium 153):
leaked-context RSS series (MB): [377.8, 470.3, 562.6, 654.8, 747.1, 839.9,
932.4, 1025.0, 1117.8, 1210.3, 1303.0, 1395.7, 1488.3, 1581.0, 1673.6,
1766.4, 1859.2, 1951.9, 2044.4, 2137.2]
leak per context: ~93 MB
93 MB of RSS per task, forever. It is linear and it does not plateau. Two details matter for the business math:
- The driver’s own RSS stays flat — the leak is inside the chromium process tree, so a naive “monitor the Python process” alert never fires. You find out when the OOM killer starts shooting processes at random.
- On a 4 GB box running one 300 MB chromium, you cross into swap pressure after roughly 35 leaked contexts. At a modest 2,000 tasks/day with a 1% leak rate, you reboot or OOM-crash daily.
The fix for the leak itself is one line — ctx.close() in a finally block. In the matched control run, 40 tasks with proper closes drifted 2.3 MB total. But “remember to close” is not an architecture, and the other failure modes below don’t have a one-line fix. So build the supervisor.
The four failure modes nobody demos
F1 — Silent leak. Covered above: any exception path that skips ctx.close() compounds at 93 MB/task. You need enforcement, not convention.
F2 — Browser death mid-task. The chromium process can be OOM-killed by the kernel at any moment. The sync API surfaces this as TargetClosedError — but so do page timeouts on a crashed tab, so matching exception types is how you accidentally rebuild healthy browsers and leave dead ones alone.
F3 — Driver death mid-task. Playwright runs a Node.js driver per client. If the driver dies (it’s a 100+ MB Node process on a memory-starved box, prime OOM-killer bait), the Python side can hang indefinitely on a call that will never return — there is no timeout because nothing raised one.
F4 — Stuck tasks. A page that never fires load, an infinite wait_for_selector, a bad sleep in a task: the sync API has no cancellation mechanism. A task that hangs holds its slot forever.
There is also a maintenance failure mode people discover the hard way: when chromium is killed, it is re-parented to a subreaper — not pid 1. We verified this live: after killing the playwright driver with SIGKILL, the orphaned headless_shell tree’s parent was a long-lived python3 daemon that had been running on this box for weeks. Half the zombie-reaper scripts on GitHub check ppid == 1 and would never fire here. The reaper in our supervisor therefore matches by ownership (which pids belong to a live worker’s tree) rather than by parentage, and kills anything headless_shell-shaped that nobody claims.
A second process-tree gotcha bit us during development, and it is worth your first debugging hour: identifying your own chromium root by “any new headless_shell pid after launch” is wrong, because chromium’s children also match. Grab a child and your ownership set misses the real root — the reaper then kills your own live browser, and the fleet enters a crash-recovery loop it never exits. Hence the parent-not-new rule in the next section.
The design: five rules, each one a measured response
The supervisor (browserfleet.py, full source at the end) enforces:
- R1 — Context-per-task, closed in
finally. The 93 MB leak rate is why this is code, not style advice. - R2 — Browser recycling.
ctx.close()recovered 96+ MB/task in our control run but is not perfect: the browser is recycled aftermax_tasks_per_browser(default 200),max_browser_age_s(900), or an RSS ceiling breach. Recycling is cheap — a fresh launch measured 0.2 s on this box. - R3 — Recovery by liveness probing, never by exception type. After any task exception, the worker asks: is the driver pid alive? Is the chromium root pid alive? Only if one of them is dead does it rebuild. Healthy browser + exception = a task bug; report it and move on.
- R4 — Deadline kills with slot respawn. Every task gets a deadline. The supervisor kills an over-deadline worker’s driver tree — which unblocks a stuck playwright call — and abandons the thread. For a task stuck in pure Python, no signal can free it, so the unit of recovery is the slot: the thread is abandoned, the task is reported failed, and a fresh worker takes over the queue.
- R5 — A reaper that doesn’t trust ppid. Every 10 s it kills headless_shell trees that belong to no registered worker. Orphans are found by ownership, not parentage — which is the only approach that works when a subreaper adopts your corpses.
Two implementation details that each cost a real debugging session:
Finding your own chromium root needs the parent rule. As described above, the root is the new pid whose parent is not itself new (its parent is the playwright driver). Launches are also serialized behind a lock, and the reaper stands down while a launch is mid-flight — both guards exist because two workers launching at once turned the pid diff into a lottery:
before = set(_all_chrome_pids())
self.pw = sync_playwright().start()
self.browser = self.pw.chromium.launch(args=list(self.cfg.launch_args))
new_pids = set(_all_chrome_pids()) - before
self.browser_pid = next(
(p for p in new_pids
if (pp := _safe_ppid(p)) is not None and pp not in new_pids),
None)
self.driver_pid = _safe_ppid(self.browser_pid)
The sync API refuses threads with an asyncio loop. If your supervisor harness runs one (many do), sync_playwright().start() throws “It looks like you are using Playwright Sync API inside the asyncio loop.” Browsers must be started inside the worker threads, which are loop-free — not in the main thread’s fleet.start().
Proof: the four scenarios, live
Run as demo.py on the box; output below is verbatim from the run:
=== SCENARIO 1: crash recovery (chromium SIGKILLed mid-fleet) ===
s1: 10/10 tasks OK before the crash
s1: recovered in 1.1s, result=post-crash, crashes_recovered=1
=== SCENARIO 2: stuck task rescued by the deadline supervisor ===
s2: stuck task reported as DeadlineExceeded: task over 30s, slot rebuilt
after 30.8s (deadline was 30s)
s2: slot works again: slot-rebuilt
=== SCENARIO 3: throughput on a 4GB-class box (2 slots) ===
s3: 40/40 tasks in 6.2s = 6.4 tasks/s, rss during load: 589MB total chromium
=== SCENARIO 4: leak containment (the whole point) ===
s4: metrics: {'tasks_done': 52, 'tasks_failed': 0, 'crashes_recovered': 2,
'deadline_kills': 1, 'browser_recycles': 0,
'zombies_killed': 0, 'uptime_s': 47.3}
s4: chromium processes left after shutdown: 0
Read that last line again: zero chromium processes after shutdown, including after an abandoned-thread deadline kill — the case where a stuck thread’s browser would otherwise leak forever. That took a specific mechanism: when a thread is abandoned, its tree root is recorded and the reaper (or shutdown, whichever fires first) kills the whole tree by ownership, because the dead thread can no longer be relied on to close anything.
The full integration suite (12 checks: happy path, task-failure non-recovery, external chromium kill, external driver kill, deadline rescue, post-kill slot health, RSS recycle with fresh pid, RSS bounding, clean shutdown) exits 0 in ~27 s on this box.
What this is worth in money
A supervised fleet and a naive loop differ in revenue-hours, not vibes:
- A crashed slot rebuilds in ~1.1 s (measured). The naive alternative is a cron-polled process that a human notices. If your scraper bills per-records-scraped or your price-monitoring client pays for freshness, every crashed hour is unbilled.
- No daily OOM reboot on 4 GB boxes. The $4.29/month Hetzner CX23 (2 vCPU / 4 GB — current pricing as of this writing) becomes a viable browser node instead of a crash loop; the alternative is sizing up to 8 GB or paying a managed browser service (Browserless’s entry paid tier is $25/month for 10 concurrent browsers, billed annually).
- Abandoned-thread trees get reaped, so a bad deploy doesn’t leave five 300 MB chromium zombies pinned to a supervisor daemon overnight — the pattern that turns “my VPS is slow” into a 2 a.m. incident.
If you run browser automation for clients, this layer is also a product feature: “our scrapers self-heal” is a line you can defend in a demo, and the metrics dict (tasks_done, crashes_recovered, deadline_kills, zombies_killed) is a monitoring page that writes itself.
The layer also changes where it is cheap to run things. Once recycling is enforced, a fleet’s steady-state memory is a function of slots x baseline plus a small drift, not of how long the box has been up — which is what makes a $4.29/month 4 GB VPS a planning unit you can multiply. Need twelve scraped verticals? That is roughly twelve CX23 instances at ~€51/month total, or two boxes with denser slots and an RSS ceiling per worker — the config that decides it is three numbers in FleetConfig, and the same code runs unchanged on both shapes. This is also the quiet argument against jumping straight to a managed browser cloud: Browserless’s Prototyping tier runs $25/month for 10 concurrent browsers (billed annually, ~20k units), which is the right call when you need their proxy and captcha stack, and the wrong call when your problem is just that your own browsers keep dying. Fix the dying first — it’s ~380 lines — then pay for someone else’s browsers only when you actually need their network position.
Honest limitations
- This is the sync API. If your stack is asyncio, the same rules apply but every mechanism needs the async variants (and the slot-respawn logic gets simpler, because
asyncio.wait_foractually cancels). - The reaper’s 20 s grace period on fresh trees trades responsiveness for safety. A process can leak for up to ~30 s before the reaper catches it. Fine for maintenance; don’t use the reaper as your primary cleanup.
- A deadline kill counts the task as failed. If a task legitimately needs 10 minutes, raise
task_deadline_sfor that fleet — the deadline is a liveness assumption, not a performance target. - Kill-based recovery loses the browser’s session state (cookies, storage). For login-walled scraping, pair this with a session-pool layer that rehydrates storage state per context — the browser is disposable, the session must not be.
- Measured on Chromium 153 / Playwright 1.63 / Linux 6.8 on a Tencent Cloud 2 vCPU / 4 GB instance. The leak rate will differ across versions and page complexity (heavier pages leak more per context); the architecture won’t.
TargetClosedErrorhandling here is defensive only — recovery decisions come from the pid probes, exactly because exception types proved unreliable during testing (a driver kill surfaced as a plainExceptionfrom the transport, not a typed error, in one of our runs).
The code
browserfleet.py (~380 lines, stdlib + playwright + psutil) and the test suite (test_fleet.py, 12 assertions) are reproduced below in full. Tested: Python 3.14.7, Playwright 1.63.0, Chromium 153.0.8010.12, psutil 7.2.2, Linux 6.8.
# browserfleet.py — a fleet supervisor for long-running Playwright workers
# on small VPS boxes (stdlib + playwright + psutil only).
from __future__ import annotations
import queue
import threading
import time
from dataclasses import dataclass, field
import psutil
from playwright.sync_api import sync_playwright
from playwright._impl._errors import TargetClosedError # not re-exported (1.63)
CHROME_PROC_RE = "ms-playwright" # narrows to Playwright-managed chromium builds
def _all_chrome_pids() -> list[int]:
out = []
for proc in psutil.process_iter(["pid", "cmdline"]):
try:
cmd = " ".join(proc.info["cmdline"] or [])
if CHROME_PROC_RE in cmd and "headless_shell" in cmd:
out.append(proc.info["pid"])
except (psutil.Error, TypeError):
continue
return out
def _rss_tree_mb(pid: int) -> float:
"""RSS of pid + all descendants, MB. 0.0 if pid is gone."""
try:
proc = psutil.Process(pid)
total = proc.memory_info().rss
kids = proc.children(recursive=True)
except psutil.Error:
return 0.0
for child in kids:
try:
total += child.memory_info().rss
except psutil.Error:
pass
return total / 1048576
def _safe_ppid(pid):
if not pid:
return None
try:
return psutil.Process(pid).ppid()
except psutil.Error:
return None
def _kill_tree(pid):
"""SIGKILL pid and descendants; ignore races."""
if not pid:
return
try:
proc = psutil.Process(pid)
kids = proc.children(recursive=True)
except psutil.Error:
return
for child in kids:
try:
child.kill()
except psutil.Error:
pass
try:
proc.kill()
except psutil.Error:
pass
@dataclass
class FleetConfig:
slots: int = 2
max_tasks_per_browser: int = 200 # R2
max_browser_age_s: float = 900.0 # R2
rss_ceiling_mb: float = 1400.0 # R2: per-worker chromium-tree ceiling
task_retry_on_crash: int = 1 # R3
task_deadline_s: float = 60.0 # R4
launch_args: tuple = (
"--disable-gpu",
"--disable-dev-shm-usage",
"--no-sandbox",
"--disable-extensions",
"--disable-background-networking",
"--disable-component-update",
"--mute-audio",
)
@dataclass
class Metrics:
tasks_done: int = 0
tasks_failed: int = 0
crashes_recovered: int = 0
deadline_kills: int = 0
browser_recycles: int = 0
zombies_killed: int = 0
started_at: float = field(default_factory=time.time)
def summary(self) -> dict:
return {
"tasks_done": self.tasks_done,
"tasks_failed": self.tasks_failed,
"crashes_recovered": self.crashes_recovered,
"deadline_kills": self.deadline_kills,
"browser_recycles": self.browser_recycles,
"zombies_killed": self.zombies_killed,
"uptime_s": round(time.time() - self.started_at, 1),
}
class _Worker:
def __init__(self, name, cfg: FleetConfig, metrics: Metrics, log,
launch_lock: threading.Lock):
self.name, self.cfg, self.metrics, self.log = name, cfg, metrics, log
self.launch_lock = launch_lock
self.pw = None
self.browser = None
self.browser_pid = None
self.driver_pid = None
self.orphan_tree_root = None # set when a thread is abandoned alive
self.tasks_on_browser = 0
self.browser_started = 0.0
self.rss_flag = False
self.deadline = None # set while a task runs
self.abandoned = False # supervisor gave up on this thread
self.current_payload = None
# ---- lifecycle ---------------------------------------------------------
def start_browser(self):
self.launch_lock.acquire()
try:
before = set(_all_chrome_pids())
self.pw = sync_playwright().start()
self.browser = self.pw.chromium.launch(
args=list(self.cfg.launch_args))
new_pids = set(_all_chrome_pids()) - before
# R6: the root is the new pid whose parent is not itself new
self.browser_pid = next(
(p for p in new_pids
if (pp := _safe_ppid(p)) is not None and pp not in new_pids),
None)
self.driver_pid = _safe_ppid(self.browser_pid)
self.tasks_on_browser = 0
self.browser_started = time.time()
self.rss_flag = False
self.log(f"[{self.name}] browser up chrome={self.browser_pid} "
f"driver={self.driver_pid}")
finally:
self.launch_lock.release()
def stop_browser(self, graceful=True):
if graceful and self.browser is not None:
timer = threading.Timer(5.0, _kill_tree, args=(self.driver_pid,))
timer.daemon = True
timer.start()
try:
self.browser.close()
except Exception:
pass
timer.cancel()
try:
if self.pw is not None:
self.pw.stop()
except Exception:
pass
# hard fallback: nothing of this worker may survive
_kill_tree(self.driver_pid)
_kill_tree(self.browser_pid)
self.pw = self.browser = self.browser_pid = self.driver_pid = None
self.deadline = None
self.current_payload = None
def liveness(self) -> str:
"""R3: probe instead of guessing from exception types."""
driver_ok = bool(self.driver_pid and psutil.pid_exists(self.driver_pid))
chrome_ok = bool(self.browser_pid and psutil.pid_exists(self.browser_pid))
if driver_ok and chrome_ok:
return "ok"
return "chrome-dead" if driver_ok else "driver-dead"
def needs_recycle(self) -> bool:
if self.rss_flag:
return True
if self.tasks_on_browser >= self.cfg.max_tasks_per_browser:
return True
return time.time() - self.browser_started >= self.cfg.max_browser_age_s
def recycle(self):
reason = ("rss-ceiling" if self.rss_flag
else "task-count"
if self.tasks_on_browser >= self.cfg.max_tasks_per_browser
else "age")
self.stop_browser()
self.start_browser()
self.metrics.browser_recycles += 1
self.log(f"[{self.name}] recycled browser ({reason})")
def abandon(self):
"""Supervisor-side: this worker's thread is stuck and its slot is
being respawned. Record the tree root so the reaper can finish the
job, and drop object references the dead thread still 'owns'."""
self.orphan_tree_root = self.browser_pid
self.pw = self.browser = self.browser_pid = self.driver_pid = None
self.deadline = None
self.current_payload = None
self.abandoned = True
def chrome_rss_mb(self) -> float:
return _rss_tree_mb(self.browser_pid) if self.browser_pid else 0.0
# ---- task execution ------------------------------------------------------
def run_task(self, fn, payload):
attempts = self.cfg.task_retry_on_crash + 1
last_err = None
for attempt in range(attempts):
ctx = None
try:
if self.abandoned:
raise RuntimeError("worker abandoned by supervisor")
self.current_payload = payload
self.deadline = time.time() + self.cfg.task_deadline_s
ctx = self.browser.new_context()
result = fn(ctx, payload)
self.metrics.tasks_done += 1
return result
except Exception as e: # deliberate broad catch
last_err = e
if self.abandoned:
raise # supervisor already reported this task
state = self.liveness()
if state == "ok":
# browser healthy -> task-level failure, no rebuild
self.metrics.tasks_failed += 1
self.log(f"[{self.name}] task failed: "
f"{type(e).__name__}: {str(e)[:120]}")
break
# infrastructure died underneath us -> rebuild and retry
self.metrics.crashes_recovered += 1
self.log(f"[{self.name}] infra down ({state}), recovering "
f"(attempt {attempt + 1}/{attempts}): "
f"{type(e).__name__}")
self.stop_browser(graceful=False)
self.start_browser()
finally:
self.deadline = None
if ctx is not None:
try:
ctx.close()
except Exception:
pass
self.tasks_on_browser += 1
raise last_err
class BrowserFleet:
def __init__(self, cfg=None, log=print):
self.cfg = cfg or FleetConfig()
self.metrics = Metrics()
self.log = log
self.task_q = queue.Queue()
self.result_q = queue.Queue()
self.workers = []
self._stop = threading.Event()
self._threads = []
self._launch_lock = threading.Lock()
self._pending = 0 # submitted-but-unreported tasks
self._pending_lock = threading.Lock()
# ---- public API ----------------------------------------------------------
def submit(self, fn, payload):
with self._pending_lock:
self._pending += 1
self.task_q.put((fn, payload))
def _emit(self, payload, res, err):
self.result_q.put((payload, res, err))
with self._pending_lock:
self._pending -= 1
def drain(self):
"""Yield (payload, result, error) as tasks complete; returns once
every submitted task has been reported. Accounting is fleet-level
(_pending): an abandoned thread never calls task_done, so the
queue's own counter would hang the drain forever."""
while True:
try:
yield self.result_q.get(timeout=0.25)
except queue.Empty:
pass
if self._pending <= 0 and self.result_q.empty():
return
def start(self):
threading.Thread(target=self._supervisor_loop, daemon=True).start()
threading.Thread(target=self._reaper_loop, daemon=True).start()
for i in range(self.cfg.slots):
self._spawn_worker(f"w{i}")
def shutdown(self):
self._stop.set()
for w in self.workers:
w.abandoned = True # unblock stale sleeps via supervisor stop
for t in self._threads:
t.join(timeout=5)
for w in self.workers:
w.stop_browser()
# abandoned-alive threads may leave a chromium tree behind; the
# reaper is stopped now, so purge those roots directly
for w in self.workers:
if w.orphan_tree_root:
_kill_tree(w.orphan_tree_root)
self.log(f"[fleet] shutdown metrics={self.metrics.summary()}")
# ---- internals -------------------------------------------------------------
def _spawn_worker(self, name):
w = _Worker(name, self.cfg, self.metrics, self.log, self._launch_lock)
self.workers.append(w)
# the browser starts inside the worker thread: the sync API refuses
# to attach to any thread already running an asyncio loop
t = threading.Thread(target=self._worker_loop, args=(w,), daemon=True)
t.start()
self._threads.append(t)
def _worker_loop(self, w):
w.start_browser()
while not self._stop.is_set():
try:
fn, payload = self.task_q.get(timeout=0.25)
except queue.Empty:
# idle: honour a pending recycle flag immediately
if w.needs_recycle() and self.task_q.empty():
w.recycle()
continue
try:
if w.needs_recycle():
w.recycle()
res = w.run_task(fn, payload)
self._emit(payload, res, None)
except Exception as e:
if not w.abandoned:
self._emit(payload, None, f"{type(e).__name__}: {e}")
finally:
try:
self.task_q.task_done() # may double-fire; harmless
except ValueError:
pass
def _supervisor_loop(self):
"""RSS ceiling flagging + task deadline enforcement (R2/R4).
The sync Playwright API has no cancellation. A task stuck in a
playwright call is freed by killing the driver tree; a task stuck
in pure Python never frees. Either way the thread may be a loss,
so the unit of recovery is the SLOT: report the task as failed,
abandon the thread, and spawn a replacement worker."""
while not self._stop.is_set():
for w in list(self.workers):
if (w.deadline is not None and time.time() > w.deadline
and w.driver_pid
and psutil.pid_exists(w.driver_pid)):
self.metrics.deadline_kills += 1
self.log(f"[supervisor] {w.name} task over "
f"{self.cfg.task_deadline_s:.0f}s -> killing "
f"driver tree, abandoning thread, respawning slot")
_kill_tree(w.driver_pid)
w.abandon()
self._emit(
w.current_payload, None,
f"DeadlineExceeded: task over "
f"{self.cfg.task_deadline_s:.0f}s, slot rebuilt")
self.workers.remove(w)
self._spawn_worker(f"{w.name}+")
continue
if w.browser_pid and not w.rss_flag:
rss = w.chrome_rss_mb()
if rss > self.cfg.rss_ceiling_mb:
w.rss_flag = True
self.log(f"[supervisor] {w.name} rss={rss:.0f}MB > "
f"{self.cfg.rss_ceiling_mb:.0f}MB -> recycle "
f"after current task")
time.sleep(0.5)
def _reaper_loop(self):
"""R5/R6: kill headless_shell trees owned by nobody."""
while not self._stop.is_set():
if self._launch_lock.locked():
time.sleep(2) # never reap a mid-flight launch
continue
owned = set()
orphan_roots = set()
for w in self.workers:
if w.orphan_tree_root:
orphan_roots.add(w.orphan_tree_root)
if w.browser_pid:
owned.add(w.browser_pid)
try:
proc = psutil.Process(w.browser_pid)
owned.update(c.pid
for c in proc.children(recursive=True))
except psutil.Error:
pass
now = time.time()
for proc in psutil.process_iter(
["pid", "cmdline", "create_time"]):
try:
cmd = " ".join(proc.info["cmdline"] or [])
if CHROME_PROC_RE not in cmd or "headless_shell" not in cmd:
continue
if now - proc.info["create_time"] < 20:
continue # grace: fresh trees may still register
pid = proc.info["pid"]
if pid in owned:
continue
if pid in orphan_roots:
_kill_tree(pid)
self.metrics.zombies_killed += 1
self.log(f"[reaper] SIGKILL abandoned worker tree "
f"pid={pid}")
continue
_kill_tree(pid)
self.metrics.zombies_killed += 1
self.log(f"[reaper] SIGKILL orphan chromium pid={pid}")
except (psutil.Error, ProcessLookupError, TypeError):
continue
time.sleep(10)
Usage — the whole point is that task code stays boring:
from browserfleet import BrowserFleet, FleetConfig
def render(ctx, payload):
page = ctx.new_page()
page.goto(payload["url"], timeout=30000)
return {"title": page.title(), "len": len(page.content())}
fleet = BrowserFleet(FleetConfig(slots=2, task_deadline_s=60.0))
fleet.start()
for url in urls:
fleet.submit(render, {"url": url})
for payload, result, err in fleet.drain():
if err:
print("failed:", payload, err)
else:
save(result)
fleet.shutdown()
Drop-in, one dependency you already have (psutil), and it turns “my scraper needs babysitting” into “my scraper files a report about itself.” That difference is the entire margin between a $500 automation gig and a support-ticket hobby.
Building something that has to survive contact with production — scraper fleets, AI pipelines, automation infra? That’s literally our day job. Email [email protected] for a custom version.