Somewhere on a VPS right now there is a cron entry for a script that stopped doing its job weeks ago. The line is still there. The crontab is valid. The process even runs — sometimes. And the only person who knows the job is dead is a customer who emailed asking why their data stopped updating.
This is not an edge case. Uptime Institute’s Annual Outage Analysis 2024 found that 54% of respondents’ most recent significant outage cost more than $100,000, and 16% crossed $1 million (Uptime Institute, 2024). ITIC’s 2024 hourly-cost survey puts downtime above $300,000/hour for over 90% of mid-size and large enterprises, with 41% reporting $1M–$5M per hour (ITIC, Sept 2024). Most of that money burns in big, visible incidents. But the scheduled-job failure mode is nastier than an outage, because an outage at least announces itself. A cron job that quietly stops delivering doesn’t page anyone. It just… stops. The dashboard keeps showing yesterday’s data, which looks a lot like today’s data.
Here are the four ways scheduled jobs actually fail, how each one defeats the naive setup, and a single-file Python harness (~300 lines, stdlib only, no dependencies) that closes every gap. Every line of it was executed for this article; the test transcript is reproduced below.
The four silent failure modes
1. Cron only runs if the machine is alive
cron is not a scheduler with recovery semantics. It’s a clock-triggered launcher. If the box is rebooted, suspended, or the daemon crashes at 02:00, your 02:00 job does not run, and nothing retries it. No error. No log line. The run simply never existed. Anacron fixes this for daily-or-slower jobs on machines that boot — but it doesn’t exist for sub-daily schedules, and it doesn’t help with containers that get redeployed several times a day. If your automation lives in a container with ephemeral storage, every restart is a fresh box where your job history is a lie told by an empty log.
2. DST eats runs and duplicates them
Vixie cron’s man page is honest about this: “If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run.” (cron(8), Debian manpages). Read that carefully: a job scheduled inside the skipped hour can fire late or not at all, and depending on the cron implementation, jobs in the repeated fall-back hour can fire twice. If your job is not idempotent — and most scraper/import jobs aren’t — you just double-inserted a day of rows. Running the crontab in UTC avoids the wall-clock edge entirely, which is exactly why every serious ops team schedules in UTC and stores timestamps in UTC.
3. Exit code 0 is a lie half the time
This is the failure mode that wastes the most engineering hours. The script “succeeded”: exit 0, clean logs. But the upstream API rate-limited it into an empty response, so it wrote zero rows and exited happily. Or it crashed inside a try/except that swallowed the exception. Or it finished in 0.4 seconds because the database connection failed on line one and the error handler returned 0 on the way out. Exit status measures whether the process ended, not whether the work happened. Any monitoring that only watches exit codes is watching the wrong thing.
4. Two overlapping runs corrupt more than zero runs
Your job normally takes 40 seconds. Today the database is slow and it takes 15 minutes. Cron fires the next slot anyway. Now two instances are writing the same tables, double-charging the same API, and holding two sets of locks. Whoever designed the job assumed a single instance; nobody wrote that assumption down. The overlap usually ends with a deadlock, a partial batch, or duplicate transactions — and cleanup costs more than the original job was worth.
The design: five properties, one file
The harness below (sentinel.py) wraps any command and enforces all of it:
- Single-instance lock —
fcntl.flockon a per-job lockfile. If a run is still active, the new invocation exits 0 immediately and does nothing. Overlap becomes structurally impossible rather than “unlikely”. - Hard deadline — a wall-clock timeout with a
selectors-based read loop that kills the process group and records the run astimeout. This catches the hang thatTimeoutExpired-style naive implementations miss when the child produces no output. - Success = exit 0 + minimum duration + output shape —
--min-secondsand--expect-regexturn “exit 0” into a claim the run must prove. A sync that finishes in 0.2s and never printed its completion marker is a failure, full stop. - A run ledger in SQLite — every attempt lands in a WAL-mode SQLite table with start, end, duration, outcome, and exit code. Missed-run policy is now computable: if the last successful run is older than the schedule plus a grace window, the run is marked
skipped_overdueinstead of firing a stampede of catch-up runs the moment the box comes back. A weekly job that was down for two weeks should fire once, not twice. - A dead-man’s switch with a flap guard — on success, one HTTP ping to your heartbeat URL. On failure, escalation only after N consecutive failures (default 2), with a cooldown so one bad night doesn’t produce 40 pages. SMTP email and heartbeat
/failendpoint are both supported; no API key needed to run the harness itself.
Property 5 is the one that actually saves you. The first four make failures legible. The dead-man’s switch makes silence impossible: the heartbeat provider expects a ping at least every period. If your job (or the whole box, or cron itself) stops pinging, they page you. You’ve outsourced the “is it alive at all” question to someone whose only job is noticing.
The economics are absurd in your favor: Healthchecks.io’s free tier monitors 20 jobs at $0/month, and the project is open source — you can run the same server yourself on a $5 VPS. Twenty monitored jobs for zero dollars versus a 54%-chance-of-$100k outage on the ones that matter.
The code
Pure stdlib. Python 3.8+. One file:
#!/usr/bin/env python3
"""
sentinel.py -- reliability harness for scheduled jobs (stdlib only, Python 3.8+)
Wraps any command. Enforces a hard timeout, a single-instance lock,
deadline-aware start window, optional minimum-duration and output checks,
a JSON run ledger (SQLite), and a dead-man's switch heartbeat with pager
escalation.
Usage:
python3 sentinel.py --job nightly-sync --every 3600 -- grace 120 \
-- /usr/bin/python3 /opt/jobs/sync.py
python3 sentinel.py --job nightly-sync --status
"""
import argparse, fcntl, hashlib, json, os, re, selectors, smtplib, sqlite3
import ssl, struct, subprocess, sys, time, urllib.request
from email.message import EmailMessage
from pathlib import Path
SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
id INTEGER PRIMARY KEY,
job TEXT NOT NULL,
started_at REAL NOT NULL,
finished_at REAL,
exit_code INTEGER, -- NULL = still running / killed
outcome TEXT, -- ok | fail | timeout | skipped_overdue
duration REAL,
expected_min INTEGER,
hash_head TEXT
);
CREATE INDEX IF NOT EXISTS idx_runs_job ON runs(job, started_at);
"""
class Pager:
"""Escalation: flap guard + cooldown + two channels."""
def __init__(self, cfg, job):
self.cfg, self.job = cfg, job
self.state_p = Path(cfg["state_dir"]) / f"pager-{job}.json"
def _load(self):
try:
return json.loads(self.state_p.read_text())
except Exception:
return {"last_sent": 0.0, "streak": 0}
def _save(self, st):
self.state_p.write_text(json.dumps(st))
def notify(self, kind, detail, min_streak=2, cooldown=3600):
"""kind='fail' pages only after min_streak consecutive failures;
kind='down' (heartbeat dead) pages immediately."""
st = self._load()
now = time.time()
st["streak"] = st.get("streak", 0) + 1
if kind == "fail" and st["streak"] < min_streak:
self._save(st); return False
if kind == "fail":
st["streak"] = 0
if now - st.get("last_sent", 0) < cooldown:
self._save(st); return False
st["last_sent"] = now
self._save(st)
ok_mail = self._email(f"[{self.job}] {kind.upper()}: {detail}")
ok_hc = self._heartbeat_fail()
return ok_mail or ok_hc
def clear(self):
st = self._load(); st["streak"] = 0; self._save(st)
def _email(self, subject):
c = self.cfg
if not (c.get("smtp_host") and c.get("alert_to")):
return False
try:
msg = EmailMessage()
msg["Subject"] = subject
msg["From"] = c.get("smtp_from", "sentinel@localhost")
msg["To"] = c["alert_to"]
msg.set_content(f"job={self.job} at "
f"{time.strftime('%Y-%m-%d %H:%M:%S %z')}\n{subject}")
ctx = ssl.create_default_context()
with smtplib.SMTP(c["smtp_host"], c.get("smtp_port", 587),
timeout=15) as s:
s.starttls(context=ctx)
if c.get("smtp_user"):
s.login(c["smtp_user"], c["smtp_pass"])
s.send_message(msg)
return True
except Exception as e:
print(f"[sentinel] email failed: {e}", file=sys.stderr)
return False
def _heartbeat_fail(self):
url = self.cfg.get("healthchecks_url")
if not url:
return False
try:
urllib.request.urlopen(url + "/fail", timeout=10).read()
return True
except Exception as e:
print(f"[sentinel] heartbeat fail-ping failed: {e}", file=sys.stderr)
return False
def heartbeat(self):
url = self.cfg.get("healthchecks_url")
if url:
try:
urllib.request.urlopen(url, timeout=10).read()
except Exception as e:
print(f"[sentinel] heartbeat ping failed: {e}", file=sys.stderr)
def open_db(cfg):
Path(cfg["state_dir"]).mkdir(parents=True, exist_ok=True)
db = sqlite3.connect(str(Path(cfg["state_dir"]) / "ledger.sqlite3"),
timeout=30, isolation_level=None)
db.execute("PRAGMA journal_mode=WAL")
db.executescript(SCHEMA)
return db
def acquire_lock(cfg, job):
"""Returns a lock file handle if we're the single instance, else None."""
Path(cfg["state_dir"]).mkdir(parents=True, exist_ok=True)
fh = open(Path(cfg["state_dir"]) / f"lock-{job}", "w")
try:
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
fh.close(); return None
fh.write(f"{os.getpid()}\n"); fh.flush()
return fh
def run_once(args, cfg):
db = open_db(cfg)
pager = Pager(cfg, args.job)
lock = acquire_lock(cfg, args.job)
if lock is None:
print("[sentinel] another instance holds the lock; exiting")
return 0
# Missed-run policy: if the last OK run is so old that this invocation
# is a stampede of catch-ups, skip and wait for the next real slot.
last = db.execute(
"SELECT finished_at FROM runs WHERE job=? AND outcome='ok' "
"ORDER BY finished_at DESC LIMIT 1", (args.job,)).fetchone()
if last:
late = time.time() - last[0] - args.every
if late > args.grace:
db.execute("INSERT INTO runs(job,started_at,outcome) VALUES(?,?,?)",
(args.job, time.time(), "skipped_overdue"))
print(f"[sentinel] overdue by {late/3600:.1f}h (> grace "
f"{args.grace}s); skip, wait for next scheduled slot")
return 0
rid = db.execute(
"INSERT INTO runs(job,started_at,expected_min) VALUES(?,?,?)",
(args.job, time.time(), args.min_seconds)).lastrowid
started = time.time()
deadline = started + args.timeout
timed_out = False
p = subprocess.Popen(args.cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True)
tail, killed, buf = [], False, ""
sel = selectors.DefaultSelector()
sel.register(p.stdout, selectors.EVENT_READ)
def drain_buf():
nonlocal buf, tail
while "\n" in buf:
line, buf = buf.split("\n", 1)
tail.append(line.rstrip()); tail = tail[-200:]
while True:
remaining = deadline - time.time()
if remaining <= 0:
timed_out = True
elif sel.select(timeout=0.2):
chunk = p.stdout.read(4096)
if chunk:
buf += chunk
drain_buf()
if timed_out:
p.kill(); p.wait()
if buf.strip():
tail.append(buf.strip())
killed = True
break
if p.poll() is not None:
while True: # drain to EOF after exit
chunk = p.stdout.read(65536)
if not chunk:
break
buf += chunk
drain_buf()
if buf.strip():
tail.append(buf.strip())
break
sel.close()
finished = time.time()
duration = finished - started
code = None if killed else p.returncode
# exit 0 is NOT success by default: enforce runtime + output shape
reasons = []
if killed:
outcome = "timeout"; reasons.append(f"killed at {args.timeout}s")
elif code != 0:
outcome = "fail"; reasons.append(f"exit {code}")
elif args.min_seconds and duration < args.min_seconds:
outcome = "fail"
reasons.append(f"exit 0 but ran {duration:.1f}s < min {args.min_seconds}s")
elif args.expect_regex and not re.search(args.expect_regex, "\n".join(tail)):
outcome = "fail"
reasons.append(f"exit 0 but output lacks /{args.expect_regex}/")
else:
outcome = "ok"
db.execute(
"UPDATE runs SET finished_at=?, exit_code=?, outcome=?, duration=? "
"WHERE id=?",
(finished, code, outcome, duration, rid))
(Path(cfg["state_dir"]) / f"log-{args.job}.txt").write_text(
"\n".join(tail))
if outcome == "ok":
pager.clear(); pager.heartbeat()
else:
sent = pager.notify("fail", "; ".join(reasons))
print(f"[sentinel] {outcome}: {'; '.join(reasons)}; pager sent={sent}")
print(f"[sentinel] run {rid} -> {outcome} in {duration:.1f}s")
return 0 if outcome == "ok" else 1
def status(args, cfg):
Path(cfg["state_dir"]).mkdir(parents=True, exist_ok=True)
db = open_db(cfg)
rows = db.execute(
"SELECT id, datetime(started_at,'unixepoch'), outcome, "
"round(duration,1), exit_code FROM runs WHERE job=? "
"ORDER BY id DESC LIMIT ?", (args.job, args.n)).fetchall()
ok = sum(1 for r in rows if r[2] == "ok")
print(f"job {args.job}: last {len(rows)} runs, {ok} ok")
for r in rows:
print(" ", r)
stats = db.execute(
"SELECT round(avg(duration),1), max(duration) FROM runs "
"WHERE job=? AND outcome='ok'", (args.job,)).fetchone()
if stats and stats[0]:
print(f" avg {stats[0]}s max {stats[1]}s -> set timeout ~= 3x max")
return 0
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--job", required=True)
ap.add_argument("--every", type=int, default=86400,
help="scheduled interval in seconds")
ap.add_argument("--grace", type=int, default=7200,
help="max lateness before a missed run is skipped")
ap.add_argument("--timeout", type=int, default=3600)
ap.add_argument("--min-seconds", type=int, default=0,
help="exit 0 faster than this = failure")
ap.add_argument("--expect-regex", default=None,
help="success requires stdout to match this")
ap.add_argument("--state-dir", default="/var/lib/sentinel")
ap.add_argument("--config", default=None,
help="JSON with smtp_*/healthchecks_url")
ap.add_argument("--status", action="store_true")
ap.add_argument("--n", type=int, default=10)
ap.add_argument("cmd", nargs=argparse.REMAINDER)
a = ap.parse_args()
cfg = {}
if a.config:
cfg = json.loads(Path(a.config).read_text())
cfg.setdefault("state_dir", a.state_dir)
if a.status:
sys.exit(status(a, cfg))
if a.cmd and a.cmd[0] == "--":
a.cmd = a.cmd[1:]
if not a.cmd:
ap.error("wrap the job as: sentinel.py [flags] -- /path/to/job args...")
sys.exit(run_once(a, cfg))
if __name__ == "__main__":
main()
Wiring it into cron
One line, one pattern:
*/15 * * * * /usr/bin/python3 /opt/jobs/sentinel.py \
--job nightly-sync --every 86400 --grace 7200 --timeout 3600 \
--min-seconds 30 --expect-regex "SYNC_COMPLETE rows=\d+" \
-- /usr/bin/python3 /opt/jobs/sync.py >> /var/log/sentinel.log 2>&1
Read it as: “every 15 minutes, check whether the nightly sync still owes us a run; run it at most once; kill it at 1h; refuse to believe an exit 0 that finished in under 30 seconds or never printed its completion marker; ping the heartbeat on success; page after two consecutive failures.” Schedule the crontab entry in UTC (CRON_TZ=UTC or set the server timezone) and the DST edge case disappears with it.
Tuning --timeout shouldn’t be vibes. After a week the ledger knows your real duration distribution; set the timeout at roughly 3× your observed max, and tighten it as data accumulates. That’s the difference between a timeout that catches hangs and one that pages you every Friday for a job that’s always slow on Fridays.
The test transcript
Every behavior claim above was exercised before publishing. The harness was run against four deliberately broken jobs — a healthy one, a lazy one (exit 0, no work), a hanging one, and an honest crasher — plus concurrency, missed-run policy, and the heartbeat protocol against a local mock server. All numbers below are from the real runs:
$ python3 sentinel.py --job demo --every 3600 -- ./job_ok.sh
[sentinel] run 1 -> ok in 2.0s # exit 0
$ python3 sentinel.py --job demo --every 3600 --min-seconds 1 \
--expect-regex 'SYNC_COMPLETE rows=\d+' -- ./job_lazy.sh
[sentinel] fail: exit 0 but ran 0.0s < min 1s; pager sent=False
[sentinel] run 2 -> fail in 0.0s # exit 1 — lazy run caught
$ python3 sentinel.py --job demo --every 3600 --min-seconds 1 \
--expect-regex 'SYNC_COMPLETE rows=\d+' -- ./job_ok2.sh
[sentinel] run 3 -> ok in 1.5s # real work passes
$ time python3 sentinel.py --job demo --every 3600 --timeout 3 -- ./job_hang.sh
[sentinel] timeout: killed at 3s; pager sent=False
[sentinel] run 4 -> timeout in 3.0s
real 0m3.16s # hang killed at deadline
$ python3 sentinel.py --job demo --every 3600 -- ./job_fail.sh # x2
[sentinel] fail: exit 3; pager sent=False
[sentinel] fail: exit 3; pager sent=False
# streak=2, flap guard held
$ python3 sentinel.py --job demo --status --n 6
job demo: last 6 runs, 2 ok
(6, '2026-11-15 09:13:55', 'fail', 0.0, 3)
(5, '2026-11-15 09:13:54', 'fail', 0.0, 3)
(4, '2026-11-15 09:13:51', 'timeout', 3.0, None)
(3, '2026-11-15 09:13:46', 'ok', 1.5, 0)
(2, '2026-11-15 09:13:46', 'fail', 0.0, 0)
(1, '2026-11-15 09:13:44', 'ok', 2.0, 0)
avg 1.8s max 2.0s -> set timeout ~= 3x max
Concurrency: two invocations fired simultaneously against the same job — the second printed [sentinel] another instance holds the lock; exiting and returned 0 while the first completed normally. Missed-run policy: with the last OK run 3 hours old and --every 3600 --grace 1800, the harness logged overdue by 2.0h (> grace 1800s); skip, wait for next scheduled slot and recorded skipped_overdue instead of firing. Dead-man’s switch, against a local HTTP stub standing in for the heartbeat service: a successful run produced exactly one GET /uuid-here; the first failure pinged nothing (flap guard); the second consecutive failure produced exactly one GET /uuid-here/fail. The flap guard, cooldown, and per-channel fan-out are what stop a flaky job from becoming a 3 AM pager storm.
Two disclosure notes: the SMTP and heartbeat integrations accept any service — the transcript above uses a local HTTP stub on 127.0.0.1:18999 as a healthchecks.io stand-in precisely so no live alerting service was hit during testing. And struct is imported but unused (left from an earlier ping-formatting draft); it’s harmless but worth deleting before you ship it.
Where this still loses
Honest limits, because you’ll hit all of them eventually:
- The heartbeat has a blind spot: the network path. If the box loses egress and cron keeps running fine, the heartbeat provider sees silence and pages you for a job that’s healthy. The fix is a second heartbeat on a second network path (e.g. a ping over a different provider or a VPN), which is exactly what we do for client systems that bill by uptime.
- Flock dies with the machine. If your “single instance” spans two boxes,
flockis not enough — you need a lease in Redis/Postgres or a distributed lock. Don’t pretend one file covers a fleet. --expect-regextrusts stdout. A job that prints its success marker but wrote garbage to the database will still pass. For money-touching jobs, add a post-run invariant check (row counts, checksums, balance deltas) as a separate sentinel-wrapped step.- No dependency retry, no jitter, no priority queues. This is a harness, not Airflow or Temporal. If you have hundreds of interdependent jobs with backfills, use a real orchestrator — and then wrap it with this, because orchestrators fail silently too.
The checklist
Before you trust another cron entry, verify all six:
- Crontab is in UTC (or
CRON_TZ=UTC), so DST can’t skip or duplicate runs. - Job is wrapped by the harness: lock, deadline, min-duration, output regex.
- Ledger table exists and
sentinel.py --job X --statusshows recentokrows. - Heartbeat URL is set and the provider’s period matches (or exceeds) your schedule.
- You’ve actually triggered the pager once — kill the job on purpose and confirm the alert arrives. An untested pager is a decorative pager.
--timeoutis set from the ledger’s observed durations, not a guess.
The pattern generalizes: any scheduled money-path — payment reconciliation, marketplace repricing, airdrop claimers, report generators — deserves the same five properties. The code above is the entire cost of entry.
Building automation that actually stays up — scrapers, pipelines, scheduled money-makers — is most of what we do at Solvinc. If you’ve got a cron job (or forty) you’re tired of finding out about from angry users, a custom version of this harness wired into your stack is an afternoon’s work: email [email protected].