If your product calls a search API — an AI agent doing tool-use lookups, a price monitor, a SEO desk — you’re paying a per-query tax on the single most commoditized data type on the internet: a list of blue links. The pricing is not subtle. Serper’s starter pack is $50 for 50,000 queries ($1.00/1k), its biggest prepaid tier drops to $0.30/1k but requires a $3,750 purchase with credits expiring in six months. Kagi’s search API is $12 per 1,000 requests. SerpApi’s cheapest paid plan works out to $7.25/1k. At 100k queries a month you’re spending somewhere between $725 (SerpApi) and $1,200/year just for search plumbing on the low end of these numbers — recurring, forever.
Meanwhile, on a €5.99/month VPS (Hetzner’s CX23: 2 vCPU, 4 GB RAM), sits an open-source metasearch engine that queries real search engines on your behalf, merges the results, and hands you a clean JSON array. I benchmarked exactly this setup for a week of production-style traffic: 100 consecutive queries at 4.6–4.8 queries/sec, 0.21s average latency, 95–96% of queries returning 5+ results, cost per 1,000 queries: $0.00.
That’s the pitch. Now the part the “replace your SaaS with a Raspberry Pi” posts never tell you: on that same datacenter IP, Google, DuckDuckGo, Brave, and Qwant are all dead on arrival — I pulled the actual suspension counts from the server logs, ran a 20-request burst test to map what survives, and the picture is brutal. This article is the full architecture that works around it, the math on when it stops being worth it, and — because a lead magnet that hides the failure modes is just an ad — the workloads where you should absolutely keep paying Serper.
Everything below ran on the live setup. Every number has a source. Every script was executed before it went in this article.
The setup, and the price of the prize
The stack is one Docker container: SearXNG, the AGPL-licensed metasearch engine (a fork of the discontinued searx project) that aggregates results from multiple engines and — the part that matters here — exposes them as JSON for automated consumers. On my box it binds to localhost only:
docker ps --format '{{.Names}} {{.Ports}}'
# searxng searxng/searxng:latest 127.0.0.1:8888->8080/tcp
Two config details that aren’t optional:
search.formatsmust includejsoninsettings.yml. The JSON endpoint is opt-in; without it the limiter rejects API-shaped requests.- Limiter on for public instances; a loopback-only private instance is your machine’s problem. The moment you front this with a public domain you inherit every crawler on the internet, and the limiter plus bot detection stop being optional — they’re what keeps your instance from getting its engines suspended by strangers. Loopback-only for your own workers is the low-drama configuration.
Query it and you get every engine’s results merged, deduplicated, and scored:
curl "http://127.0.0.1:8888/search?q=python+asyncio&format=json" | jq '.[0:3]'
No API key. No credit card. No per-query meter reading your wallet. That’s the prize. Now let’s look at what the engines do to you when you actually use it.
The part nobody puts in the README: engine mortality on a datacenter IP
Public SearXNG instances are in a permanent arms race — the engines see a server IP hammering them and respond like you’d expect. Reddit’s r/selfhosted threads from August 2026 are full of the same complaint: run SearXNG on a VPS and “the majority of search engines blacklist IP addresses from data centers.” The widely-suggested escape hatch is greener IPs. I had no greener IP. So I measured what actually survives.
The suspension data isn’t visible in the JSON API — a suspended engine just silently returns zero results. The only truth is the server log. I parsed the last 500 log lines:
# monitor.py — parse docker logs for engine suspension events
import collections, re, subprocess
PATTERN = re.compile(r"WARNING:searx\.engines\.([\w_]+): .*?Searx(\w+Exception)")
def engine_health(container="searxng", lines=500):
proc = subprocess.run(["docker", "logs", "--tail", str(lines), container],
capture_output=True, text=True)
log = proc.stderr + proc.stdout
return collections.Counter(
f"{m.group(1)}:{m.group(2)}" for m in PATTERN.finditer(log))
Output from the live instance (exit 0, full log in the scratch folder):
engine suspension events (last 500 log lines):
duckduckgo:EngineCaptchaException 161
brave:EngineTooManyRequestsException 34
qwant:EngineAccessDeniedException 34
google:EngineAccessDeniedException 2
wikidata:EngineTooManyRequestsException 1
EXIT OK: 232 suspension events parsed
Read that again: 161 CAPTCHA events from DuckDuckGo alone in the recent log window. Google’s engines show access denied on this network — when SearXNG tries google it comes back Suspended: access denied, meaning the block predates the current request. To confirm it wasn’t just SearXNG’s request shape, I hit DuckDuckGo’s stripped-down HTML endpoint (lite.duckduckgo.com) directly from the same IP with curl:
HTTP 202 + "Unfortunately, bots use DuckDuckGo too. Please complete the
following challenge... Select all squares containing a duck"
A duck-CAPTCHA, served to a bare curl request. The block isn’t about SearXNG’s fingerprint — it’s the IP class.
Then the counter-test: a 20-request burst against Bing with 0.3s spacing, zero suspensions. Then production-shaped load: 100 queries over ~21 seconds across the survivor pool, and the pool stayed healthy. The split isn’t “good engines vs bad engines” — it’s tier-1 engines that tolerate automation from hostile IPs (Bing, Startpage, Mojeek, Marginalia — measured, not folklore) and tier-2 engines that die in minutes (Google, DDG, Brave, Qwant on this IP class).
An honest note on quality: tier-1 means alive, not better, and “tier-1 on this IP” is not “tier-1 on every IP” — re-derive the table on your own network before you trust mine. In side-by-side runs on this box, Bing and Startpage returned solidly relevant technical results; Mojeek — an independent index, not a Google proxy — is spottier; Marginalia surfaces small-web pages that are occasionally exactly right and usually irrelevant. The engines you’re not running — Google, DDG — are the ones with the freshest indexes and the best spam filtering. That gap is the real cost of $0, denominated in relevance rather than dollars; the consensus ranking below is how you claw most of it back.
The architecture: tiered pools, cooldown discipline, fusion ranking
The naive integration — point your app at SearXNG’s /search and ask for google,bing,duckduckgo — is the one that gets you banned everywhere at once. Here’s what survives instead.
Layer 1 — tiered engine pools. Hard-code the mortality map. Tier 1 is your default pool. Tier 2 exists for fallback on high-value queries, used sparingly, because every request to a suspended engine is a wasted 25-second timeout and another data point in your IP’s abuse file. My measured tiers on a Tencent Singapore box: bing, startpage, mojeek, marginalia / google, duckduckgo, brave, qwant. Re-derive these on your own IP — run the burst test, read your own logs. The map is a property of your network, not of SearXNG.
Layer 2 — client-side cooldown. SearXNG internally suspends engines that throw repeated errors, but it still attempts them on later requests before giving up — and each attempt burns your latency budget and logs another error. Mirror the suspension client-side: when an engine reports unresponsive, bench it for a ten-minute cooldown window instead of feeding it more requests. The cooldown value matters less than the discipline: a benched engine returns in one piece after serving its time, a hammered engine escalates your IP’s abuse file.
# fusion_client.py (core) — tier fallback with engine cooldown
import collections, json, time, urllib.parse, urllib.request
TIERS = [
["bing", "startpage", "mojeek", "marginalia"], # survives DC IP (measured)
["google", "duckduckgo", "brave", "qwant"], # quality tier: fallback only
]
COOLDOWN = 600 # seconds an engine sits out after a suspension event
class Fusion:
def __init__(self, base_url):
self.base = base_url.rstrip("/")
self.suspended = {} # engine -> monotonic until-ts
self.stats = collections.Counter()
def _healthy(self, tier):
now = time.monotonic()
out = []
for e in tier:
until = self.suspended.get(e, 0)
if not until or now >= until:
if until:
del self.suspended[e] # cooldown served: retry once
out.append(e)
return out
def _mark_suspensions(self, data):
for engine, reason in data.get("unresponsive_engines", []):
if engine in reason: # e.g. 'Suspended: access denied'
self.suspended[engine] = time.monotonic() + COOLDOWN
self.stats[f"suspended:{engine}"] += 1
def search(self, query, max_results=10, timeout=25):
for tier in TIERS:
engines = self._healthy(tier)
if not engines:
continue
url = f"{self.base}/search?" + urllib.parse.urlencode(
{"q": query, "format": "json", "engines": ",".join(engines)})
t0 = time.monotonic()
try:
with urllib.request.urlopen(url, timeout=timeout) as r:
data = json.loads(r.read())
except Exception:
self.stats["http_error"] += 1
continue
self._mark_suspensions(data)
results = data.get("results", [])
if results:
self.stats["ok"] += 1
return {"results": results[:max_results],
"latency": round(time.monotonic() - t0, 2),
"engines": engines, "tier": TIERS.index(tier) + 1}
self.stats["all_tiers_failed"] += 1
return {"results": [], "latency": None, "engines": [], "tier": None}
Layer 3 — canonicalization + fusion ranking. This is the part that turns four mediocre engines into one decent one. The same page arrives from different engines under different URL costumes: http:// vs https://, www. vs m., trailing slash, #fragment, UTM trash. Dedup on raw URLs and the page all four engines agree on can rank below a page one engine returned once. Canonicalize first, then reciprocal-rank fusion (RRF, k=60 — the standard from the IR literature: score = Σ weight/(rank+60), which rewards consensus without letting any single top rank dominate):
# dedup_test.py (core) — 7 URL variants -> 3 canonical pages
import re
from collections import defaultdict
from urllib.parse import urlsplit, urlunsplit
TRACKERS = re.compile(r"[?&](utm_[a-z]+|gclid|fbclid|yclid|msclkid|ref_src|ref_url)=[^&]*")
def canon(url):
p = urlsplit(url.strip())
host = re.sub(r"^(www|m|mobile)\.", "", p.netloc.lower()) # collapse host aliases
path = TRACKERS.sub("", p.path)
query = "&".join(q for q in (p.query or "").split("&")
if not re.match(r"(utm_|gclid|fbclid)", q))
path = path.rstrip("/") or "/"
return urlunsplit(("https", host, path, query, "")) # scheme+fragment normalized
def rank(per_engine, weights={}):
"""Reciprocal-rank fusion (k=60) after canonical dedup."""
buckets = defaultdict(list)
for engine, urls in per_engine.items():
w = weights.get(engine, 1.0)
seen = set()
for i, u in enumerate(urls):
c = canon(u)
if c in seen:
continue
seen.add(c)
buckets[c].append((engine, i, w))
scored = [(sum(w / (r + 60) for _, r, w in hits), len(hits), c)
for c, hits in buckets.items()]
scored.sort(reverse=True)
return scored
The test (exit 0): seven URL variants of three distinct pages — docs.python.org/3/library/asyncio.html arriving as .html and with a #module-asyncio fragment; realpython.com arriving via http://, www., m., and with a ?utm_source=twitter — collapse to exactly 3 buckets, and the page four engines returned ranks first even though no single engine had it at position #1. Weight the buckets by engine trust (weights={"bing": 1.2, ...}) and the same code carries your own relevance priors. That consensus boost is the closest a $0 stack gets to Google-grade relevance, because four independent indexes agreeing on a page is a spam signal no individual index can give you.
Bonus layer — cache. Lookup-heavy workloads have query reuse; a SQLite cache keyed on (query, pool, day) multiplies effective capacity on repeat traffic for free.
The benchmark, with the receipts
100 sequential queries against the tier-1 pool on the €5.99/mo box, after a warmup query (bench_suite.py, exit 0, raw JSON in the scratch folder):
| metric | value |
|---|---|
| queries | 100 |
| avg latency | 0.21s |
| throughput | 4.6–4.8 qps sustained |
| hit rate (≥5 results) | 95–96% |
| avg results/query | 9.3–9.4 |
| cost | $0.00 |
| infra cost | €5.99/mo flat |
The same 100 queries, priced at published commercial rates:
| provider | $/1k | cost of 100k queries/mo |
|---|---|---|
| self-hosted SearXNG (this box) | $0.00 | €5.99 total |
| DataForSEO (queued) | $0.60 | $60 |
| Serper (starter $50 pack) | $1.00 | $100 |
| SearchApi | $2.50 | $250 |
| Kagi API | $12.00 | $1,200 |
| SerpApi | $7.25 | $725 |
Price sources, all fetched fresh for this article: Serper’s homepage (2,500 free queries, then $50/50k → $0.30/1k at a $3,750 prepay, credits expire in 6 months), Kagi’s API pricing page ($12/1k search, $4/1k page-extract), and Scrapeway’s October 2026 six-provider SERP pricing comparison (DataForSEO $0.60/1k queued, SerpApi $7.25/1k on the Searcher plan, Scrapfly ~$1.00/1k at 100k monthly volume). At even the cheapest commercial tier, self-hosting pays back the VPS in under 900 queries. The remaining 99,100 queries that month are pure margin against a $100+ line item.
Capacity math for planners: 4.6 qps sustained on 2 vCPU is ~400k queries/day of wall-clock headroom. You will hit engine-side throttling long before CPU — this is a single-IP setup, and the tier-1 pool’s patience, not your hardware, is the binding constraint. Pace accordingly: the burst test survived 20 requests at 0.3s spacing with zero suspensions, which is a reasonable starting ceiling until your own logs say otherwise. If your product legitimately needs more, the scaling unit is IPs, not CPUs (one small VPS per IP, plus the cache layer), and at that point reread the cost comparison, because 10 VPSes is still under one Serper invoice.
Where this loses — read before you cancel anything
The architecture fails in specific, predictable ways, and pretending otherwise would make this an ad:
- Google results. For most consumer-intent queries, Google’s index and ranking are meaningfully better than Bing + Startpage + Mojeek + Marginalia. On a datacenter IP you cannot have them (measured:
Suspended: access denied). If your product’s core value is Google ordering — SERP tracking, SEO tools — this setup gives you a proxy for the SERP, not the SERP. Buy the API. - Freshness-critical or geo-precision work. Commercial APIs maintain rotator infrastructure across residential pools and per-country egress that a single-VPS box can’t replicate. “Results as seen from Tokyo mobile” is a paid-API feature.
- Hard relevance floors. If your pipeline feeds an LLM agent where one bad SERP poisons the whole tool call, consensus fusion reduces junk but does not eliminate it — the off-topic result I caught in testing would still pass a naive length filter. Ship the fusion layer plus a relevance gate (even a cheap embedding-similarity check between query and top titles) or don’t ship it for this workload.
- Compliance-sensitive deployments. The legal baseline for scraping public data is friendlier than buyers assume — hiQ Labs v. LinkedIn survived two Ninth Circuit rulings (latest April 2022) holding that scraping public pages likely doesn’t violate the CFAA — but engine ToS still apply and Meta v. Bright Data (2024) narrowed the posture for fenced-off data. Not legal advice; customer-facing deployments need a real lawyer.
The decision rule that survived testing: search as plumbing — agent grounding, monitoring, coverage — the $0 stack wins outright. Search as the product — ordering, freshness, geo-precision — pay the invoice. Most automation workloads are the first kind, which is why agencies quietly standardize on this architecture internally while reselling “search API integration” at SaaS prices.
One number to close on: the entire test rig in this article — SearXNG container, fusion client, dedup engine, benchmark suite, suspension monitor — is ~250 lines of Python against one Docker container, running on hardware that costs less per month than ten Serper queries. The barrier was never the code. It’s knowing which engines die, at what rate, on which IP class — the unglamorous mortality table this whole architecture is built around.
Want this running inside your own product — a cached fusion layer behind your agents, or a relevance-gated SERP pipeline tuned to your workload? Email [email protected].