<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Web-Scraping on Solvinc — AI's Uncharted Edge</title><link>https://blog.solvinc.io/tags/web-scraping/</link><description>Recent content in Web-Scraping on Solvinc — AI's Uncharted Edge</description><generator>Hugo -- 0.139.0</generator><language>en-us</language><lastBuildDate>Tue, 17 Nov 2026 00:00:00 +0700</lastBuildDate><atom:link href="https://blog.solvinc.io/tags/web-scraping/index.xml" rel="self" type="application/rss+xml"/><item><title>The Price Was Never in the HTML: Font Obfuscation, and the Cracker That Recovers Every Glyph</title><link>https://blog.solvinc.io/posts/font-obfuscation-glyph-mapping-cracker/</link><pubDate>Tue, 17 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/font-obfuscation-glyph-mapping-cracker/</guid><description>Sites ship randomized fonts so your scraper stores garbage instead of prices. I built the obfuscator, cracked it: 11/11 glyphs recovered, code included.</description></item><item><title>You Already Shipped a Search Engine and Didn't Charge for It: SQLite FTS5 Over Scraped Data, Tested</title><link>https://blog.solvinc.io/posts/fts5-search-engine-over-scraped-data-sqlite/</link><pubDate>Tue, 17 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/fts5-search-engine-over-scraped-data-sqlite/</guid><description>Your scraper already collects searchable data. Ship it as a product: SQLite FTS5 with bm25 ranking, triggers, and zero infrastructure, tested at 50k rows.</description></item><item><title>The JSON Is Already in the Page: Scraping Next.js Sites Through Their Own Data Endpoints</title><link>https://blog.solvinc.io/posts/nextjs-data-endpoints-scraping-embedded-payloads/</link><pubDate>Mon, 16 Nov 2026 15:09:30 +0000</pubDate><guid>https://blog.solvinc.io/posts/nextjs-data-endpoints-scraping-embedded-payloads/</guid><description>Stop parsing HTML on Next.js sites. The data endpoints, embedded payloads, and flight streams are already there - tested tooling, real extraction, real numbers.</description></item><item><title>The API Lost Its .proto File? Good. Here's How to Recover the Whole Schema From Wire Bytes Alone</title><link>https://blog.solvinc.io/posts/protobuf-wire-format-reverse-engineering-no-schema/</link><pubDate>Mon, 16 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/protobuf-wire-format-reverse-engineering-no-schema/</guid><description>Reverse-engineer any protobuf or gRPC API without its .proto: decode the wire format in pure Python, infer a schema from traffic, and prove it by re-encoding.</description></item><item><title>The Free Discount Hiding in Your Scraper: Conditional GET, 304s, and the Proxy Bill You're Paying for Nothing</title><link>https://blog.solvinc.io/posts/conditional-get-304-monitoring-cheaper-proxy-bills/</link><pubDate>Mon, 16 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/conditional-get-304-monitoring-cheaper-proxy-bills/</guid><description>Stock HTTP clients never send If-None-Match, so your monitors re-download unchanged pages at proxy rates. Tested validator store, A/B harness, and a challenge audit for lying ETags.</description></item><item><title>The Point Was Scored 8 Seconds Ago: Racing the Live-Score Feed With a Tested Python Poller (and Pricing What a Lead Is Worth)</title><link>https://blog.solvinc.io/posts/live-score-latency-race-espn-feed-betting/</link><pubDate>Sun, 15 Nov 2026 22:30:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/live-score-latency-race-espn-feed-betting/</guid><description>TV lags the venue by 5-10 seconds. I polled ESPN&amp;#39;s scoreboard API through a live NFL touchdown and measured where the free feed is fast, and where it lies.</description></item><item><title>I Replaced a $1,200/yr SERP API With a $6/mo VPS: the Tiered-Engine Architecture, the Ban Math, and Where It Loses</title><link>https://blog.solvinc.io/posts/self-hosted-searxng-serp-api-replacement/</link><pubDate>Sat, 14 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/self-hosted-searxng-serp-api-replacement/</guid><description>Build a $0 SERP API on self-hosted SearXNG: tiered engine pools, ban math, dedup + fusion ranking, and the honest benchmarks where it loses to paid APIs.</description></item><item><title>The Anime Girl Is a Hash Puzzle: I Built an Anubis Solver and Priced the Crawler Economy It Was Supposed to End</title><link>https://blog.solvinc.io/posts/anubis-proof-of-work-solver-crawler-economics/</link><pubDate>Fri, 13 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/anubis-proof-of-work-solver-crawler-economics/</guid><description>Anubis guards kernel archives and GNOME with SHA-256 puzzles. I built a solver, benchmarked every difficulty tier, and ran the crawler economics.</description></item><item><title>Your Playwright Fleet Loses 92 MB per Task — a Supervisor That Fixes Leaks, Crashes, and Zombie Browsers</title><link>https://blog.solvinc.io/posts/browser-fleet-supervisor-playwright-crash-recovery/</link><pubDate>Fri, 13 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/browser-fleet-supervisor-playwright-crash-recovery/</guid><description>Leaked Playwright contexts cost ~93 MB of RSS each on Chromium 153. A tested fleet supervisor: crash recovery by liveness probing, deadline kills, and a zombie reaper.</description></item><item><title>The Other Side of the Anti-Bot Arms Race: A Tripwire Layer That Fingerprints Bots by What They Can't Resist (Tested Python)</title><link>https://blog.solvinc.io/posts/scraper-tripwire-layer-seed-fields-trap-paths-defender-side/</link><pubDate>Thu, 12 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/scraper-tripwire-layer-seed-fields-trap-paths-defender-side/</guid><description>Trap paths, seed fields and keyless decoys: a defender-side bot tripwire that catches scrapers and form spam at the app layer. Tested, 37 assertions green.</description></item><item><title>Your Scraper Doesn't Need More Retries. It Needs a Congestion Controller.</title><link>https://blog.solvinc.io/posts/adaptive-throttling-aimd-circuit-breaker-scrapers/</link><pubDate>Thu, 12 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/adaptive-throttling-aimd-circuit-breaker-scrapers/</guid><description>Scrapers die by 403, not 429. A tested client-side AIMD controller and circuit breaker that converges on the target&amp;#39;s real rate limit instead of guessing.</description></item><item><title>Your Scraper Is Throttled by the Client, Not the Target: A Throughput Engineering Guide (Tested aiohttp vs httpx vs requests)</title><link>https://blog.solvinc.io/posts/async-scraper-throughput-engineering-aiohttp/</link><pubDate>Wed, 11 Nov 2026 09:35:00 +0000</pubDate><guid>https://blog.solvinc.io/posts/async-scraper-throughput-engineering-aiohttp/</guid><description>Most scrapers bottleneck on the HTTP client, not anti-bot. Benchmarked aiohttp vs httpx vs requests, then built a tested async fetcher.</description></item><item><title>Your Scraper Needs Japan at 3 A.M. Self-Hosted Geo Egress With WireGuard and HAProxy</title><link>https://blog.solvinc.io/posts/self-hosted-geo-egress-pool-wireguard-haproxy/</link><pubDate>Wed, 11 Nov 2026 04:08:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/self-hosted-geo-egress-pool-wireguard-haproxy/</guid><description>WireGuard + HAProxy geo egress pool on cheap VPSes, instead of $5.88/GB residential proxies. Tested configs, tested asyncio router, real per-GB math.</description></item><item><title>Your Scraper Logged In Once and Never Again: A Session Pool That Survives Login Walls, Challenges, and Token Rot</title><link>https://blog.solvinc.io/posts/authenticated-session-pool-login-wall-scraping-python/</link><pubDate>Tue, 10 Nov 2026 04:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/authenticated-session-pool-login-wall-scraping-python/</guid><description>A tested authenticated session pool for login-wall scraping — EWMA health scoring, challenge quarantine, auto-replenishment. Python, SQLite, zero API keys.</description></item><item><title>Your TLS Handshake Passes and Your JS Is Clean, But Akamai Still Bans You: The HTTP/2 SETTINGS Layer Nobody Forges</title><link>https://blog.solvinc.io/posts/http2-settings-fingerprint-akamai-bypass-python/</link><pubDate>Tue, 10 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/http2-settings-fingerprint-akamai-bypass-python/</guid><description>Akamai fingerprints your HTTP/2 SETTINGS frame before your first request. We measured six clients and forged three profiles in Python to cut proxy spend.</description></item><item><title>Stop Burning Proxy Credits: A Health-Scoring Pool That Auto-Quarantines Burned IPs Before They Cost You</title><link>https://blog.solvinc.io/posts/proxy-pool-health-scoring-auto-quarantine-sqlite-python/</link><pubDate>Mon, 09 Nov 2026 15:08:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/proxy-pool-health-scoring-auto-quarantine-sqlite-python/</guid><description>A proxy pool that scores every IP by EWMA block-rate, quarantines burned IPs, and steers traffic to healthy ones. Tested Python, SQLite, zero API keys.</description></item><item><title>Headless Chrome Is Not a Browser Disguise: Every JS-Layer Bot Signal, Measured, and the Cross-Origin Hole Nobody Patches</title><link>https://blog.solvinc.io/posts/headless-chrome-detection-signals-cdp-oopif-2026/</link><pubDate>Mon, 09 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/headless-chrome-detection-signals-cdp-oopif-2026/</guid><description>We drove raw Chrome over CDP and measured every JS-layer bot signal: UA-CH brand leaks, SwiftShader tells, and the cross-origin iframe hole. Exit-0 code.</description></item><item><title>The $6.85 Question: A Cost Engine That Tells You Whether to Buy a Scraping API, Roll Your Own, or Bolt a Captcha Solver Onto a Weak One</title><link>https://blog.solvinc.io/posts/captcha-antibot-cost-engine-solver-retry-python/</link><pubDate>Sun, 08 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/captcha-antibot-cost-engine-solver-retry-python/</guid><description>Anti-bot is a cost-engineering problem, not a hacking problem. A tested Monte Carlo engine that prices per-successful-row across APIs, solver retries, and risk.</description></item><item><title>Your Scraper Didn't Crash. It Started Lying: A Drift Sentinel That Catches Silent Breakage (Tested Python + SQLite)</title><link>https://blog.solvinc.io/posts/scraper-drift-detection-silent-breakage-sentinel/</link><pubDate>Sun, 08 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/scraper-drift-detection-silent-breakage-sentinel/</guid><description>Scrapers fail quietly: empty runs, null rates creeping up, layout drift. A tested DB-level drift sentinel that catches all three before your data customers do.</description></item><item><title>The Crawl Frontier: The Scraping Infrastructure Nobody Talks About (Tested Python + SQLite)</title><link>https://blog.solvinc.io/posts/crawl-frontier-politeness-python-sqlite/</link><pubDate>Sat, 07 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/crawl-frontier-politeness-python-sqlite/</guid><description>The queue, not the HTTP layer, is where scrapers die. A tested two-level frontier with per-domain politeness, robots.txt, and crash recovery in ~180 lines.</description></item><item><title>Your Scraper's TLS Handshake Is a Confession: JA4, Akamai, and the Fix That Actually Works</title><link>https://blog.solvinc.io/posts/ja4-tls-fingerprinting-scrapers-cloudflare-akamai/</link><pubDate>Sat, 07 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/ja4-tls-fingerprinting-scrapers-cloudflare-akamai/</guid><description>Akamai and Cloudflare fingerprint your TLS handshake pre-HTTP. Live JA4 captures, a tested 403-vs-200 proof, and an honest map of what impersonation fixes.</description></item><item><title>The Lottery Publishes Its Own Weakness Every Day — Python Is How You Read It</title><link>https://blog.solvinc.io/posts/lottery-remaining-prizes-advantage-scanner-texas-python/</link><pubDate>Fri, 06 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/lottery-remaining-prizes-advantage-scanner-texas-python/</guid><description>Scratch-off tickets keep getting sold after their top prizes are gone. Texas publishes the claim data daily. Build the scanner that tells you which games still have real money in play.</description></item><item><title>The Settlement Stack: 91% of Class Action Money Goes Unclaimed — Farm Your Share With Python</title><link>https://blog.solvinc.io/posts/class-action-settlement-stack-python-claims-scanner/</link><pubDate>Mon, 02 Nov 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/class-action-settlement-stack-python-claims-scanner/</guid><description>Median claim rate on class action settlements is 9%. Build a Python scanner that live-tracks no-proof settlements and stack every check you&amp;#39;re owed.</description></item><item><title>Residential Proxy Rotation for Web Data: Build a Multi-Source Price Scraper</title><link>https://blog.solvinc.io/posts/residential-proxy-arbitrage-web-data/</link><pubDate>Thu, 22 Oct 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/residential-proxy-arbitrage-web-data/</guid><description>A complete guide to building resilient web data pipelines using Python, rotating residential proxies, and LLMs for structured extraction.</description></item><item><title>Sell Scrapers on the Apify Store: Pay-Per-Result Data Products in Python</title><link>https://blog.solvinc.io/posts/sell-scrapers-apify-store-pay-per-result-data-products-python/</link><pubDate>Wed, 21 Oct 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/sell-scrapers-apify-store-pay-per-result-data-products-python/</guid><description>Publish a scraper on the Apify Store, charge per result, keep 80% minus compute. Working Python Actor, real margins, and the legal and ban risks.</description></item><item><title>Web Scraping as a Service: How to Sell Public Web Data Legally (and Price It Above $500)</title><link>https://blog.solvinc.io/posts/web-scraping-as-a-service-sell-public-web-data-legally/</link><pubDate>Fri, 16 Oct 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/web-scraping-as-a-service-sell-public-web-data-legally/</guid><description>Sell public web data as a service: what hiQ v. LinkedIn decided, where scraping turns illegal, a tested Python scraper, real costs and pricing.</description></item><item><title>Build a Competitor Price Tracker with Python: A Step-by-Step Guide</title><link>https://blog.solvinc.io/posts/competitor-price-tracker-python/</link><pubDate>Fri, 09 Oct 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/competitor-price-tracker-python/</guid><description>Build your own Python price tracker to monitor competitor pricing across 50+ products daily and automate your repricing decisions.</description></item><item><title>Web Scraping for Small Business: What's Legal, What Works</title><link>https://blog.solvinc.io/posts/web-scraping-small-business-guide/</link><pubDate>Wed, 07 Oct 2026 00:00:00 +0700</pubDate><guid>https://blog.solvinc.io/posts/web-scraping-small-business-guide/</guid><description>Legit uses for web scraping in a small business, the real legal rules (ToS vs law, robots.txt, personal data), and a minimal Python scraper that won&amp;#39;t get you sued.</description></item></channel></rss>