Your scraper sends a perfect User-Agent header, a perfect header order, residential proxies, the works. The request still gets a 403 from a site that loads fine in Chrome. The usual advice — “rotate your proxies” — misses the actual mechanism, and understanding the mechanism is worth real money, because it decides whether your data pipeline costs you a $1,500/month unblocker API or a few cents of compute.

The mechanism is TLS fingerprinting. Before your scraper sends a single HTTP header — before the server even knows which URL you want — your client completes a TLS handshake, and that handshake describes your software in detail. Akamai, Cloudflare, AWS WAF, and Google Cloud Armor all read it. In this article we’ll reproduce the JA4 algorithm from the official spec byte-for-byte, capture the fingerprints of real Chrome, python-requests, and curl-impersonate on live requests, and run the decisive experiment: identical HTTP requests to a real Akamai-protected site where the only variable is the handshake — one gets 403, the other 200, every time.

JA4: what the server actually reads

Every HTTPS connection starts with a Client Hello: a message listing the TLS versions, cipher suites, extensions, and signature algorithms your client supports. This list is written by your TLS library, not your HTTP code — which is why faking User-Agent changes nothing about it. python-requests uses OpenSSL with urllib3’s defaults; Chrome uses BoringSSL with its own curated list. They are different in obvious, enumerable ways.

JA3, published by Salesforce in 2017, hashed those fields into an MD5 fingerprint. It worked for years, but it has a structural flaw: it hashes the fields in wire order, and modern browsers randomize parts of that order (Chrome reorders and adds GREASE — deliberately meaningless values — to frustrate exactly this kind of parsing). A single browser can produce many JA3 values, so blocklists built on JA3 both false-positive and miss.

JA4, published in 2023 by FoxIO (open source, JA4 itself under BSD-3-Clause), fixes this by sorting values before hashing and structuring the fingerprint into three human-readable segments:

t13d1516h2_8daaf6152771_e5627efa2ab1
└┬┘└┬┘└┬┘└┬┘└┬┘ └────┬────┘ └────┬────┘
 │ │ │  │  │      │           │
 │ │ │  │  │      │           └─ sha256 of sorted extensions
 │ │ │  │  │      │              (minus SNI/ALPN) + sig algs, 12 chars
 │ │ │  │  │      └─ sha256 of sorted cipher suites, 12 chars
 │ │ │  │  └─ ALPN (h2 = HTTP/2, h1 = HTTP/1.1)
 │ │ │  └─ extension count
 │ │ └─ cipher count
 │ └─ d = SNI present, i = absent
 └─ t = TCP, 13 = TLS 1.3

That example is not invented — it is the worked example from the FoxIO spec, and here is the algorithm reproduced from scratch in ~30 lines of Python. No TLS library involved; you only need the parsed Client Hello fields, which any fingerprinting service will hand you:

import hashlib

# Fields read from a real Client Hello (this set is the FoxIO spec's example)
ciphers = "1301,1302,1303,c02b,c02f,c02c,c030,cca9,cca8,c013,c014,009c,009d,002f,0035".split(",")
extensions = "001b,0000,0033,0010,4469,0017,002d,000d,0005,0023,0012,002b,ff01,000b,000a,0015".split(",")
sig_algs = "0403,0804,0401,0503,0805,0501,0806,0601".split(",")

# Segment b: ciphers sorted as hex strings, hashed, truncated to 12 chars
b = hashlib.sha256(",".join(sorted(ciphers)).encode()).hexdigest()[:12]
assert b == "8daaf6152771"

# Segment c: extensions sorted (SNI 0000 and ALPN 0010 removed),
# then sig algs appended in wire order, underscore-delimited
kept = sorted(e for e in extensions if e not in ("0000", "0010"))
c = hashlib.sha256((",".join(kept) + "_" + ",".join(sig_algs)).encode()).hexdigest()[:12]
assert c == "e5627efa2ab1"

print(f"t13d1516h2_{b}_{c}")
# t13d1516h2_8daaf6152771_e5627efa2ab1

Run that and it prints the spec’s fingerprint exactly. JA4 is now supported natively in Zeek, Suricata, and Wireshark, and it’s exposed in Cloudflare’s Firewall Rules, Bot Analytics, and Workers. When Cloudflare, Akamai, or AWS scores your traffic, a JA4-family fingerprint is part of the score.

What every client actually looks like, measured live

The site tls.peet.ws (TrackMe, open source on GitHub) echoes back the fingerprint your client presents, including JA4 and the Akamai HTTP/2 fingerprint — a hash of your HTTP/2 SETTINGS frame, WINDOW_UPDATE, PRIORITY, and pseudo-header order. That second fingerprint matters because TLS is only half of it; the HTTP/2 layer leaks you just as hard.

Here is a live run from this VPS (curl_cffi 0.17.0, full script tested, exit 0):

from curl_cffi import requests as creq
import requests as std_requests

URL = "https://tls.peet.ws/api/all"

for label, resp in [
    ("python-requests", std_requests.get(URL, timeout=30)),
    ("curl_cffi (no target)", creq.get(URL, timeout=30)),
    ("curl_cffi chrome", creq.get(URL, timeout=30, impersonate="chrome")),
    ("curl_cffi firefox", creq.get(URL, timeout=30, impersonate="firefox")),
]:
    d = resp.json()
    print(f"{label:22s} JA4={d['tls']['ja4']}")

Output (this is the actual captured data, not an illustration):

python-requests        JA4=t13d1712h1_ab0a1bf427ad_882d495ac381
curl_cffi (no target)  JA4=t13d2712h2_b4f9224df7c1_cbb9361b6bf9
curl_cffi chrome       JA4=t13d1517h2_8daaf6152771_cb7bf5808d99
curl_cffi firefox      JA4=t13d1517h2_8daaf6152771_3cbfd9057e0d
REAL Chrome 156        JA4=t13d1519h2_8daaf6152771_3d1b1b7bef36

(Real Chrome 156.0.8078.4 was captured by driving an actual headless instance over the Chrome DevTools Protocol and reading what its own network stack presented — script also tested, exit 0.)

Read those five lines carefully, because they contain three lessons:

1. Default clients are trivially identifiable. t13d1712h1_ab0a1bf427ad_... — no browser on earth sends h1 (HTTP/1.1 ALPN) with 17 ciphers in that arrangement. A WAF rule that says “block anything whose JA4 segment-a isn’t a known browser prefix” kills python-requests, httpx, Go’s net/http, and every tutorial scraper with one line of config.

2. Cipher lists alone don’t identify a browser. curl_cffi’s Chrome and Firefox targets share the same cipher-hash segment (8daaf6152771) — and so does real Chrome. The cipher list is what curl-impersonate patches most easily; what actually separates browser families is the extension hash (cb7bf5808d99 vs 3cbfd9057e0d vs real Chrome’s 3d1b1b7bef36) and the HTTP/2 SETTINGS fingerprint (1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p for real Chrome — the window-update increment and pseudo-header order are browser-specific).

3. “Impersonating Chrome” is a moving target. Real Chrome 156 shows 19 extensions (1519h2); the freshest curl_cffi target ships 17 (1517h2). The cipher hash matches, the structure matches, but the extension delta is real and fingerprint databases notice. This is why the maintainers keep shipping new targets with every Chrome release (curl_cffi 0.17.0 updated to curl-impersonate 2.3.0 with Chrome 151/152 and Firefox 153–155 profiles) and why a scraper that “works” this quarter can silently rot. Version-pinning your impersonation target and re-checking against tls.peet.ws should be a recurring test in your CI, not a one-time setup.

The experiment: same request, same headers, opposite verdicts

Fingerprints are interesting in the abstract; here is the consequence. One Akamai-protected production site, two clients, byte-identical HTTP semantics (same URL, same User-Agent header), only the TLS stack differs:

import requests
from curl_cffi import requests as creq

URL = "https://www.idealo.de/robots.txt"
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36")

r1 = requests.get(URL, timeout=30, headers={"User-Agent": UA})
r2 = creq.get(URL, timeout=30, headers={"User-Agent": UA}, impersonate="chrome")

print(f"python-requests        : HTTP {r1.status_code}")
print(f"curl_cffi + chrome TLS : HTTP {r2.status_code}")

Actual output from this machine, stable across repeated runs:

python-requests        : HTTP 403
curl_cffi + chrome TLS : HTTP 200
robots.txt fetched     : 5571 bytes

Nothing about the HTTP request changed. The User-Agent is identical. The URL is a boring robots.txt. The 403 verdict is Akamai’s edge reading a TLS Client Hello that no real browser would send and applying a rule that needs no CAPTCHA, no JavaScript, no challenge — the connection is simply denied, 100ms after it opened. Swap the handshake for Chrome’s and the same request sails through. That is the entire business model of the unblocker-API industry: they run browser-grade TLS termination in front of your scraper and charge you per request for it. A $49–$1,975/month ScraperAPI subscription (their published pricing tiers) is, substantially, someone selling you this handshake.

What impersonation fixes — and what it honestly doesn’t

After a week of testing, the honest boundary is this:

Fixed by TLS/HTTP2 impersonation alone: fingerprint-based blocks (the idealo case), rate-limit tiering that treats unknown fingerprints worse than known ones, and “bot score” components driven by JA3/JA4/JA4H signals. This covers a large fraction of the 403s people blame on “my IP is banned” — often it was never the IP, it was the confession the IP made on connect.

Not fixed by impersonation alone: anything requiring JavaScript execution. A great example: nowsecure.nl, the classic bot-detection test site, serves Cloudflare’s non-interactive Turnstile (sitekey 3x00000000000000000000FF — the 3x prefix marks the invisible variant). We fetched it with plain python-requests and got HTTP 200 — but 200 is not a pass. The page contains a Turnstile widget that never runs because there’s no JavaScript engine, so the token the site’s backend checks for is never generated. A client can be served and still fail. curl_cffi cannot run Turnstile; you need a real browser (Playwright with a patched stack, or something like nodriver/patchright) or a solver service. Conflating “got a 200” with “passed the protection” is the single most common self-deception in this field — always assert on the data you came for, never the status code.

The compounding layer: modern scoring also uses inter-request behavior — timing distributions, session continuity, header-order consistency across requests. Impersonation gets you in the door; behavioral discipline (human-ish pacing, consistent session state, no 200-requests-per-second bursts from one identity) keeps you there. Teams that get both right run scraping pipelines against “hard” targets for months; teams that get only the first right burn their IP pool in days.

The engineering takeaway

If you operate scrapers, do three things this week:

  1. Diagnose before you proxy. Next 403, check the response headers first. cf-mitigated: challenge means a JS challenge (impersonation alone won’t help — you need a browser). A bare AkamaiGHost 403 with no challenge marker is very likely fingerprint-based — curl_cffi may fix it in one line. We observed exactly this distinction live: scrapingcourse.com’s Cloudflare challenge page returned 403 + cf-mitigated: challenge to both plain and impersonated clients (correct — it demands JS), while idealo returned a bare Akamai 403 that impersonation defeated. Same status code, completely different failures, completely different fixes. Retrying harder is the wrong response to both.

  2. Version-pin and monitor your fingerprints. Add a tls.peet.ws check to your scraper’s CI: assert your client’s JA4 segment-a matches a real browser prefix and the cipher hash matches the current release of your impersonation library. When curl_cffi ships a new Chrome target, your CI tells you before your customers do.

  3. Know what layer rejected you. The stack, from outermost in: IP reputation → TLS fingerprint (JA3/JA4) → HTTP/2 fingerprint → HTTP header consistency → JS challenges → behavioral analysis. Each layer has a different tool and a different cost. Most people throw proxies (layer 1) at a layer-2 problem, then a browser (layer 5) at a layer-3 problem, and wonder why the bill triples.

There is a money angle here, and it cuts both ways. The unblocker-API market exists because most teams never learn this layer of the stack — convenience pricing of four figures a month stands on that ignorance. On the other side, “we build and maintain scrapers that survive Akamai-class fingerprinting” is a line that closes contracts, because the buyer has usually already paid two freelancers whose scrapers died in week two. The skill is verifiable in five minutes: run the diagnostic above against the prospect’s blocked target, explain which layer is rejecting them, and quote accordingly.

If you’ve got a data source that keeps 403-ing your pipeline, or you want an audit of which anti-bot layer is actually blocking you — get in touch or email [email protected]. This is most of what we do.