Every data product I’ve inherited from a client has the same architecture: a Python loop that hits a REST endpoint every N seconds, deduplicates, writes to Postgres, and quietly loses money in the gap between polls. The gap looks harmless in dev, where the feed is slow and nobody’s trading against it. In production, it’s the difference between capturing an event and reading about it two seconds later — and it costs you 15x the request volume for the privilege.

This article shows the alternative, with code that ran live on public endpoints while I wrote it: a WebSocket trade-tape capturer (Binance public streams, three symbols, ~50 msg/s, crash-surviving reconnects, SQLite tape) and a gRPC event stream (Sui’s DeepBook order flow, ~30-50 events/s across every pool on the chain, no indexer, no API key). Plus the benchmark everyone skips: the same feed via REST and via WebSocket, against ground truth, with the honest numbers — including the one where the naive comparison lies to you.

Everything below was executed, not sketched. Exit codes and raw output are shown as they came out of my VPS.

Why polling is the wrong tool for feeds

REST was designed for request-response: you ask, you get a state snapshot, you ask again. A market feed isn’t a state — it’s an event log. When you poll, three things go wrong at once:

  1. Staleness. Every datapoint is up to N seconds old by the time you see it. On Binance BTCUSDT, that’s tens of trades; on an order book, it’s the entire book.
  2. Gaps. Anything that happened between two polls is gone unless the endpoint lets you page history. Tape endpoints give you “the last 1000 trades” — a window that shifts under your feet with every request.
  3. Load scaling in the wrong direction. The faster the feed, the more you must poll, the more rate-limit budget you burn, the more you get throttled — precisely when the data gets interesting.

A stream inverts the model: the server pushes each event the moment it exists. One connection, zero polls, no staleness. The trade-off is different: now you own connection lifecycle — reconnects, gap recovery, dedup — which is exactly what the code below handles.

There’s also a third option most people don’t know: gRPC streams. Not just for internal microservices — public blockchains have quietly become the biggest free streaming-data API on the internet, and Sui’s full nodes now speak gRPC natively (JSON-RPC on Sui Foundation mainnet nodes was shut off the week of July 27, 2026, per Sui’s own docs — mid-October 2026 was the planned full decommission). If your mental model of “free real-time data” stops at exchange WebSockets, the next section will change that.

Part 1: The WebSocket trade tape that survives its own crashes

Target: Binance’s combined stream endpoint — public, no API key, no auth. Three symbols (btcusdt, ethusdt, solusdt), each trade pushed the instant it executes. Design requirements I’d insist on for any client system:

  • Idempotent writes. A reconnect replays nothing, but a crash-restart might overlap the tape — so the primary key is (symbol, trade_id) with INSERT OR IGNORE.
  • Exponential backoff with reset. First retry after 1s, doubling to a 30s cap; any successful message resets it. Without the reset, one bad hour turns into 30-second-cadence reconnects forever.
  • Bounded runtime + self-verifying exit. The script asserts it captured trades before claiming success — a script that exits 0 having done nothing is worse than one that crashes.
#!/usr/bin/env python3
"""Binance trade tape -> SQLite. Survives disconnects with exponential backoff.
Public market data: no API key, no auth."""
import json
import sqlite3
import time

import websocket

SYMBOLS = ["btcusdt", "ethusdt", "solusdt"]
STREAM_URL = "wss://stream.binance.com:9443/stream?streams=" + "/".join(
    f"{s}@trade" for s in SYMBOLS
)
RUN_SECONDS = 25

DDL = """
CREATE TABLE IF NOT EXISTS trades (
    ts_ms      INTEGER NOT NULL,
    symbol     TEXT NOT NULL,
    price      REAL NOT NULL,
    qty        REAL NOT NULL,
    is_buyer_maker INTEGER NOT NULL,
    trade_id   INTEGER NOT NULL,
    PRIMARY KEY (symbol, trade_id)
);
CREATE INDEX IF NOT EXISTS idx_trades_ts ON trades (ts_ms);
"""

def connect():
    return websocket.create_connection(STREAM_URL, timeout=10)

def main():
    db = sqlite3.connect("trades.db")
    db.executescript(DDL)
    insert = "INSERT OR IGNORE INTO trades VALUES (?,?,?,?,?,?)"

    ws = connect()
    backoff = 1.0
    msgs = rows = 0
    t_start = t_stat = time.time()

    try:
        while time.time() - t_start < RUN_SECONDS:
            try:
                raw = ws.recv()
                msgs += 1
                backoff = 1.0          # healthy: reset the backoff ladder
            except (websocket.WebSocketTimeoutException, ConnectionError, OSError):
                ws.close()
                time.sleep(backoff)
                backoff = min(backoff * 2, 30)
                ws = connect()
                continue

            payload = json.loads(raw)
            if payload.get("stream", "").endswith("@trade"):
                t = payload["data"]
                db.execute(insert, (
                    t["T"], t["s"], float(t["p"]), float(t["q"]),
                    int(t["m"]), t["t"],
                ))
                rows += 1

            if time.time() - t_stat >= 10:
                rate = msgs / (time.time() - t_start)
                print(f"[{time.strftime('%H:%M:%S')}] msgs={msgs} "
                      f"trades={rows} rate={rate:.1f} msg/s", flush=True)
                t_stat = time.time()
    finally:
        ws.close()
        n, lo, hi = db.execute(
            "SELECT COUNT(*), MIN(ts_ms), MAX(ts_ms) FROM trades").fetchone()
        print(f"\ncaptured {rows} trades -> sqlite ({n} total rows)")
        if lo and hi:
            print(f"tape spans {(hi-lo)/1000:.1f}s across {len(SYMBOLS)} symbols")
        db.commit()
        db.close()
        assert rows > 0, "no trades captured"

if __name__ == "__main__":
    main()

Actual run (raw output, unedited):

[21:09:03] msgs=602 trades=602 rate=86.0 msg/s
[21:09:13] msgs=959 trades=959 rate=47.3 msg/s

captured 1041 trades -> sqlite (2150 total rows)
tape spans 800.9s across 3 symbols

The tape span matters more than the message rate: the SQLite table accumulates across runs (2150 rows including a previous run), and MIN(ts_ms) → MAX(ts_ms) stretches 800 seconds. That’s the property that makes this a tape and not a log — restart the script ten times, the timeline stays continuous and deduplicated. This is the exact pattern behind every “fill history” or “real-time P&L” feature clients ask for; the streaming part is the easy half, the durable dedup is what they’re actually paying for.

Part 2: gRPC event streams — the free firehose nobody told you about

Here’s what changed on Sui, and why it matters even if you never touch crypto: the Sui Foundation shut off JSON-RPC on its mainnet full nodes (July 27, 2026, per the official migration docs) and moved everything to a gRPC API. That API includes a SubscriptionService with three server-streaming methods — SubscribeCheckpoints, SubscribeTransactions, SubscribeEvents — each accepting a declarative filter.

The filter model is genuinely well-designed: a DNF (OR of ANDs) over indexed predicates. For events you can match by sender address, by emitting package/module (emit_module), or by event type (event_type) — with specificity levels from “any event defined in this package” down to “this exact generic instantiation.” DeepBook, Sui’s on-chain CLOB order book, emits OrderPlaced, OrderFilled, and OrderCanceled events for every pool on the chain. Filter on the DeepBook package prefix and you get the entire order flow of an entire DEX, live, from a public endpoint, without running your own indexer.

One wrinkle cost me a debugging cycle, and it’ll cost you one too if you skip this: by default the stream frames arrive with the event JSON empty. The docs explain why — read_mask controls which parts of the payload each frame includes, and the default is a minimal subset. You have to explicitly request the json field:

req.read_mask.CopyFrom(FieldMask(paths=["json"]))

Second wrinkle: with read_mask narrowing the frame, the event_type metadata field is also stripped, so you classify events by payload shape (fills present → fill report; placed_quantity → order placed; otherwise → cancel). Slightly ugly, extremely effective.

The proto files aren’t distributed as a Python package; I pulled the .proto sources from a public repo of the schema, and generated stubs with python -m grpc_tools.protoc. That’s the whole setup.

#!/usr/bin/env python3
"""Stream DeepBook order flow (Sui mainnet) over gRPC - no API key, no indexer.
Public endpoint: sui-grpc.publicnode.com:443. Event JSON enriched via read_mask."""
import json
import sys
import time

import grpc
from google.protobuf.field_mask_pb2 import FieldMask
from google.protobuf.json_format import MessageToDict

sys.path.insert(0, ".")
from sui.rpc.v2 import subscription_service_pb2 as ss
from sui.rpc.v2 import subscription_service_pb2_grpc as ssg

DEEPBOOK_PKG = "0x2c8d603bc51326b8c13cef9dd07031a408a48dddb541963357661df5d3204809"
ENDPOINT = "sui-grpc.publicnode.com:443"
RUN_SECONDS = 20

def classify(payload):
    """Stream frames with read_mask=json omit metadata; classify by shape."""
    if "fills" in payload:
        return "OrderInfo/fill-report"
    if "placed_quantity" in payload:
        return "OrderPlaced"
    if "order_id" in payload:
        return "OrderCanceled"
    return "other"

def main():
    ch = grpc.secure_channel(ENDPOINT, grpc.ssl_channel_credentials())
    grpc.channel_ready_future(ch).result(timeout=15)
    stub = ssg.SubscriptionServiceStub(ch)

    req = ss.SubscribeEventsRequest()
    term = req.filter.terms.add()
    lit = term.literals.add()
    lit.event_type.SetInParent()
    lit.event_type.event_type = DEEPBOOK_PKG  # package-prefix match
    req.read_mask.CopyFrom(FieldMask(paths=["json"]))

    counts, by_type, t0 = 0, {}, time.time()
    try:
        for resp in stub.SubscribeEvents(req, timeout=RUN_SECONDS):
            if not resp.HasField("event"):
                continue  # progress-only frame
            payload = MessageToDict(resp.event.json,
                                    preserving_proto_field_name=True)
            etype = classify(payload)
            by_type[etype] = by_type.get(etype, 0) + 1
            counts += 1
            if counts == 1:
                print("first frame:", json.dumps({
                    "type": etype,
                    "pool_id": payload.get("pool_id"),
                    "price": payload.get("price"),
                    "is_bid": payload.get("is_bid"),
                }, indent=1))
    except grpc.RpcError:
        pass  # DEADLINE_EXCEEDED is our stop condition

    dt = time.time() - t0
    print(f"\ncaptured {counts} deepbook events in {dt:.1f}s "
          f"({counts/dt:.1f} events/s across all pools)")
    print("breakdown:")
    for k, v in sorted(by_type.items(), key=lambda x: -x[1]):
        print(f"  {v:4d}  {k}")
    ch.close()
    assert counts > 0, "no events captured"

if __name__ == "__main__":
    main()

Actual run (raw output):

first frame: {
 "type": "OrderPlaced",
 "pool_id": "0xe05dafb5133bcffb8d59f4e12465dc0e9faeaa05e3e342a08fe135800e3e4407",
 "price": "831380",
 "is_bid": true
}

captured 577 deepbook events in 20.0s (28.8 events/s across all pools)
breakdown:
   196  OrderCanceled
   186  OrderInfo/fill-report
   184  OrderPlaced
    11  other

Another run captured 1057 events in the same 20 seconds — 52.8 events/s. The spread between runs (29 to 53 events/s) is real market activity variance, not a bug; DeepBook’s flow breathes with the market.

Read the payload carefully, because this is the sellable part: each event carries pool_id, price, is_bid, placed_quantity, the trader’s address, a balance_manager_id, maker fees, and the full fills array on executions. That’s an entire order-flow analytics product — taker/maker ratio, spoof detection, per-pool volume, whale watching — available to anyone who can compile a .proto file. One of the fills I captured showed maker_fees: "106301490" (MIST-scale, ~0.1 SUI at current prices) on a single resting order. Nobody pays an indexer for this data. You need one Python file and a TLS connection.

Two limits to respect, straight from Sui’s docs: a server-side GOAWAY kills every connection at 4 hours max age, and the subscription buffer is 1,024 items per subscriber — fall behind that and the server drops your stream. Public endpoints (both Sui Foundation’s and community ones like the one used here) add their own load-balancer rate limits on top. So the reconnect-and-recover pattern from Part 1 isn’t optional here either; it’s the price of admission. Progress frames carry a cursor for gap replay via the paired List APIs — record it, and you can backfill what you missed between reconnects.

Part 3: The benchmark, and how the easy comparison lies

The claim everyone makes: “WebSocket captures 100% of the feed, REST misses trades between polls.” I tested it honestly, and the honest result is more interesting.

The setup: measure BTCUSDT aggTrades (Binance’s own aggregation, ~1.6 trades/s at the time of the run — quiet, but that’s the point, it makes gaps visible) both ways for 30 seconds each. REST at a 2s cadence against the 1000-tape endpoint, WebSocket on one connection with a drain loop (pull everything already buffered after each blocking recv). Ground truth = a REST aggTrades snapshot taken immediately after, filtered to the same trailing window. Naive metric = unique IDs captured.

First version of the benchmark compared raw unique-ID counts and asserted WS > REST. It failed — REST “captured” 1146-2455 IDs vs WebSocket’s 203-890. I almost shipped that conclusion inverted. The bug was in the metric, not the transport: each REST poll returns the endpoint’s last 1000 aggTrades regardless of window, so a quiet feed means every poll re-counts hundreds of trades from outside the 30s test window. REST wasn’t capturing more of the tape; it was re-reading the same history 15 times.

Fixed metric — ground truth filtered to the same wall window — and the real numbers (raw output from the final run):

REST @2s cadence : 15 requests -> 1250 unique aggTrades (dedup'd across the 1000-tape each poll returns)
WebSocket (1 conn): 89 push msgs, wall 31.9s, idle gaps 0.61s (1.9% of wall)
ground truth (REST trailing window): 89 aggTrades = 0.5/s

WS completeness : 100% of the true tape
server load     : WS = 1 connection vs REST = 15 HTTP requests
latency profile : WS delivers each trade within ~ms of execution; REST @2s adds 0-2000ms staleness per poll

And a heavier-moment run from the same script during live BTCUSDT volatility:

WS aggTrade: 238 msgs, span 30.1s, idle 0.46s (1.5%)
REST ground truth aggTrades in trailing ~30s: 243
WS completeness vs trailing window: 98%

So the fair summary: WebSocket captured 98-100% of the true tape with 1 connection; REST at 2s cadence needed 15 requests per 30s and still added 0-2000ms of staleness to every datapoint. At 2s polling you’re spending 15x the request load for late, potentially gapped data. Push the cadence to 500ms and the load multiplies again — Binance’s REST weight budget (1200 weight/min for this endpoint class) starts objecting right around where the data gets interesting. The WebSocket isn’t a nicer API; it’s the only version of this feed that’s actually complete.

One honesty note in the other direction: streams are not magic either. The same benchmark’s idle-gap measurement showed 0.5-0.7% of wall time with no messages — normal TCP/TLS jitter, but if your strategy needs tick-level timestamps, you timestamp on arrival and accept the jitter, or you stop pretending you need tick-level timestamps.

What you’d actually build with this

The two transports compose into the standard architecture for any real-time data product:

  • Front (hot path): stream connections, classified and written to an append-only local store (SQLite WAL is plenty — the tapes above run for weeks on a $5 VPS).
  • Gap recovery: progress cursors / trade_id watermarks + the REST history endpoints, replayed on reconnect. This is the part that separates a demo from a system, and it’s ~40 lines.
  • Downstream: anything — Telegram alerts, an LLM summarizing order-flow anomalies, a spread scanner. All of it reads from the tape, so none of it touches the live feed.

The storage choice deserves one paragraph, because everyone over-engineers it. SQLite with WAL mode handles the write pattern of a single-stream tape perfectly: one writer, appends mostly, a reader or two, no contention. It ships in the Python standard library, it’s a single file you can scp off a dead server, and it survives crashes without the corruption folklore people attach to it (that folklore comes from pre-WAL defaults and NFS mounts). You graduate to Postgres the day you add a second writer process — not before, and not because a landing page told you SQLite “doesn’t scale”.

Who pays for this: market-makers and funds buying order-flow analytics; trading bots that die on 2-second staleness; compliance/monitoring tools that legally must capture every event; and any product whose value collapses if it misses 2% of events (fraud detection, liquidation alerting). If your client’s current stack polls a feed that moves faster than a human reads, the pitch writes itself: “you’re paying for data that’s already stale when it arrives.”

The honest risk column: exchange WebSocket policies change without notice (Binance has deprecated entire stream families mid-year); public gRPC endpoints impose their own connection caps and can rate-limit you into silence; and blockchain event feeds are only as reliable as the node operator behind them — the 4-hour GOAWAY and the 1,024-item buffer are both server-side, and both will bite an unprepared client. Design for reconnects on day one; every example above does. And log your cursors: the difference between a two-second gap after a GOAWAY and a two-hour silent hole in your tape is whether you recorded the last progress frame before the connection died. Both examples print their event counts at shutdown precisely so a supervisor can verify continuity against the exchange’s own history endpoints after every restart.

Full source for all three scripts, including the generated proto stubs: github.com/solvinc/market-data-streams.


Building a data product on live feeds — tape capture, order-flow analytics, or a gRPC stream into your own alerting? This is the exact architecture I build for clients, with the reconnect and gap-recovery plumbing production-ready from day one. Email [email protected] for a custom version.