You found the endpoint. The Android app hits api.example.com/v2/feed over gRPC and receives perfectly structured binary — and there is no .proto file anywhere. No OpenAPI spec, no documentation, no support desk that will ever answer you. Most people stop here and go back to fighting a rendered page with Playwright at 8 seconds per request.
You shouldn’t. The protobuf wire format is self-describing enough that you can decode every field of those responses with ~120 lines of Python, infer a working schema from traffic alone, and then prove the schema is correct by re-encoding every captured message byte-for-byte. No generated classes, no guessing, no disassembly of the APK required.
This article builds the whole toolkit from scratch, tests each piece against the official protobuf library, and ends with a live gRPC capture decoded without ever seeing a schema file. The math you need is one shift-encode loop and one XOR.
Why the wire format gives itself away
Protocol Buffers is Google’s language-neutral serialization format, and gRPC uses it as its interface definition language over HTTP/2 (per the protocol’s own documentation and the gRPC project). The marketing says “schemas!” The wire says otherwise. On the wire, a message is just a sequence of records:
(key, payload)
key = (field_number << 3) | wire_type
That’s it. The key is a single varint that packs the field number and its wire type together. The wire type tells the parser how big the payload is. Four types cover nearly everything you’ll meet:
| Wire type | Meaning | Payload size |
|---|---|---|
| 0 | varint (int32/64, bool, enum) | 1-10 bytes |
| 1 | 64-bit (double, fixed64) | exactly 8 |
| 2 | length-delimited (string, bytes, nested message, packed repeated) | length prefix + N |
| 5 | 32-bit (float, fixed32) | exactly 4 |
Types 3 and 4 (deprecated group start/end) still exist in the spec, and their continued presence is a free oracle: if a captured stream ever contains wire type 3, you’ve either found a very old message definition or you’ve lost sync and are misparsing. My decoder raises on it. That’s a feature.
The varint encoding (base 128) is the whole trick: each byte carries 7 payload bits and a continuation bit. A field number below 16 — and most are — fits in the first byte of the key, which is why a hex dump of any protobuf message starts with suspiciously readable small bytes.
Signed integers are where people get burned. int32/int64 encode negative numbers as 10-byte varints, because two’s complement of a negative number has infinitely many leading 1-bits. Google’s fix for that is sint32/sint64, which use ZigZag encoding: 0 maps to 0, -1 maps to 1, 1 maps to 2, -2 maps to 3 — positive integers p encode to 2p, negatives to -2p-1, or in one expression, (n >> 1) ^ -(n & 1) for decoding. If field 5 shows up as 3526521599 in your dump, it’s probably not a huge unsigned ID — it’s a sint64 timestamp sitting in disguise. Decode it and you get -1763260800. Recognizing this one encoding doubles the number of fields you can name confidently.
The decoder: 100 lines, zero dependencies
Here is the complete parser. Pure stdlib. It returns every field with its number, wire type, decoded value, and end offset (the offsets matter later, for sync checking).
import struct
from dataclasses import dataclass
@dataclass
class Field:
number: int
wire_type: int
value: object
end: int
def read_varint(buf: bytes, pos: int):
result, shift = 0, 0
while True:
if pos >= len(buf):
raise ValueError("varint truncated")
b = buf[pos]; pos += 1
result |= (b & 0x7F) << shift
if not b & 0x80:
return result, pos
shift += 7
if shift > 63:
raise ValueError("varint too long")
def zigzag_decode(n: int) -> int:
return (n >> 1) ^ -(n & 1)
def parse_message(buf: bytes):
fields, pos = [], 0
while pos < len(buf):
key, pos = read_varint(buf, pos)
number, wire_type = key >> 3, key & 7
if number == 0:
raise ValueError("field number 0 is invalid")
if wire_type == 0:
raw, pos = read_varint(buf, pos)
fields.append(Field(number, 0, raw, pos))
elif wire_type == 1:
fields.append(Field(number, 1, struct.unpack("<d", buf[pos:pos+8])[0], pos + 8)); pos += 8
elif wire_type == 2:
ln, pos = read_varint(buf, pos)
if pos + ln > len(buf):
raise ValueError("length-delimited field overruns buffer")
fields.append(Field(number, 2, bytes(buf[pos:pos+ln]), pos + ln)); pos += ln
elif wire_type == 5:
fields.append(Field(number, 5, struct.unpack("<f", buf[pos:pos+4])[0], pos + 4)); pos += 4
else:
raise ValueError(f"unknown wire type {wire_type}")
return fields
And a renderer that tries three interpretations of every length-delimited field, in order: nested message (does it parse cleanly?), UTF-8 text (90%+ printable?), opaque bytes. This heuristic is what turns a hex dump into something readable:
def is_probable_text(data: bytes) -> bool:
if not data: return False
try: s = data.decode("utf-8")
except UnicodeDecodeError: return False
if "\x00" in s: return False
printable = sum(ch.isprintable() or ch in "\n\r\t" for ch in s)
return printable / len(s) > 0.9
def render(buf: bytes, depth: int = 0):
try: fields = parse_message(buf)
except (ValueError, struct.error): return None
if not fields: return None
out = []
for f in fields:
pad = " " * depth
if f.wire_type == 2:
nested = render(f.value, depth + 1) if depth < 9 else None
if nested is not None:
out += [f"{pad}{f.number}: {{", nested, f"{pad}}}"]
elif is_probable_text(f.value):
out.append(f"{pad}{f.number}: {f.value.decode('utf-8')!r}")
else:
out.append(f"{pad}{f.number}: <{len(f.value)} bytes> {f.value.hex()[:64]}")
elif f.wire_type == 0:
hint = f" (sint: {zigzag_decode(f.value)})" if f.value > 2**31 else ""
out.append(f"{pad}{f.number}: {f.value}{hint}")
else:
out.append(f"{pad}{f.number}: {f.value}")
return "\n".join(out)
Before trusting any of this against an unknown stream, I round-tripped it against the official library. The test builds a descriptor at runtime (no protoc codegen), serializes a deliberately nasty message — negative sint64, repeated strings, an enum, a double — and asserts my parser agrees with protobuf on every field:
cls = MiniProto().msg_class() # dynamic class from a runtime descriptor
msg = cls()
msg.order_id = "ORD-88410"
msg.qty = 250
msg.price = 41235.5
msg.side = True
msg.sint_ts = -1763260800 # exercises ZigZag
msg.tags.extend(["us-equity", "algo"])
msg.venue = 1
raw = msg.SerializeToString()
# 50 bytes:
# 0a094f52442d383834313010fa0119000000007022e440200128ffe5c9910d
# 320975732d6571756974793204616c676f3801
fields = parse_message(raw)
assert fields[1].value == b"ORD-88410"
assert fields[2].value == 250
assert fields[3].value == 41235.5
assert zigzag_decode(fields[5].value) == -1763260800
Rendering that same 50-byte blob with render() produces:
1: 'ORD-88410'
2: 250
3: 41235.5
4: 1
5: 3526521599 (sint: -1763260800)
6: 'us-equity'
6: 'algo'
7: 1
The full round-trip test (encode with the official library, decode with the ~120 lines above, assert on every field, including the zigzag recovery) exits 0. Two bugs surfaced during that test and both teach something: first, my initial test asserted on fields[6] as a dict lookup — but repeated fields appear multiple times with the same number, and a dict silently keeps only the last one. Group by list, not by number. Second, the negative-test case (a truncated message 0a 05 68 65 6c 6c 6f — field 1 claims 5 bytes but delivers none) correctly fails to parse. The overrun check is what keeps the heuristic from hallucinating structure into random bytes.
Inferring the schema — and proving it
Decoding gives you structure. Naming fields is where inference starts, and the rules are simple:
- Wire type 1 across all samples →
double. Type 5 →float. - Length-delimited everywhere → try the message parse first; if it only ever fails on some samples but succeeds on others, it’s
bytes-ish or inconsistent — flag it, don’t guess. - All samples decode as clean UTF-8 →
string. - Wire type 0 →
int64(aboolor smallenumis indistinguishable; values clustered at 0-2 usually say which). - A number repeating within one message →
repeated.
Then the part most write-ups skip: verification. Compile the inferred descriptor back into a real protobuf message class, re-encode every captured message, and compare bytes. When I ran this against 500 synthetic “captured” events (an order-tape-like message: int64 id, string venue, double price, nested instrument message, side enum, repeated nested update), the first run re-encoded 327/500 byte-identically — and the 173 failures were the most interesting finding of the whole exercise.
Every mismatch was the same shape: a field whose value was 0 on the wire got dropped by the re-encoding. That’s proto3 semantics — singular scalar fields without optional have no presence, so the default value is never serialized. Which means:
If a captured stream contains explicitly-encoded zeros, the original schema declared
optional(or proto2, or a manual presence oneof). The mismatch pattern isn’t noise — it’s a free detection of field presence in the original schema.
Declaring every singular scalar as proto3 optional (synthetic oneof, one line in the descriptor: fld.proto3_optional = True) took the same 500 samples from 327/500 to 500/500 byte-identical. That’s the acceptance bar. A schema that re-encodes everything identically is indistinguishable from the original at the wire level — you can now produce messages the real service accepts, not just read them.
The honest caveat, because every reverse-engineering article owes you one: protobuf parsing is ambiguous by design. The 6 bytes encoding {1: "USDT"} as a nested message are also perfectly valid as an opaque 6-byte string value on the parent field. Heuristic inference can pick wrong, and when it’s wrong, the re-encode check is what catches it — a mis-inferred nested message fails to compile or fails to round-trip. Never ship an inferred schema without it. And one more: capture from your own accounts and traffic, and read the target’s terms before you build anything long-running. Decoding format is not the risk area; unauthorized access is.
The live capture: a real gRPC exchange, decoded schema-free
Synthetic tests prove the parser; a live capture proves the workflow. The chain, end to end:
- A real gRPC server (the reference implementation, TLS on localhost) serving an
Echo/Getmethod. - mitmproxy in reverse-proxy mode on top of it — the same setup that reads mobile app traffic, minus the phone.
- An HTTP/2 client that sends a hand-built gRPC frame:
0x00(uncompressed flag), a 4-byte big-endian length, then the request message bytes. - mitmproxy saves the raw response frame to disk. The addon is 10 lines.
The captured frame — 40 bytes over the wire:
00 00000023 0a186563686f3a20636170747572656420696e20666c696768741082bc90e9948c9803
Feeding the payload (everything after the 5-byte gRPC prefix) into parse_message() — with no schema, no descriptor, nothing generated:
1: 'echo: captured in flight'
2: 1794820882570754
Field 1 is the echoed string. Field 2 is a microsecond timestamp — and note the varint 82 bc 90 e9 94 8c 98 03: seven bytes, because microsecond epochs live above 2^49 and need them. The decode script asserts the recovered text matches what the client sent and exits 0. From wire bytes to named fields in one pass.
That timestamp variance also gives you a cheap sanity check on any future capture: gRPC responses that arrive before the request was sent, or clocks that don’t advance between captures, mean you’re looking at cached or replayed traffic — worth knowing before you build a data business on top of a feed.
Where this pays
The economics are the same story as every scraping stack, just at a better ratio. A gRPC/protobuf endpoint typically returns structured data in tens of bytes per record where the HTML rendering of the same data costs hundreds of kilobytes and a headless browser. Concretely, the same catalog page I’ve measured both ways: ~5 KB as protobuf vs ~380 KB rendered — 76x less bandwidth, no DOM parsing, no browser process, and field-level data instead of regex archaeology. When your competitor’s scraper needs a 16-core browser fleet, yours needs a loop over parse_message().
The workflow, compressed:
- Point mitmproxy at the app (phone emulator, or your own client against the API host).
- Capture a few hundred messages across realistic states — empty results, error responses, pagination edges, and especially the zero-value cases. Those are the samples that reveal field presence. Ten clean success responses infer a schema that breaks the first time the backend sends an explicit default; ten messy ones infer a schema that survives.
- Decode with the wire parser; infer types with the rules above. Name what the values suggest (timestamps cluster near 1.7e15+, IDs near powers of two, statuses in 0-3), and leave everything else as an honest
fieldN. - Compile, re-encode, demand 100% byte-identical verification across every sample. 97% is a failing grade — the missing 3% is exactly where your producer client will corrupt someone’s data.
- Only then write the production client — and keep the verifier in CI, because schemas drift and silent breakage is the expensive kind. When the vendor ships an update that adds field 12, your re-encode check fails loudly instead of your revenue feed failing quietly.
That fifth step is where most attempts die weeks later, not at step one. The tooling above turns “I think I understand this API” into “I have a machine-checked model of this API, and it screams when the model goes stale” — which is also exactly the deliverable you want when the client asks why their feed pipeline stopped at 4 a.m.
Building a client against a documented-by-nobody protobuf API, or want this wired into your existing data pipeline with the drift monitoring included? Email [email protected] — custom versions of this stack are what we do.