Skip to content

Decode trace format 4 and pick the debug file by the executable's debug id - #31

Open
robobun wants to merge 2 commits into
oven-sh:mainfrom
robobun:trace-v4-debug-id
Open

Decode trace format 4 and pick the debug file by the executable's debug id#31
robobun wants to merge 2 commits into
oven-sh:mainfrom
robobun:trace-v4-debug-id

Conversation

@robobun

@robobun robobun commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Companion to oven-sh/bun#38838, which has to wait for this to be deployed: from that change on bun emits trace format '4', which this server currently rejects.

Problem

  • A trace names its build by platform character plus 7-character sha, and debug-store.ts turns that into exactly one artifact. A commit is published as more than one build per platform character: on Linux the glibc, musl and android builds all report 'l'/'L' (bun's Platform::CURRENT only distinguishes os and arch), and this server downloads bun-linux-<arch>-profile.zip, the glibc one, for all of them. Every musl or android trace is therefore remapped against a different binary's symbols today, and the result looks like a normal remap.
  • The same thing with a separately built -baseline binary is what the Windows x64 reports a downstream build of commit 8bb8d04c4 sends look like (BUN-4B1Q, BUN-4BST, BUN-4B64, BUN-4BW6, BUN-4B2Y, about 60 events a day): symbolized against the other link's PDB they read as date/time-zone crashes; against the right one they are a GC marking-thread crash and an Intl.Segmenter crash (evidence in crash_handler(windows): unwind through LLInt and vmEntryToJavaScript frames and stop the walk at non-code PCs bun#38789).

Fix

  • Format '4' (lib/parser.ts): after the sha, a VLQ field count followed by fields of (VLQ tag, VLQ char count, chars). Tag 0 is the build flags VLQ (bit 0 = canary, replacing the '1'/'2' split), tag 1 the executable's debug id (PDB GUID / GNU build-id / LC_UUID) as lowercase hex in the byte order the platform's tools print it. Unknown tags are skipped by their length, so bun can add fields (registers, whatever comes next) without another version character and without this server having to deploy first; malformed counts, lengths, ids and flags fail the parse rather than being read into the fields after them. Exposed as Parse.debug_id.
  • backend/debug-id.ts reads the same id out of the bun-profile executable in a profile zip, lists the builds a commit is published as for a platform character (publishedLinks: plain, then musl and android on Linux, then the -baseline name on x64), and holds the selection policy (selectDebugFile): the plain build if it carries the id, otherwise the first of the others that does; the plain build flagged unverified if its own id cannot be read; flagged mismatch if nothing carries it. Traces without an id (formats 1 to 3) take exactly the old path.
  • backend/debug-store.ts / db.ts download by build name, record each artifact's id when it is downloaded (nullable debug_id column added to debug_file in place), and keep the plain build's cache rows and directories under their existing names so the current cache stays valid (cacheName).
  • backend/remap.ts: on mismatch the symbolizer is not run and the addresses stay raw; Sentry still gets the event, tagged. The build that matched is reported as Remap.variant (musl / android / baseline), which becomes Sentry's dist and a variant tag (backend/sentry.ts), is printed in the issue markdown, and is returned by /remap for the frontend footer; Remap.arch stays the trace's arch. debug_id and debug_file are carried on the remap and as tags, and the markdown and frontend say so when a trace matched nothing.
  • lib/util.ts: the remap cache key includes the id, so two builds of one commit never share an entry; keys for traces without an id are unchanged.
  • Why this is the right fix: the debug id is the one value the linker writes into both the binary the user ran and the debug file this server downloads, so comparing them is the only way to know a remap is against the right code, and trying the commit's other builds is what turns a musl or baseline trace into a correct remap instead of a wrong one. Leaving a total mismatch unsymbolicated is deliberate: a confidently wrong remap is what this is fixing.

Verification

  • bun test: 206 pass. New: format-4 roundtrips (20-byte and 16-byte ids, no id, flags) and a test that fields with unknown tags are skipped and the fields after the header still decode; rejection of bad counts, lengths, ids and flags; field order independence; cache-key behaviour; readExecutableDebugId on synthetic PE/ELF/Mach-O files (the PE case pinned to a GUID llvm-readobj printed for a real bun-debug.exe) and on the test runner's own binary; publishedLinks for every os/arch; every branch of selectDebugFile, including a musl trace landing on the musl build and the downstream baseline case; cacheName keeping legacy names and never colliding. Parse fixtures: a real format-4 trace from a Linux debug build of the bun PR (its id is the binary's readelf -n build id), and the real Windows x64 CI crash from the bun PR with its header re-encoded in the final layout, which decodes to the same frames, GUID and fault address as before. Snapshots reviewed by hand.
  • bun x tsc --noEmit clean; bun build.ts builds the frontend.
  • The debug_file migration was exercised against a database created with the old table shape: legacy rows read back without an id, new rows with one, and a second open does not re-run the ALTER.
  • bin/ci-remap-server.ts only gains the ability to parse format 4 (it symbolizes against the local binary and skips the store); bun's bun-tracestrings pin will be bumped to this commit once it lands.

Background

  • Debug id: PE executables carry a CodeView (RSDS) record with the GUID of their PDB; ELF executables carry a hash of themselves in a .note.gnu.build-id note; Mach-O executables carry an LC_UUID load command that the dSYM repeats. Debuggers use these to match a binary to its symbols. bun's crash handler now puts its own into the trace, and backend/debug-id.ts reads the same value out of the executable shipped in every profile zip.
  • Builds per commit: upstream publishes, per Linux arch, a glibc, a musl and an android build, plus -baseline copies of the x64 zips (an alias of the same binary upstream since ci: single arm64 debian-13 build host; ThinLTO everywhere; baseline-only x64; rust+link merge; sysroots; WebKit a36c188; rust 2026-07-20 bun#34782, a separate binary in trees that still build one). Only baseline ever had its own platform characters, and those are no longer emitted.

…ug id

oven-sh/bun#38838 makes bun's crash handler emit trace format '4': after the
sha, the build-flags VLQ of format 3 (bit 0 = canary) and then the id the
linker stamped into the executable (PDB GUID, GNU build-id, LC_UUID) as a VLQ
byte count plus lowercase hex.

Until now a trace named its build by platform char plus sha, which is not a
binary: a commit published as both bun-windows-x64 and bun-windows-x64-baseline
reports 'w' from both links, and traces from the second were symbolized with
the first one's PDB, producing plausible-looking nonsense.

- lib/parser.ts: parse '4'; Parse.debug_id.
- backend/debug-id.ts: read the same id out of the bun-profile executable in a
  profile zip (PE debug directory, ELF PT_NOTE, Mach-O LC_UUID), and the
  selection policy: check the trace's own arch artifact, fall back to the
  sibling x64 link that carries the id, flag a total mismatch.
- backend/debug-store.ts, db.ts: record the artifact's id when it is
  downloaded (new nullable debug_file.debug_id column, added in place).
- backend/remap.ts: on a mismatch leave the addresses unsymbolicated instead
  of remapping them against the wrong binary; Remap.arch is the link actually
  used; Remap.debug_id / debug_file carry the outcome.
- backend/sentry.ts: arch/dist/baseline tags follow the link actually used;
  new debug_id and debug_file tags.
- markdown, /remap response and the frontend footer say when a trace matched
  nothing.
- lib/util.ts: the remap cache key includes the id so two links of one
  commit do not share entries; keys of older traces are unchanged.
- tests: v4 roundtrips and field validation, synthetic PE/ELF/Mach-O files
  (the PE one pinned to a GUID llvm-readobj printed for a real bun-debug.exe),
  the selection policy, and two real v4 captures as parse fixtures.
…orm char

The header after the sha is now a VLQ field count followed by (tag, char
count, chars) fields, as bun's encoder writes it: tag 0 is the build flags
VLQ, tag 1 the debug id in hex. Unknown tags are skipped, so bun can add
fields without another version char and without the decoder deploying first.

Selection now iterates every build a commit is published as under the same
platform char: on Linux the glibc, musl and android builds all report 'l'/'L'
and were all symbolized against the glibc binary; x64 additionally tries the
-baseline name for trees that still build one. The build that matched is
reported as Remap.variant (musl / android / baseline) and becomes Sentry's
dist and a tag; Remap.arch stays the trace's arch. Cache rows and dirs for
the plain build keep their existing names.

Fixtures: the Linux capture is a real trace from the final encoder; the
Windows one keeps its real frames with the header re-encoded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant