Build the coverage line table in UTF-16 code units - #38710
Conversation
JSC reports the offsets of executed and unexecuted blocks in code units of the source string it holds, but ByteRangeMapping looked them up in a table of byte offsets built from the UTF-8 form of that string, so line attribution drifted after any non-ASCII text in the module. Today a non-ASCII preserved comment is enough: every line after it is misattributed and functions that never ran are reported as covered. The table now stores each line start in code units (the byte offsets are kept as-is while the text is ASCII), and the per-row column tables the line offset table allocates, which were never used here, are freed instead of being kept for the life of the mapping.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. The encoding math checks out and the test is well-constructed, but since this is a runtime logic change with a struct field replacement and explicit drop_elements() handling, a human sign-off would be worthwhile.
What was reviewed:
- Verified the code-unit re-measurement is correct for both Latin-1 and UTF-16 JSC strings (© = 1 unit / 2 UTF-8 bytes; 🐰 = 2 units / 4 bytes;
element_length_utf8_into_utf16via simdutf agrees). - Checked
find_indexis a plain binary search over sortedu32offsets, so feeding it code-unit offsets instead of byte offsets is sound. - Confirmed no other consumers of the removed
line_offset_tablefield;Box::from(byte_starts)copies beforedrop_elements(), and the borrow ends before the&mutcall. - The comment at
test_command.rs:1598("owns aMultiArrayList") is now stale but the code it justifies is unchanged and still correct.
Extended reasoning...
Overview
This PR fixes coverage line misattribution for source files containing non-ASCII characters. ByteRangeMapping in src/sourcemap_jsc/CodeCoverage.rs previously stored a LineOffsetTable (byte offsets of line starts) and looked up JSC's code-unit block offsets against it — the two units only agree for pure ASCII. The struct now stores line_starts: Box<[u32]> in UTF-16 code units, computed by re-measuring each line with strings::element_length_utf8_into_utf16 when the source is not all-ASCII. The full LineOffsetTable (including per-line columns_for_non_ascii boxes) is now freed immediately via drop_elements() rather than living in the thread-local map. A regression test in test/cli/test/coverage.test.ts runs identical modules with ASCII vs. ©🐰 banners of equal code-unit length and requires identical lcov DA: records.
Security risks
None. This is pure offset arithmetic on already-decoded module source text; no new inputs, no parsing of untrusted data, no allocation sizing driven by external values beyond what already existed.
Level of scrutiny
Medium. This is a real algorithmic change to runtime code, not a mechanical tweak. The reasoning (JSC 8-bit Latin-1 and 16-bit UTF-16 both round-trip through UTF-8 to the same code-unit count JSC reports) is correct but subtle enough that a maintainer should confirm the approach — particularly since the column_position computed in the sourcemapped branch is now in code units rather than the previous mixed-unit subtraction, which is more correct but interacts with the internal-sourcemap column semantics that PR 4 of the stack (#33866) addresses.
Other factors
- The struct field change is fully contained: grep confirms no other reader of
.line_offset_tableonByteRangeMapping; the only external references to the type (test_command.rs, C++ FFI inZigSourceProvider.cpp/InternalModuleRegistry.cpp) go through unchanged accessors. - Memory handling is sound:
Box::from(&[u32])deep-copies beforedrop_elements()runs;drop_elements()setslen = 0so the subsequentMultiArrayListDrop only frees the slab. - The test follows harness conventions (
tempDir,bunEnv, concurrent spawns with drained stderr, content assertion before exit-code assertion) and asserts specificDA:7,0/DA:10,0lines on the ASCII baseline before the equality check, so it cannot pass vacuously. - Minor: the loop variable
byte_offsetandline_start_byte_offsetingenerate_report_from_blocksare now misnomers (they hold code units), andtest_command.rs:1598's comment is stale — neither affects correctness.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| // `ByteRangeMapping` owns its line table and is not `Copy`, so collect | ||
| // mutable borrows into the thread-local map instead — no double-free | ||
| // risk. |
There was a problem hiding this comment.
If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code
| /// Offset of the start of each line, in UTF-16 code units: the unit JSC | ||
| /// reports `BasicBlockRange` offsets in. Equal to the byte offset only | ||
| /// while the source text is pure ASCII. |
There was a problem hiding this comment.
If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code
| // `source_contents` is the UTF-8 form of the string JSC holds | ||
| // (8-bit or 16-bit), and JSC's offsets count that string's code | ||
| // units, so re-measure each line in those. |
There was a problem hiding this comment.
If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code
| // `MultiArrayList`'s own `Drop` frees the slab only; this drops each | ||
| // row's `columns_for_non_ascii` box. |
There was a problem hiding this comment.
If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code
f856ca8 to
785825f
Compare
| // `ByteRangeMapping` owns its line table and is not `Copy`, so collect | ||
| // mutable borrows into the thread-local map instead; there is no |
There was a problem hiding this comment.
If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. The encoding logic checks out (UTF-16 code-unit count of the UTF-8 re-encoding equals JSC's offset for both 8-bit Latin-1 and 16-bit strings), and the test exercises both BMP and astral characters. Because it's part of a 4-PR stack and the correctness argument hinges on JSC string-representation invariants, a human look would still be worthwhile.
What was reviewed:
compute()conversion: verifiedelement_length_utf8_into_utf16yields the right count for Latin-1→UTF-8 (© = 2 bytes → 1 unit) and surrogate pairs (🐰 = 4 bytes → 2 units), matching JSC's index unit in both storage modes.drop_elements()ordering:byte_startsis copied into theBox<[u32]>before the mutable borrow; the slab itself is freed byMultiArrayList::Dropat scope exit, so no leak or UAF.- No other consumers of the removed
line_offset_tablefield;find_indexis agnostic to the offset unit. - The comment-cop bot flags look like false positives — the flagged comments are 2-3 line field/unit documentation, not workaround justifications.
Extended reasoning...
Overview
This PR changes ByteRangeMapping in src/sourcemap_jsc/CodeCoverage.rs to store line-start offsets in UTF-16 code units instead of UTF-8 bytes, fixing coverage line misattribution when source text contains non-ASCII characters. The struct's line_offset_table: MultiArrayList<...> field becomes line_starts: Box<[u32]>; compute() now branches on is_all_ascii and re-measures each line via element_length_utf8_into_utf16 when needed. A comment in test_command.rs is reworded to match the new field, and a new differential test in coverage.test.ts compares lcov records for an ASCII vs. ©🐰 banner of equal code-unit length.
Security risks
None. This is offline coverage-report post-processing over source text the runtime already loaded; no new parsing of untrusted input, no auth/crypto/network paths.
Level of scrutiny
Medium-high. The change is small in line count but rests on a non-obvious encoding invariant: that for both JSC string kinds (8-bit Latin-1 and 16-bit UTF-16), the UTF-16 code-unit length of the UTF-8 re-encoding equals the index unit JSC reports. I verified this holds (Latin-1 chars are all BMP → 1 unit each; surrogate pairs round-trip to 2 units), and the PR description states it clearly, but it is exactly the kind of claim a maintainer familiar with the JSC source-provider path should confirm. The column_position calculation in the sourcemap branch now subtracts two code-unit values instead of the previous mixed byte/code-unit subtraction — strictly more correct, but worth a maintainer glance since sourcemap column semantics are involved. The drop_elements() addition changes when per-row column boxes are freed; borrow ordering is correct (the &[u32] is consumed before the &mut call) and the slab is still freed by the list's own Drop.
Other factors
- Second of a 4-PR stack split from #33866; the description notes the helper is shaped for a follow-up variant test, so a human should confirm the split boundaries are what they want.
- The new test is well-constructed: asserts specific
DA:7,0/DA:10,0lines on the ASCII baseline before comparing, uses an astral character so code-unit ≠ codepoint ≠ byte, and drains stderr/exit concurrently. - comment-cop left five automated "paragraph-long comment" flags; on inspection each is a short (2-3 line) unit-documentation comment rather than a workaround justification, so I read them as bot noise, but they are technically outstanding on the thread.
|
CI for 785825f (#96731): 161 jobs green. The Windows lanes were cancelled because no agent could be created (fleet-wide at the moment), and the x64-asan lane failed on timeouts of |
alii
left a comment
alii
left a comment
There was a problem hiding this comment.
Checked the code-unit invariant for both string shapes JSC can hold here: an 8-bit string with characters above 0x7F (to_utf8 re-encodes each byte to two, element_length_utf8_into_utf16 counts one unit each) and a 16-bit string (astral pairs four bytes / two units, lone surrogates U+FFFD / one unit); both are 1:1 with JSC's offsets. Every other offset use in CodeCoverage.rs now compares against the unit table, including the sourcemap-branch column, which matches what Bun's own maps emit. drop_elements is a real method, frees the per-row column boxes MultiArrayList's Drop leaves behind (a small pre-existing leak), and the later Drop is slab-only. The test fails on 1.3.14 exactly as described (DA:7/DA:10 vs DA:4/5/6, FNH identical). Fine to land from my side.
Nits: the body says the diff is two files but the second commit also touches src/runtime/cli/test_command.rs (a comment); is_all_ascii re-scans text LineOffsetTable::generate already classified (its first-non-ASCII column would do), once per file under --coverage so not worth blocking on; repeatUnits silently needs units % piece.length == 0, both call sites satisfy it, a one-line note would guard the follow-up variants.
|
Thanks. Body now mentions the |
2 participants
Second of the four PR stack split out of #33866, based on #38708 (1: #38708, 2: this, 3: disk sources decoded as UTF-8, 4: #33866). The diff against #38708 is
src/sourcemap_jsc/CodeCoverage.rsandtest/cli/test/coverage.test.ts, plus a one-line comment update insrc/runtime/cli/test_command.rsthat referred to the removed field; nothing here depends on #38708, it is only stacked for ordering.Problem
bun test --coveragemisattributes lines in any file whose module text contains non-ASCII characters. Today a non-ASCII preserved comment (/*! © ... */, the usual license banner) is enough: in the new test's fixture every line after the banner is shifted, the two functions that never run disappear from the lcov record, and unrelated lines are reported as unexecuted (full diff in the test output on main:DA:7,0/DA:10,0becomeDA:4,0/DA:5,0/DA:6,0).ByteRangeMapping(src/sourcemap_jsc/CodeCoverage.rs) receives the UTF-8 form of that string (ByteRangeMapping__generatecallsto_utf8()), builds a table of byte offsets of line starts from it, and looks the code unit offsets up in that table. The two agree only while the text is ASCII.Fix
element_length_utf8_into_utf16. This is correct for both kinds of string JSC can hold (an 8-bit string with characters above 0x7F, which is what the runtime produces for such a comment today, and a 16-bit string), because the code unit count of the UTF-8 re-encoding of either equals the offsets JSC uses.byte_offsetlocals in the report loops and theByteRangeMappingname itself follow JSC's own "byte range" terminology for these offsets and are left as they are; the unit is documented on the field they are compared against.test/cli/test/coverage.test.ts"a non-ASCII preserved comment does not shift coverage lines" runs the same module twice, once with an ASCII banner and once with a©🐰banner of the same length in code units (astral characters make code units differ from both bytes and code points), and requires identical lcov records, after checking the ASCII one reports the two unexecuted functions on lines 7 and 10. Fails on main as described above; passes here along with the rest of the file. The helper is shaped so the regex / tagged template variant, which needs Preserve non-ASCII in regex .source and tagged template .raw #33866, can be added there as a second test.Background
DA:<line>,<count>records are what--coverage-reporter=lcovwrites per line; comparing records of two files that differ only in text is a direct check of line attribution.to_utf8()re-encodes either into UTF-8, which is longer than the string's unit count as soon as there is a character above 0x7F.[review] gate passed · iteration 10 · 3 files touched
fails on main (without fix)
passes on PR (with fix)
diff hotspot
gate history · 1 passed · 0 rejected · iteration 10
evidence per changed file