Skip to content

Make EString::string() join folded ropes, fixing require.resolve() and other readers of the first segment - #38944

Open
robobun wants to merge 4 commits into
mainfrom
farm/6b2621cb/require-resolve-folded-string
Open

Make EString::string() join folded ropes, fixing require.resolve() and other readers of the first segment#38944
robobun wants to merge 4 commits into
mainfrom
farm/6b2621cb/require-resolve-folded-string

Conversation

@robobun

@robobun robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Problem

  • require.resolve("./fo" + "o") is transpiled to require.resolve("./fo") and throws Cannot find module './fo' at runtime, while require("./fo" + "o") next to it works. Same through Bun.Transpiler (transformSync and scan() report ./fo), bun build, and each branch of require.resolve(x ? "./fo" + "o" : "./ba" + "r"). Node resolves ./foo.
  • Cause: folding "./fo" + "o" produces a rope (see Background) whose data holds only the first segment, and EString::string() (src/ast/e.rs) returned data without walking the rope. transpose_require_resolve_known_string records the import record path through string(); transpose_import and transpose_require happen to flatten first, which is why require() and import() work.
  • Other readers of the same first segment, each reproduced on the unfixed build (repros in the details block):
    • through string(): the --allow-unresolved shape ("./lo" + \cales/${x}.json`was reported as./lo*.json, so a pattern matching the real specifier was rejected) and macro result destructuring (const { ["a" + "b"]: x } = m()dropped theab` property).
    • through data directly: import() attributes (with: { type: "js" + "on" } selected the js loader, and Object::get did not match folded computed keys), feature() from bun:bundle under minify.syntax, the index visitor without minifySyntax (ns[K.X] and exports[K.X] with enum K { X = "fo" + "o" } bound and exported fo instead of foo; E["fo" + "o"] inside require() picked E.fo), the React Fast Refresh signature hash (useState("a" + "b") and useState("a" + "c") got the same signature), and import.meta.hot.accept() matching (not reachable with a rope today, HMR is only on in dev mode where folding is off).
  • Long-standing: the Zig sources read these strings the same way.

Fix

  • EString::string() (and string_cloned()) join the rope into the arena when next is set, through the same helper resolve_rope_if_needed now uses, and eql_bytes compares the whole rope like eql_comptime already did. That alone fixes require.resolve, the shape and the macro case; those call sites are unchanged from main. slice() keeps storing the joined bytes back into the node for callers that read it repeatedly.
  • The readers that need data itself call resolve_rope_if_needed first (import_record_loader, which now takes the arena from its one caller; feature(); the index visitor; hot.accept), and the refresh hasher feeds the segments back to back so a folded string hashes like the equivalent literal.
  • hash(), order() and eql_string() still read data only; they now debug_assert they are not handed a rope. Every caller normalizes first (Data::eql and extract_string_values resolve both sides, the react compiler's JsString asserts the same at construction, package.json sorting never sees ropes), so this only guards future callers.
  • Why this is correct: a rope is a deferred concatenation, so joining it yields exactly the string the program would have built, and the printer already joins every rope when printing, so output is unchanged except where the first segment was wrong. require.resolve now records the same path require records for the same argument, which is what Node resolves.
  • Known rope-unaware reads left alone: the decorated class name taken from an object key (that block is replaced, with a ["a" + "b"] test, in js_parser: name lowered anonymous classes after numeric, non-ASCII and private property keys #38787), the _name helper variables for decorated auto-accessors (cosmetic), and the "str"[i] / charCodeAt(i) folds (they bail past the first segment, so they only miss an optimization). Four different rope bugs found while sweeping are tracked separately (React Compiler template text loss, Template::fold mutating an inlined enum's rope, .length folding counting bytes, and the fold flag staying on after an import()).
  • bundler: bundle template-literal require()/import() via a __glob lookup map #35680 (glob bundling of template specifiers) also walks template ropes when it rewrites extract_dynamic_specifier_shape; that function is untouched here, so there is no conflict, and its per-segment string() calls keep working since string() now handles ropes.
  • Verified with bun bd test; every new test fails on the unfixed build:
    • test/bundler/transpiler/transpiler.test.js: printed output for two- and three-segment concatenations and the ternary form, scan() paths, require() / import() controls, and E["fo" + "o"] inside require() / require.resolve() / import().
    • test/js/bun/resolve/resolve.test.ts: require.resolve of folded specifiers returns the right files at runtime.
    • test/bundler/bundler_allow_unresolved.test.ts: rope head from +, rope head from template folding, rope tail, and the require.resolve() path accept a pattern matching the full specifier.
    • test/bundler/transpiler/macro-test.test.ts: a computed folded key keeps its property when destructuring a macro result.
    • test/bundler/bundler_loader.test.ts: folded type value, and folded computed with / type keys, select the JSON loader.
    • test/bundler/bundler_feature_flag.test.ts: a folded flag name is looked up whole (CLI and API backends).
    • test/bundler/bundler_edgecase.test.ts: ns[K.X] reads the foo export and exports[K.X] exports foo, for a string enum member built with +, in a plain (unminified) bundle.
    • test/bundler/bun-build-api.test.ts: "a" + "b" gets the same refresh signature as "ab" and a different one from "a" + "c".
    • The asserts stayed quiet across test/bundler/transpiler/*, test/bundler/esbuild/{default,ts,dce}.test.ts, bundler_edgecase, bundler_minify, bundler_string, bundler_loader, bun-build-api, the TOML tests and test/cli/install/npmrc.test.ts on the debug build.

Background

  • Rope: when the visit pass folds "a" + "b" it does not copy bytes. It links the right operand onto the left E::String through its next pointer and bumps rope_len; data still holds only "a". Template literals get the same treatment for their head and for the text after each ${}. Ropes are produced in two places (fold_string_addition and Template::fold) and read in many, which is why the fix goes into the accessor.
  • When folding happens: file-wide when minify_syntax or inlining is on (bun run and target: "bun" builds enable both); always, whatever the flags, for enum initializers and for the arguments of require(), require.resolve(), macro calls and import() including its options object. The enum rule is what makes the index visitor cases reachable from an ordinary unminified bun build: a string enum member built with + is a rope everywhere it is inlined.
  • Import record: the entry the parser creates for every static specifier it finds. The bundler resolves it and the printer prints E::RequireResolveString from the record's path, so the recorded text is both what gets resolved and what ends up in the output.
  • --allow-unresolved shape: for a dynamic specifier written as a template literal, the parser joins the literal parts with \0 standing in for each interpolation and matches that against the user's glob patterns.
  • Macro result destructuring: when const { a } = someMacro() inlines the macro's return value, the parser keeps only the properties the pattern names, matching each binding key against the inlined object.
  • Refresh signature: with React Fast Refresh, each component gets a hash of the hooks it calls and their arguments; a changed hash tells the runtime to remount instead of preserving state.
Repros on the unfixed build
$ cat index.js
try { console.log("require.resolve:", require.resolve("./fo" + "o")); } catch (e) { console.log("require.resolve threw:", e.message); }
console.log("require:", require("./fo" + "o"));
$ bun index.js
require.resolve threw: Cannot find module './fo'
require: foo-module
$ bun build --no-bundle index.js | grep require
  console.log("require.resolve:", require.resolve("./fo"));
console.log("require:", require("./foo"));

$ cat shape.js
export function load(x) { return import("./fo" + `o/${x}.js`); }
$ bun build shape.js --allow-unresolved './foo/*'
error: This import() expression will not be bundled because the argument is not a string literal
note: The specifier shape "./fo*.js" does not match any --allow-unresolved pattern. ...

$ cat macro.ts     # m.ts: export function obj() { return { ab: 1, c: 2 }; }
import { obj } from "./m.ts" with { type: "macro" };
const { ["a" + "b"]: x, c } = obj();
console.log(x, c);
$ bun macro.ts
undefined 2

$ cat attr.ts      # hello.notjson contains {"hello":"world"}
const mod = await import("./hello.notjson", { with: { type: "js" + "on" } });
$ bun build attr.ts --target=bun
error: Expected ";" but found ":"
    at hello.notjson:1:9

$ cat k.ts
enum K { X = "fo" + "o" }
exports[K.X] = "value";
$ bun build k.ts --target=bun --format=esm | tail -3
export {
  $fo as fo
};
$ cat a.ts
enum E { fo = "./fo", foo = "./foo" }
export const a = require(E["fo" + "o"]);
$ bun build --no-bundle a.ts | tail -1
export const a = require("./fo" /* fo */);

# feature("ENABLED_" + "FEATURE") with --feature=ENABLED_FEATURE --minify-syntax keeps the else branch.
# Bun.build({ reactFastRefresh: true, minify: { syntax: true } }): useState("a" + "b") and useState("a" + "c")
# both produce _s(C, "mNzOb3qKmvI="), while "ab" and "ac" produce different signatures.
Earlier revisions of this description

The first push flattened at the three string() call sites and added a debug_assert to string(); the second push added the direct data readers after review; the third push moved the fix into string() itself (review pointed out that the invariant lives in EString and that per-call-site flattening is how this bug keeps recurring), reverted the three call sites to their original code, and replaced the contrived index-visitor test with the enum cases above.


no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/bundler/bun-build-api.test.ts test/bundler/bundler_edgecase.test.ts test/js/bun/resolve/resolve.test.ts

…er shapes and macro destructuring

Constant folding turns "./fo" + "o" into a rope E::String whose `data` holds
only the first segment. Three consumers read it through EString::string(),
which does not walk the rope:

- transpose_require_resolve_known_string recorded "./fo" as the import
  record path, so require.resolve("./fo" + "o") was emitted (and resolved)
  as require.resolve("./fo"). require() and import() already flattened.
- extract_dynamic_specifier_shape built --allow-unresolved shapes from rope
  template heads and tails, reporting "./fo*" for "./fo" + `o/${x}` and
  rejecting patterns that match the real specifier.
- Expr::as_string_literal, used to match destructuring keys against an
  inlined macro result, dropped the property for a computed folded key.

Flatten the rope at each site, and debug_assert in EString::string() that it
is not handed an unresolved rope.
@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 12:13 PM PT - Aug 15th, 2026

🔄 @robobun, the build for your commit c9684ba8 (Build #98199) was cancelled — waiting for the next build...

@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 43 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b01b90db-2a76-4f1e-9655-e442a40887c3

📥 Commits

Reviewing files that changed from the base of the PR and between d3f975b and c9684ba.

📒 Files selected for processing (12)
  • src/ast/e.rs
  • src/ast/expr.rs
  • src/js_parser/p.rs
  • src/js_parser/visit/visit_expr.rs
  • test/bundler/bun-build-api.test.ts
  • test/bundler/bundler_allow_unresolved.test.ts
  • test/bundler/bundler_edgecase.test.ts
  • test/bundler/bundler_feature_flag.test.ts
  • test/bundler/bundler_loader.test.ts
  • test/bundler/transpiler/macro-test.test.ts
  • test/bundler/transpiler/transpiler.test.js
  • test/js/bun/resolve/resolve.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Status: fix pushed (head c9684ba), waiting on CI.

Reproduced on bun 1.4.0 and on main with the snippets in the description (require.resolve("./fo" + "o") throws Cannot find module './fo'; bun build --no-bundle prints require.resolve("./fo"); enum K { X = "fo" + "o" }; exports[K.X] = 1 exports fo). The fix lives in EString::string() plus the readers that use data directly; every new test fails with USE_SYSTEM_BUN=1 bun test <file> and passes with bun bd test <file>.

@github-actions

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. bundler: bundle template-literal require()/import() via a __glob lookup map #35680 - Rewrites extract_dynamic_specifier_shape in src/js_parser/p.rs with a new append_estring_rope helper that walks the EString.next rope chain for the template head and every tail, independently fixing the same truncated --allow-unresolved specifier-shape defect.

🤖 Generated with Claude Code

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Not a duplicate of #35680, but the two do touch the same function.

  • bundler: bundle template-literal require()/import() via a __glob lookup map #35680 is a feature (bundling template-literal require()/import() through a glob map). While rewriting extract_dynamic_specifier_shape it also walks template ropes, so it fixes the --allow-unresolved shape case from this PR as a side effect. It does not touch transpose_require_resolve_known_string (the require.resolve("./fo" + "o") bug this PR is for) or as_string_literal (the macro destructuring case), and neither of those is covered by its tests.
  • Overlap is the four-line resolve_rope_if_needed hunk in extract_dynamic_specifier_shape. Whichever PR lands second rebases that hunk; if this one lands first, bundler: bundle template-literal require()/import() via a __glob lookup map #35680's append_estring_rope should read each segment's data (rope segments are always UTF-8) rather than calling string() on it, since string() now debug_asserts that it is not handed a rope node with a next link.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline nit, I checked that as_string_literal's switch from &self.data to self.data is sound (Data::EString holds a Copy StoreRef, so s.slice() flattens the arena node in place via DerefMut — same pattern as the printer). Also confirmed the four extra .data readers the nit names all bypass string(), so the new debug_assert won't fire on them.

Extended reasoning...

The three fixed sites each adopt the exact flatten-first pattern their working siblings (transpose_import, transpose_require) already use, and every fix ships with a test that fails on the unfixed build. The only finding is a nit about pre-existing same-class .data readers the .string() audit didn't cover; it doesn't affect the correctness of what this PR changes. I verified the StoreRef copy in as_string_literal and that the new debug assertion can't be tripped by those extra sites since they read the field directly.

Comment thread src/ast/e.rs Outdated
Same class as the previous commit, found by sweeping direct `.data` reads
rather than string() callers:

- Import::import_record_loader: import() options are folded unconditionally,
  so `with: { type: "js" + "on" }` looked up the "js" loader. Object::get
  goes through eql_bytes, which now compares the whole rope, so folded
  computed keys in the options object match too.
- feature() from "bun:bundle" looked up the first segment of a folded name.
- The e_index visitor passed the first segment to maybe_rewrite_property_access
  when minify_syntax was off but the index had still been folded (inside
  require()/import()/macro arguments), binding ns["fo" + "o"] to `fo`.
- The React Fast Refresh signature hash covered only the first segment.
- import.meta.hot.accept() specifier matching (not reachable with a rope
  today, HMR is only on in dev mode where folding is off).
Comment thread src/ast/e.rs Outdated
Comment thread src/ast/e.rs Outdated
Comment thread src/ast/expr.rs Outdated
Comment thread src/ast/expr.rs Outdated
Comment thread src/js_parser/p.rs Outdated
Comment thread src/js_parser/p.rs Outdated
Comment thread src/js_parser/visit/visit_expr.rs Outdated
Comment thread src/js_parser/visit/visit_expr.rs Outdated
@robobun robobun changed the title js_parser: read the whole folded string in require.resolve(), --allow-unresolved shapes and macro destructuring Read the whole folded string, not its first rope segment, in require.resolve() and other post-visit string readers Aug 15, 2026
Comment thread src/js_parser/visit/visit_expr.rs Outdated
Comment thread src/js_parser/visit/visit_expr.rs
string() and string_cloned() now return the whole folded string, which fixes
require.resolve(), the --allow-unresolved shape and macro destructuring
without touching those call sites; they go back to their original code. The
comparators that still read only `data` (hash, order, eql_string) assert
they are not handed a rope; every caller normalizes first.

Tests now cover the realistic way a rope reaches the index visitor: string
enum members built with "+" are always folded, so ns[K.X] and exports[K.X]
used to pick the member named by the first segment, as did E["fo" + "o"]
inside require()/require.resolve()/import().
Comment thread src/ast/e.rs
Comment thread src/ast/e.rs
@robobun robobun changed the title Read the whole folded string, not its first rope segment, in require.resolve() and other post-visit string readers Make EString::string() join folded ropes, fixing require.resolve() and other readers of the first segment Aug 15, 2026
@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed c9684ba, which restructures the fix following review:

  • EString::string() / string_cloned() now join ropes themselves (shared helper with resolve_rope_if_needed), so transpose_require_resolve_known_string, extract_dynamic_specifier_shape and as_string_literal are back to their original code and the debug_assert in string() is gone. The comparators that still read data only (hash, order, eql_string) assert instead; all their callers normalize first.
  • The index-visitor coverage now uses the realistic reach: string enum members built with + are always folded, so ns[K.X] and exports[K.X] in an unminified bundle, plus E["fo" + "o"] inside require() / require.resolve() / import(), replace the earlier contrived require(ns["fo" + "o"] ...) case.
  • Description rewritten to match; the decorator object-key site stays with js_parser: name lowered anonymous classes after numeric, non-ASCII and private property keys #38787, which replaces that block and tests ["a" + "b"].

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the final revision (c9684ba) and didn't find any bugs. The refactor — making EString::string() join ropes itself instead of asserting and pushing the flatten to every caller — is cleaner than the earlier approach and subsumes the .string()-caller fixes (transpose_require_resolve_known_string, extract_dynamic_specifier_shape, as_string_literal). Given this touches parser/AST hot paths and coordinates with #35680 and #38787, a maintainer sign-off is still worthwhile.

What was reviewed:

  • join_rope extraction and its two callers; string_cloned's new delegation preserves the fresh-copy contract for the non-rope UTF-8 case.
  • eql_bytes now walks the rope via eql8_rope, so Object::get matches folded computed keys (covered by the ["wi" + "th"] loader test).
  • The new debug_asserts on hash/order/eql_string — checked their callers (JSON sorting, alphabetize_*, mangled-prop dedup) all run on parser-produced or JSON-tape strings, never post-fold ropes.
  • write_to_hasher's segment loop feeds bytes back-to-back with the trailing \x00, so "a"+"b" and "ab" hash identically as the test asserts.
Extended reasoning...

Overview

This PR fixes a class of bugs where post-visit readers of E::String saw only the first segment of a folded rope ("./fo" + "o""./fo"). It touches four Rust source files (src/ast/e.rs, src/ast/expr.rs, src/js_parser/p.rs, src/js_parser/visit/visit_expr.rs) and adds tests across eight files. The PR went through three substantive revisions: the initial three-site fix, a sweep of direct .data readers after my first review, and finally c9684ba which replaced the per-caller debug_assert approach with a rope-aware string() accessor plus a shared join_rope helper.

What changed since my last review

c9684ba is a design change: EString::string() now joins ropes into the arena on every call rather than asserting the caller flattened first. This is the right layering per the repo's own review rules ("fix bugs at the layer that owns the violated invariant") — every future .string() caller is now correct by construction. resolve_rope_if_needed and string() share join_rope; slice() remains the caching variant. string_cloned correctly keeps its own copy for the non-rope UTF-8 case (where string() returns a borrow, not a fresh allocation). The direct-.data fixes at import_record_loader, maybe_replace_bundler_feature_call, the e_index visitor, and rewrite_import_meta_hot_accept_string are still needed and remain in place. The new debug_asserts on hash()/order()/eql_string() cover the remaining rope-unaware readers; I traced their callers and none receive folded strings.

Security risks

None. The changes are to compile-time constant folding of string literals in the transpiler; no untrusted-input parsing, no auth/crypto/permissions.

Level of scrutiny

High — this is parser/AST code on the hot path of every transpile and bundle. The fix is well-tested (each of the eight fixed sites has a targeted test that fails on the unfixed build), but the surface area, the cross-PR coordination (#35680 rewrites extract_dynamic_specifier_shape; #38787 replaces the deferred decorator-key reader), and the two open comment-cop nags on the new string() doc comment mean a maintainer should sign off rather than an automated approval.

Other factors

Both of my earlier inline threads have justified responses (the decorator-key reader is owned by #38787 with tests there; the prev_should_fold = true pre-existing bug is filed separately as it changes when ropes are made, not how they're read). Test coverage is thorough and follows harness conventions. The two comment-cop comments at 14:45:40 look like false positives (one is a doc comment, one is pre-existing repositioned code) but are unaddressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant