Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,26 @@ Then re-run tests.
handlers, module-level lifecycle-util statics) must run inside
`TeggScope.run(app._teggScopeBag, ...)`. See the "Multi-App Isolation
(TeggScope)" section in `tegg/CLAUDE.md` for the full rules.
- **V8 startup snapshot dependencies**: the egg-bundler can build a V8 startup
snapshot (`snapshot: true`), where the app boots only to `configWillLoad` at
BUILD time. Any module loaded or instantiated during that boot that creates a
non-serializable native binding — llhttp `HTTPParser` (http/https/undici),
`nghttp2` (http2, and anything built on it), tls `SecureContext`, dns
`ChannelWrap`, a `WebAssembly` instance (undici's llhttp; WASM is disabled under
`--build-snapshot`), fs watchers, native addons, open sockets — makes the
snapshot build FATAL ("global handle not serialized"). Such modules must be kept
EXTERNAL (not inlined) so the prelude stubs them at build and forwards to the
real module via `globalThis.__RUNTIME_REQUIRE` at restore. The framework default
list is `DEFAULT_SNAPSHOT_LAZY_MODULES` in `tools/egg-bundler/src/lib/prelude.ts`
(network builtins + `inspector` + `undici` + `urllib`); apps extend it via
`egg.snapshot.lazyModules` in `package.json`. **When adding a framework
dependency that touches the network/native stack during boot, check whether it
must be added to that list.** A package that only reaches the network stack
_transitively_ is already covered because those builtins are lazy (e.g.
`@modelcontextprotocol/sdk` → `@hono/node-server` → `http2`, `@grpc/grpc-js` →
`http2`); only a package that DIRECTLY creates native/WASM state at module-eval
or boot-time instantiation (like `undici`) needs adding. See the "Snapshot
lazy-external defaults" section in `wiki/packages/egg-bundler.md` for details.

## TypeScript Global Types

Expand Down
18 changes: 15 additions & 3 deletions tools/egg-bundler/src/lib/prelude.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ const debug = debuglog('egg/bundler/snapshot-prelude');
export const SNAPSHOT_PRELUDE_MARKER = '@eggjs/egg-bundler:snapshot-prelude';

/**
* Node built-in modules that produce non-serializable native bindings when loaded
* inside a V8 startup snapshot builder. They are kept as lazy externals: build time
* returns a member-proxy stub; restore time forwards to the real module via
* Modules that produce non-serializable native bindings when loaded inside a V8
* startup snapshot builder. They are kept as lazy externals: build time returns a
* member-proxy stub; restore time forwards to the real module via
* `globalThis.__RUNTIME_REQUIRE`.
*
* - network stack (HTTPParser, nghttp2, SecureContext, ChannelWrap):
Expand All @@ -50,6 +50,16 @@ export const SNAPSHOT_PRELUDE_MARKER = '@eggjs/egg-bundler:snapshot-prelude';
* (readline/repl + http2 nghttp2 native), making the heap unserializable. Keep
* it lazy so the build-time stub is used; the live process gets the real module
* on restore.
* - `undici` / `urllib`: egg's HTTP client stack, built during boot
* (`class HttpClient extends urllib.HttpClient`, and urllib's own
* `class BaseAgent extends undici.Agent`). undici instantiates an llhttp
* `WebAssembly` module (disabled under `--build-snapshot`) + an `HTTPParser`, so
* it must not be evaluated at build. Unlike the builtins above these are npm
* packages that would otherwise be **inlined** into the bundle and evaluated at
* build; listing them here forces them external (see {@link Bundler}) so the
* member-proxy stub is used at build and the real module is required on restore.
* Listing them by default means an app gets a serializable snapshot without
* adding them to `egg.snapshot.lazyModules` itself.
*/
export const DEFAULT_SNAPSHOT_LAZY_MODULES: readonly string[] = [
'http',
Expand All @@ -64,6 +74,8 @@ export const DEFAULT_SNAPSHOT_LAZY_MODULES: readonly string[] = [
'node:dns',
'inspector',
'node:inspector',
'undici',
'urllib',
];

/**
Expand Down
12 changes: 12 additions & 0 deletions tools/egg-bundler/test/snapshot-lazy-external.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,18 @@ describe('snapshot lazy-external', () => {
expect(result).toContain('node:dns');
});

it("lazy-externalizes egg's HTTP client stack (undici + urllib) by default", async () => {
// Egg builds its HttpClient (urllib -> undici) during boot; undici's llhttp
// WebAssembly + HTTPParser cannot be snapshot-serialized. As npm packages they
// would otherwise be inlined, so they must be forced external (this list) to get
// the member-proxy stub at build — without an app listing them itself.
expect(DEFAULT_SNAPSHOT_LAZY_MODULES).toContain('undici');
expect(DEFAULT_SNAPSHOT_LAZY_MODULES).toContain('urllib');
const result = await resolveSnapshotLazyModules(tmp);
expect(result).toContain('undici');
expect(result).toContain('urllib');
});

it('returns the default list when package.json has no egg.snapshot.lazyModules', async () => {
await fs.writeFile(path.join(tmp, 'package.json'), JSON.stringify({ name: 'app', egg: {} }));
expect(await resolveSnapshotLazyModules(tmp)).toEqual([...DEFAULT_SNAPSHOT_LAZY_MODULES]);
Expand Down
137 changes: 137 additions & 0 deletions tools/egg-bundler/test/snapshot-lazy.realbuild.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -232,4 +232,141 @@ describe('snapshot lazy-external — real @utoo/pack build', () => {
expect(probe.BlobType).toBe('function');
expect(probe.blobText).toBe('z'); // real node:buffer Blob
}, 60_000);

it('forced-external npm package: class extends pkg.Base is a stub at build, real base after restore', async () => {
// The scenario this PR's undici/urllib defaults enable: a bundled module extends a
// class exported by a forced-external npm package (egg's
// `class HttpClient extends urllib.HttpClient`). The `extends` clause is evaluated
// at BUILD time against the member-proxy stub; at RESTORE the member-proxy replays
// the access path against the real module, so `super(...)` and inherited methods
// resolve to the real base class. A local `lazy-base` package stands in for urllib
// so the test stays hermetic (ExternalsResolver is mocked, so it is forced external
// only via egg.snapshot.lazyModules).
await fs.writeFile(
path.join(baseDir, 'package.json'),
JSON.stringify({ name: 'snaplazy-extends-app', egg: { snapshot: { lazyModules: ['lazy-base'] } } }),
);
const basePkgDir = path.join(baseDir, 'node_modules', 'lazy-base');
await fs.mkdir(basePkgDir, { recursive: true });
await fs.writeFile(path.join(basePkgDir, 'package.json'), JSON.stringify({ name: 'lazy-base', main: 'index.js' }));
await fs.writeFile(
path.join(basePkgDir, 'index.js'),
[
// Module-eval load counter: lets the runners prove WHEN the real package is
// actually loaded (it must be never at build, and only at restore once the
// member-proxy resolves through the lazy hook — not via native resolution).
'globalThis.__LAZY_BASE_LOADS = (globalThis.__LAZY_BASE_LOADS || 0) + 1;',
'class Base {',
' constructor(opt) { this.opt = opt; }',
" greet() { return 'base#' + this.opt; }",
'}',
'module.exports = { Base };',
'',
].join('\n'),
);

const entryDir = path.join(baseDir, '.egg-bundle', 'entries');
await fs.mkdir(entryDir, { recursive: true });
const entry = path.join(entryDir, 'worker.entry.ts');
await fs.writeFile(
entry,
[
'// @ts-nocheck',
// captured at module-eval; the `extends` link freezes against whatever this is
"const { Base } = require('lazy-base');",
'class Sub extends Base {',
' constructor(opt) { super(opt); this.tag = "sub"; }',
' describe() { return this.tag + ":" + this.greet(); }',
'}',
// defer instantiation so the runner controls the build/restore boundary
'globalThis.__makeSub = (n) => new Sub(n);',
'process.stdout.write("ENTRY_OK");',
'',
].join('\n'),
);
mocks.workerEntry = entry;
mocks.entryDir = entryDir;

const outputDir = path.join(baseDir, 'dist');
await bundle({ baseDir, outputDir, snapshot: true });

const workerPath = path.join(outputDir, 'worker.js');
const worker = await fs.readFile(workerPath, 'utf8');
expect(worker).toContain(SNAPSHOT_PRELUDE_MARKER);
expect(worker).toMatch(/function\s+externalRequire\s*\(/);
await execFileAsync(process.execPath, ['--check', workerPath]);

const basePathLiteral = JSON.stringify(basePkgDir);

// BUILD context: define + construct the subclass with no __RUNTIME_REQUIRE. The base
// is the member-proxy stub, so the instance is not a real lazy-base instance, nothing
// throws, and crucially the real lazy-base module is NEVER loaded (realLoads === 0) —
// proving the bundle used the build-time stub rather than resolving the package.
const buildRunner = path.join(outputDir, 'build-runner.cjs');
await fs.writeFile(
buildRunner,
[
'const probe = { phase: "build" };',
'try {',
' require("./worker.js");',
' globalThis.__makeSub(7);',
' probe.restored = !!globalThis.__RUNTIME_REQUIRE;',
' probe.realLoads = globalThis.__LAZY_BASE_LOADS || 0;',
' probe.threw = false;',
'} catch (e) { probe.threw = true; probe.err = e.message; }',
'process.stdout.write("\\n" + JSON.stringify(probe));',
'',
].join('\n'),
);
const built = await execFileAsync(process.execPath, [buildRunner], { cwd: outputDir });
expect(JSON.parse(built.stdout.split('\n').pop() ?? '')).toMatchObject({
restored: false,
threw: false,
realLoads: 0, // real lazy-base never loaded at build
});

// RESTORE context (cross-phase, one process): require worker.js with __RUNTIME_REQUIRE
// still unset (freezing `extends` against the stub), THEN install it, THEN instantiate.
// The load counters prove the real lazy-base is pulled in ONLY when the member-proxy
// resolves through the `__RUNTIME_REQUIRE` hook (loadedBeforeMakeSub === 0,
// loadedAfterMakeSub === 1) — i.e. via the lazy hook, not native pre-resolution.
//
// `instanceof Base` is intentionally NOT asserted: makeMember's `protoProxy` exposes
// only a `get` trap (no `getPrototypeOf`), so inherited methods forward to the real
// prototype but the snapshot-frozen subclass's prototype chain does not literally
// contain the real `Base.prototype` — identity-by-prototype is a known non-goal of the
// upstream member-proxy.
const restoreRunner = path.join(outputDir, 'restore-runner.cjs');
await fs.writeFile(
restoreRunner,
[
'require("./worker.js");',
'const loadedBeforeRT = globalThis.__LAZY_BASE_LOADS || 0;',
`globalThis.__RUNTIME_REQUIRE = (id) => id === "lazy-base" ? require(${basePathLiteral}) : require(id);`,
'const loadedBeforeMakeSub = globalThis.__LAZY_BASE_LOADS || 0;',
'const inst = globalThis.__makeSub(7);',
'const probe = {',
' restored: true,',
' loadedBeforeRT,',
' loadedBeforeMakeSub,',
' loadedAfterMakeSub: globalThis.__LAZY_BASE_LOADS || 0,',
' tag: inst.tag,',
' greet: inst.greet(),',
' describe: inst.describe(),',
'};',
'process.stdout.write("\\n" + JSON.stringify(probe));',
'',
].join('\n'),
);
const restored = await execFileAsync(process.execPath, [restoreRunner], { cwd: outputDir });
expect(JSON.parse(restored.stdout.split('\n').pop() ?? '')).toEqual({
restored: true,
loadedBeforeRT: 0, // building the worker did not load the real module
loadedBeforeMakeSub: 0, // installing __RUNTIME_REQUIRE does not eagerly load it
loadedAfterMakeSub: 1, // loaded exactly once, when the member-proxy resolved via the hook
tag: 'sub', // subclass constructor ran
greet: 'base#7', // inherited real method + real field (opt=7) — real base constructed
describe: 'sub:base#7', // subclass method invoking the inherited one
});
}, 60_000);
});
9 changes: 9 additions & 0 deletions wiki/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

Dates use the workspace-local Asia/Shanghai calendar date.

## [2026-06-28] package | snapshot bundler lazy-externalizes undici + urllib by default (PR #6011)

- sources touched: `tools/egg-bundler/src/lib/prelude.ts`, `tools/egg-bundler/test/snapshot-lazy-external.test.ts`, `tools/egg-bundler/test/snapshot-lazy.realbuild.test.ts`
- pages updated: `wiki/packages/egg-bundler.md`, `wiki/log.md`
- branch: `feat/snapshot-default-lazy-undici-urllib` (off `next`)
- change: Added `undici` + `urllib` to `DEFAULT_SNAPSHOT_LAZY_MODULES` so an app gets a serializable V8 snapshot without listing them in `egg.snapshot.lazyModules`. Egg builds its HttpClient (urllib → undici) during boot, and undici's llhttp `WebAssembly` + `HTTPParser` cannot be snapshot-serialized. As npm packages they would be inlined; listing them forces them external (`Bundler` adds lazy ids to the externals map) so the prelude member-proxy stub is used at build and the real module is required on restore.
- history note: the PR originally (off the older `next`) shipped a bespoke per-export forwarder in `__makeLazyExt` to make `class HttpClient extends urllib.HttpClient` survive the build→restore boundary. While the PR was open, #6003 landed on `next` and rewrote `__makeLazyExt` into a general **access-path-recording member-proxy** (`makeMember`) that already handles `class X extends pkg.Klass` / `DataTypes.INTEGER(11).UNSIGNED` plus `ownKeys`/`getOwnPropertyDescriptor` via `__EXTERNAL_EXPORTS`. The PR was rebased onto that and **reduced to just the default-list addition** (forwarder dropped as superseded). Note `makeMember`'s `protoProxy` has only a `get` trap (no `getPrototypeOf`), so `instanceof RealBase` on a snapshot-frozen subclass is `false` — methods/fields/super() work, identity-by-prototype does not.
- verification: unit test asserts undici+urllib in the default list; new real `@utoo/pack` build test exercises a **forced-external npm package** `class Sub extends pkg.Base` across the build-stub / restore-real boundary in one process (upstream only realbuild-tested the `node:http` builtin). 28 lazy/realbuild tests green; tsgo + oxlint clean. Pre-existing macOS `ManifestLoader`/`EntryGenerator` tmpdir-symlink failures unrelated.

## [2026-06-27] concept | fix concurrent-import race in multi-app boot (oxc-node PR #5965)

- sources touched: `packages/utils/src/import.ts`
Expand Down
51 changes: 50 additions & 1 deletion wiki/packages/egg-bundler.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@ source_files:
- tools/egg-bundler/src/lib/Bundler.ts
- tools/egg-bundler/src/lib/EntryGenerator.ts
- tools/egg-bundler/src/lib/ExternalsResolver.ts
- tools/egg-bundler/src/lib/prelude.ts
- tools/egg-bin/src/commands/bundle.ts
- tools/egg-bundler/docs/output-structure.md
updated_at: 2026-05-06
updated_at: 2026-06-28
status: active
---

Expand Down Expand Up @@ -66,3 +67,51 @@ CommonJS artifact from an Egg application.
external.
- `BundlerConfig.tegg` is accepted but intentionally not wired into the current
implementation yet.

### Snapshot lazy-external defaults

In `snapshot: true` mode the bundler keeps a set of modules **lazy-external** so a
V8 startup snapshot stays serializable: each is emitted as an `externalRequire`,
left out of the build-time heap (a member-proxy stub from the prelude's
`__makeLazyExt`), and forwarded to the real module at restore via
`globalThis.__RUNTIME_REQUIRE`. The injected hook also lazy-stubs any **non-builtin**
external at build (`!__isBuiltin(id)`); the explicit list mainly exists to (a) cover
builtins the `!isBuiltin` rule skips and (b) **force npm packages external** that
would otherwise be inlined.

- `DEFAULT_SNAPSHOT_LAZY_MODULES` (in `src/lib/prelude.ts`) covers the Node network
stack (`http`/`https`/`http2`/`tls`/`dns`), `inspector`, **and egg's HTTP client
stack `undici` + `urllib`**. Egg builds its `HttpClient` (urllib → undici) during
boot, and undici instantiates an llhttp `WebAssembly` (disabled under
`--build-snapshot`) + `HTTPParser` that cannot be serialized. As npm packages
urllib/undici would be inlined; listing them forces them external (`Bundler` adds
the lazy ids to the externals map) so the member-proxy stub is used at build — an
app gets a serializable snapshot without listing them in `egg.snapshot.lazyModules`.
- The member-proxy records the build-time access path (`get`/`apply`/`construct`) and
replays it against the real module on restore, so `class HttpClient extends
urllib.HttpClient` (and urllib's own `class BaseAgent extends undici.Agent`) keep
working: the `extends` is evaluated against the build stub, then `super(...)` /
inherited methods resolve to the real base class after deserialization.

#### When does a new dependency need adding?

Only a package that **directly** creates non-serializable native/WASM state at
module-eval or boot-time instantiation needs a list entry (like `undici`, which
compiles llhttp WASM). A package that only reaches the network/native stack
**transitively** is already covered, because the underlying builtins are lazy:

- `@modelcontextprotocol/sdk` (a default tegg-controller dep, loaded at boot via
`tegg/plugin/controller` → `MCPControllerRegister`) → its StreamableHTTP
transport `require`s `@hono/node-server`, which does a top-level
`require("http2")` + `class extends globalThis.Request`. `http2` is already lazy
and `globalThis.Request` is stubbed by the prelude, and no MCP transport/server
is instantiated at boot — so the SDK does **not** need a list entry.
- `@grpc/grpc-js`, `ws` — not present in OSS tegg; gRPC would route through the
already-lazy `http2` anyway. No entry needed.

`Inference:` audited 2026-06-28 against `tegg/plugin/controller`, default egg
plugins, and the suspect packages' module-eval closures. The opt-in
`mcp-client`/`mcp-proxy`/`langchain` plugins pull the SDK _client_ transports
(`eventsource`/`cross-spawn`/`pkce-challenge`); if an app enables those and
snapshots, re-assess via `egg.snapshot.lazyModules` — that is an app concern, not
a framework default.
Loading