Skip to content

install: honor BUN_CONFIG_MAX_HTTP_REQUESTS - #38744

Open
robobun wants to merge 2 commits into
mainfrom
farm/28e362eb/install-honor-max-http-requests-env
Open

install: honor BUN_CONFIG_MAX_HTTP_REQUESTS#38744
robobun wants to merge 2 commits into
mainfrom
farm/28e362eb/install-honor-max-http-requests-env

Conversation

@robobun

@robobun robobun commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Problem

  • BUN_CONFIG_MAX_HTTP_REQUESTS has no effect on bun install: with it set to 8, a loopback registry still sees 64 requests in flight. Only --network-concurrency works. Same on 1.3.14; the docs list the variable as applying to bun install.
  • PackageManagerOptions::load (src/install/PackageManager/PackageManagerOptions.rs:661) calls async_http::load_env, which stores the variable into MAX_SIMULTANEOUS_REQUESTS. PackageManager::init (src/install/PackageManager.rs, formerly line 2276) then unconditionally stored --network-concurrency or the 64 default over it.
  • With no retry or Retry-After handling for 429s in bun install, this variable is the documented way to get past a rate limiting registry, so the documented mitigation was a no-op.

Fix

  • PackageManager::init now stores the install default (64, or the proxy default) before options.load, and after options.load stores only an explicitly passed --network-concurrency. Precedence is therefore flag, then env var, then default.
  • The fixing line is the if let Some(network_concurrency) guard on the post-options.load store; the values themselves are unchanged (--network-concurrency 0 still clamps to 1, an invalid or zero env value still logs and falls back to the default, the same as under fetch).
  • The bun install docs row for BUN_CONFIG_MAX_HTTP_REQUESTS now states the install default and that --network-concurrency overrides the variable.
  • Test: test/cli/install/bun-install.test.ts, "bun install with ... doesnt go over N concurrent requests". The existing --network-concurrency=5 case became an it.each row (same input and assertions; its 51-dependency package.json is now generated, and the > 20 throw inside the stub was dropped because the max assertion already covers it and a thrown 500 would have triggered retries). Two new rows: BUN_CONFIG_MAX_HTTP_REQUESTS=5, which observes 51 requests in flight on the unfixed build and at most 5 with the fix, and --network-concurrency=2 together with BUN_CONFIG_MAX_HTTP_REQUESTS=50, which pins the precedence.
  • bun bd test test/cli/install/bun-install.test.ts -t "concurrent requests": 3 pass with the fix; the env var row fails with Received: 51 on the unfixed debug build and on release 1.4.0.
  • Full bun-install.test.ts on the fixed build: the only failures are the 14 tests that need public internet (bitbucket/gitlab/vercel URLs) plus should support --registry CLI flag, and all of them fail identically on the unfixed build in this environment.

Background

  • MAX_SIMULTANEOUS_REQUESTS (src/http/AsyncHTTP.rs) is the process-wide cap on in-flight HTTP requests. The HTTP thread reads it on every drain, so whatever value is in it when the first request is scheduled is what applies; bun install starts the HTTP thread later in init, after both stores.
  • async_http::load_env is the shared parser for BUN_CONFIG_MAX_HTTP_REQUESTS; the runtime (bun run, fetch) calls it directly on top of the static's 256 default, and bun install reaches it through options.load. Layering the install default underneath it, instead of on top of it, is what makes the same function serve both.
Probe of the report's matrix against the fixed debug build (80 leaf deps, loopback stub holding each request 30 ms)
default        peak in-flight=64 exit=1
env=8          peak in-flight= 8 exit=1
env=abc        peak in-flight=64 exit=1 error: BUN_CONFIG_MAX_HTTP_REQUESTS value "abc" is not a valid integer between 1 and 65535
env=0          peak in-flight=64 exit=1 warn: BUN_CONFIG_MAX_HTTP_REQUESTS value must be a number between 1 and 65535
flag=8         peak in-flight= 8 exit=1
env=4 flag=8   peak in-flight= 8 exit=1

Before the fix, env=8 and env=4 flag=8 read 64 and 8 respectively; the other rows are unchanged.


no test proof · iteration 0 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/install/bun-install.test.ts

@robobun

robobun commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 1:57 AM PT - Aug 15th, 2026

@robobun, your commit 3a2a9c87179cdcda5588de5b3ce3dbbbb3f04d55 passed in Build #97230! 🎉


🧪   To try this PR locally:

bunx bun-pr 38744

That installs a local version of the PR into your bun-38744 executable, so you can run:

bun-38744 --bun

@robobun

robobun commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Status: reproduced and fixed; ready for review.

  • Reproduced with a loopback registry stub: BUN_CONFIG_MAX_HTTP_REQUESTS=5 bun install on 1.4.0 and on an unfixed debug build of main put all 51 manifest requests in flight at once; --network-concurrency 5 capped at 5.
  • With this branch the env var caps at 5, and --network-concurrency still wins when both are set (test/cli/install/bun-install.test.ts, the "doesnt go over N concurrent requests" rows).
  • Rebased onto main once (diff unchanged) to pick up PipeReader: don't re-deliver streamed bytes after a re-entrant read #38726, which fixed the unrelated test-http-chunk-problem.js failure from the first build.
  • Current build: every Linux and Windows test lane passed (the listed failures all passed on retry or when run alone). The two darwin arm64 test shards have not run: they are queued behind a backlog on the release-tier=previous agents and keep expiring before an agent picks them up, which is unrelated to this change.

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 7 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 06105359-1ba0-4499-8385-79874d4b8e05

📥 Commits

Reviewing files that changed from the base of the PR and between 39fb3c1 and 3a2a9c8.

📒 Files selected for processing (3)
  • docs/runtime/environment-variables.mdx
  • src/install/PackageManager.rs
  • test/cli/install/bun-install.test.ts

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small, well-scoped precedence fix.

What was reviewed:

  • Traced the three MAX_SIMULTANEOUS_REQUESTS stores against async_http::load_env (AsyncHTTP.rs:205-234) — invalid/zero env values still fall through to the pre-stored install default, and the .max(1) clamp on --network-concurrency 0 is preserved.
  • Checked that moving env.has_http_proxy() before options.load() is safe: options.load only reads from the already-populated DotEnvLoader and does not load additional env sources that could add proxy vars.
  • Test refactor: ctx.requested is the per-context counter incremented by the shared dummy registry (dummy.registry.ts:350), env is bunEnv aliased at import, and the it.each rows cover flag-only, env-only, and flag-overrides-env.
Extended reasoning...

Overview

Three files: src/install/PackageManager.rs reorders when MAX_SIMULTANEOUS_REQUESTS is written during PackageManager::init so that BUN_CONFIG_MAX_HTTP_REQUESTS (parsed inside options.loadasync_http::load_env) is no longer clobbered by the install default; test/cli/install/bun-install.test.ts converts the existing --network-concurrency=5 test into an it.each with two additional rows for the env var alone and for flag-overrides-env; and docs/runtime/environment-variables.mdx clarifies the install default (64) and precedence.

Security risks

None. This adjusts a process-wide concurrency cap for outbound HTTP during bun install. No auth, crypto, path handling, or untrusted-input parsing is touched. The env var parser itself (load_env) is unchanged and already bounds-checks to 1..=65535.

Level of scrutiny

Low-to-medium. The Rust change is a mechanical reordering of three .store(.., Ordering::Relaxed) calls on a static atomic during single-threaded init, before the HTTP thread starts. I verified: (1) load_env only stores when the env value parses to a nonzero u16, so an invalid/zero value now falls back to the install default that was pre-stored — matching the PR description's probe matrix; (2) cli.network_concurrency is Option<u16>, so usize::from(n.max(1)) is the same clamp as the removed n.max(1) as usize; (3) env.has_http_proxy() reads http_proxy/HTTP_PROXY/https_proxy/HTTPS_PROXY from the already-loaded DotEnvLoader, and options.load (PackageManagerOptions.rs:403-672) does not mutate those keys, so calling it before options.load returns the same result as before.

Other factors

The test refactor follows REVIEW.md harness conventions: await using proc, concurrent pipe drain via Promise.all, {...env, ...extraEnv} spread over bunEnv, and a combined-object toEqual assertion. The dropped > 20 throw inside the stub is safe to remove because a thrown handler would have returned a 500 and triggered install-side retries, inflating ctx.requested past 51 — the maxConcurrentRequests <= cap assertion already guards the invariant. The 10 ms Bun.sleep is simulated registry latency (needed for requests to overlap so the peak counter is meaningful), not a wait-for-condition; it's commented as such. The PR description shows the env-var row failing with Received: 51 on the unfixed build and passing with the fix, satisfying the fails-for-the-right-reason requirement.

PackageManager::init stored the --network-concurrency value (or the 64
default) into MAX_SIMULTANEOUS_REQUESTS after options.load had already
applied BUN_CONFIG_MAX_HTTP_REQUESTS, so the env var never had an effect
on bun install. Store the install default before options.load and only
the explicit --network-concurrency value after it, so the precedence is
flag, then env var, then default.
@robobun
robobun force-pushed the farm/28e362eb/install-honor-max-http-requests-env branch from 00df52c to 3ebff88 Compare August 15, 2026 03:26
Comment thread src/install/PackageManager.rs Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — reorders the three MAX_SIMULTANEOUS_REQUESTS stores so the env var isn't clobbered by the install default.

What was reviewed:

  • Traced options.loadasync_http::load_env (AsyncHTTP.rs:205–234): only stores when the env var parses to a nonzero u16, so invalid/zero falls through to the install default set just before it.
  • env.has_http_proxy() moved before options.loadenv is passed by shared ref, so the proxy check can't differ between the old and new position; http_thread::init still runs after all three stores.
  • Test refactor: ctx.requested is the harness's per-context counter (dummy.registry.ts:350), the dropped urls assertion was dead, and the > 20 throw would have caused 500 retries — the maxConcurrentRequests <= cap assertion is the real guard.
Extended reasoning...

Overview

The PR fixes BUN_CONFIG_MAX_HTTP_REQUESTS being ignored by bun install. Three files: PackageManager.rs reorders atomic stores in init (default before options.load, CLI flag after, guarded by if let Some); bun-install.test.ts converts the existing --network-concurrency=5 test into an it.each with two new rows (env-only, flag-overrides-env); the docs table row for the env var now states the install default (64) and flag precedence.

Security risks

None. This is init-time ordering of atomic stores to a process-wide concurrency cap. The values themselves are unchanged (same clamp, same default constants), and load_env already validates the env var as a u16 in [1, 65535].

Level of scrutiny

Low-to-medium. The change is a mechanical reordering of three stores with a clear invariant (flag > env > default). I verified: load_env is reached via PackageManagerOptions::load at line 661 and only writes when the env value is valid; env is a &DotEnvLoader throughout so has_http_proxy() returns the same value at the earlier position; the HTTP thread starts at http_thread::init further down, so all three stores land before any request is scheduled. The usize::from(network_concurrency.max(1)) matches the old .max(1) as usize semantics (the type is u16 so usize::from is lossless).

Other factors

The test changes are a strict improvement over the original: the dead urls assertion and unused stdin: "pipe" are gone, pipes are drained concurrently with Promise.all, await using handles cleanup, and ctx.requested reuses the harness counter that other tests in this file already rely on. The Bun.sleep(10) in the stub is simulated registry latency (needed to make requests overlap so peak concurrency is observable) and is commented as such — not a wait-for-condition sleep. The comment-cop bot's feedback about the paragraph-long comment was addressed in 3a2a9c8. The one CI failure (test-http-chunk-problem.js) is unrelated and was picked up via rebase onto #38726.

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up from #38759, which fixes the --network-concurrency help text (48 -> 64) and adds a test for the default in the same file, a little above the --network-concurrency=5 test this PR turns into a table.

The two branches merge cleanly right now. #38759 puts its registry stub in a module-level helper, expectInstallInFlightLimit(ctx, limit), which holds every manifest request open and asserts that exactly limit requests are in flight (with limit + 1 dependencies, so the extra one has to wait for a slot). It already clears BUN_CONFIG_MAX_HTTP_REQUESTS from the child environment. If that lands first, the rows here could become calls to it with args/env options added, instead of the Bun.sleep(10) + toBeLessThanOrEqual(cap) stub: that asserts the exact cap, so the --network-concurrency=2 over BUN_CONFIG_MAX_HTTP_REQUESTS=50 row would also distinguish "2" from "bun used fewer", and it does not depend on the burst overlapping within 10ms on a loaded debug runner. Either way, it would be good to end up with one stub in the file rather than two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants