Skip to content

Bun.write: stream ReadableStream sources instead of stringifying them - #31689

Open
robobun wants to merge 16 commits into
mainfrom
farm/f575fa0a/bun-write-readable-stream
Open

Bun.write: stream ReadableStream sources instead of stringifying them#31689
robobun wants to merge 16 commits into
mainfrom
farm/f575fa0a/bun-write-readable-stream

Conversation

@robobun

@robobun robobun commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Fixes #31681
Fixes #31682
Fixes #37658

Repro

using server = Bun.serve({ port: 0, fetch: () => new Response(Buffer.alloc(256 * 1024, 0x5a)) });
const res = await fetch(server.url);

const n = await Bun.write("/tmp/out.bin", res.body);
console.log(n);                                    // 23
console.log(await Bun.file("/tmp/out.bin").text()); // "[object ReadableStream]"

Passing a raw ReadableStream (a fetch body, a plain new ReadableStream(...), or via Bun.file(path).write(stream)) wrote the literal string "[object ReadableStream]" and resolved successfully — silent data corruption.

Cause

write_file_internal special-cases Response, Request, and Archive sources, then falls through to Blob::get, whose joiner string-coerces unknown objects. A raw ReadableStream matched none of the special cases.

Fix

  • Detect ReadableStream (and async iterable) sources in write_file_internal and pipe them into the destination with pipe_readable_stream_to_blob — the same machinery already used for S3→file writes. Covers file, fd, and S3 destinations. Already-used streams throw ReadableStream has already been used, matching the Response/Request body arms.
  • Resolve with the number of bytes written instead of 0. FileSink.written mixes "buffered" and "flushed" reports and can double-count, so this adds FileSink.received_bytes, counting every accepted chunk exactly once (UTF-8 re-encoded length for string chunks, via simdutf).
  • Truncate the destination before writing, per Bun.write's replace semantics. FileSink's open never applied O_TRUNC (Options.truncate was dead); it is now wired into flags() with a false default so Bun.file(path).writer() keeps its established in-place overwrite behavior, and the pipe path opts in.
  • Mark the internal assignToStream promise handled when it rejects synchronously, so a locked/errored stream rejects the returned promise without also surfacing a spurious unhandled rejection.

Types (Bun.write, BunFile.write, S3 writes) and docs updated to include ReadableStream.

Verification

New tests in test/js/bun/io/bun-write.test.js (Bun.write > ReadableStream source): fetch body (256 KiB), plain/pull-based/direct/empty streams, byte-count accuracy, truncation of longer pre-existing files, Bun.file().write(stream), async iterables, disturbed-stream throw, locked-stream and errored-stream rejections.

  • bun bd test test/js/bun/io/bun-write.test.js → 46 pass (11 new tests fail without the fix)
  • test/js/bun/util/filesink.test.ts (42), test/js/web/streams/streams.test.js (70), spawn stdin stream suites — all pass
  • Bun.file(path).writer() on an existing longer file still overwrites in place (unchanged behavior)

Related issues

Rebase notes

Two main changes landed between reviews that touched the same code:

  • webstreams: rewrite ReadableStream, WritableStream, and TransformStream in C++ (zero JS builtins) #33193 (webstreams C++ rewrite) deleted src/jsc/bindings/webcore/ReadableStream.cpp and replaced the JS-builtins streams with C++ classes. ReadableStream__getStoredError was ported to WebStreamsExports.cpp and now reads JSReadableStream::m_state/m_storedError directly. The separate ReadableStream__hasReader helper was dropped: the rewrite's isReadableStreamLocked() already reports reader, reader-less, and detached-native locks correctly, so the preflight in write_file_internal uses stream.is_locked() directly. The shadowed-locked-getter test is kept (the C++ lock check reads internal state, not the public getter, so it still passes).
  • FileSink: resolve a backpressured write() to its chunk's byte count #33538 (FileSink backpressured write accounting) refactored the write/write_latin1/write_utf16 tails to bytes_accepted(buffered_before, &rc) + to_result(rc, accepted). That accounting (per-call pending-promise credit) and this PR's received_bytes counter (cumulative bytes for Bun.write's resolved value) are independent; both are kept side by side.

After rebase: bun-write 54/0, filesink 46/0, streams 151/0.


no test proof · iteration 11 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/js/bun/io/bun-write.test.js

@robobun
robobun requested a review from alii as a code owner June 2, 2026 02:52
@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Adds ReadableStream, AsyncIterable/AsyncGenerator (and thunk) inputs to Bun.write, BunFile.write, and S3 write APIs; implements stream piping with mkdirp/mode handling, rejects disturbed/locked/errored streams, and returns accurate written byte counts. Tests cover stream shapes, error cases, truncation, and mode/createPath behavior.

Changes

ReadableStream and async iterable support for write operations

Layer / File(s) Summary
Type declarations and documentation
docs/runtime/file-io.mdx, packages/bun-types/bun.d.ts, packages/bun-types/s3.d.ts
Public signatures for Bun.write, BunFile.write, and S3 write methods expanded to include ReadableStream, AsyncIterable<string | ArrayBuffer | ArrayBufferView>, AsyncGenerator, and async generator factory functions alongside existing blob/array/string inputs.
FileSink byte-count tracking
src/runtime/webcore/FileSink.rs
New received_bytes: Cell<u64> field tracks total accepted bytes across write paths. Options::truncate default changed to false. count_received increments the counter for write calls and string modes compute UTF-8 lengths (via simdutf / lossy UTF-16 helper).
ReadableStream stored-error FFI and accessors
src/jsc/bindings/webcore/ReadableStream.cpp, src/runtime/webcore/ReadableStream.rs
Adds ReadableStream__hasReader and ReadableStream__getStoredError FFI functions and Rust accessors has_reader and stored_error to inspect internal reader/errored state before piping.
Stream piping trait and wiring
src/runtime/webcore/Blob.rs (trait & impl)
BlobExt::pipe_readable_stream_to_blob signature extended with mkdirp_if_not_exists and mode. Adds mkdirp_parent_of helper and threads options through S3→file and file-backed piping call sites.
File opening with truncation and mode support
src/runtime/webcore/Blob.rs (do_write paths)
Windows: open flags include truncation and use mode.unwrap_or(WRITE_PERMISSIONS); on ENOENT optionally create parent and retry. Non-Windows: FileSink::start invoked with truncate and mode, with mkdirp retry on ENOENT when enabled.
ReadableStream and async iterable input handling
src/runtime/webcore/Blob.rs (write_file_internal)
Rejects disturbed or locked streams (uses has_reader), rejects streams with stored error, and pipes valid streams to destination via pipe_readable_stream_to_blob instead of stringifying streams.
Actual byte count returns
src/runtime/webcore/Blob.rs, src/runtime/webcore/FileSink.rs (promise resolution paths)
Promise resolution paths and resolve shims now return the actual number of bytes written from FileSink.received_bytes instead of a hardcoded 0.0.
Comprehensive ReadableStream write tests
test/js/bun/io/bun-write.test.js
New tests verify fetch/plain/pull/direct ReadableStreams, async iterables/generators, Bun.file(...).write, truncation, empty streams, disturbed/locked/errored streams, parent-dir creation, createPath:false rejection, and mode option behavior (non-Windows).
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed All requirements from linked issues #31681 and #31682 are addressed: ReadableStream sources are now streamed instead of stringified, byte counts are accurate, destinations are truncated properly, and comprehensive tests validate the fixes.
Out of Scope Changes check ✅ Passed All code changes are scoped to the PR objectives: ReadableStream/async-iterable handling, byte counting, truncation support, and related type/doc updates. No unrelated changes detected.
Title check ✅ Passed The title clearly summarizes the main change: Bun.write now streams ReadableStream sources instead of stringifying them.
Description check ✅ Passed The description explains the cause, fix, scope, related issues, and verification results, although it uses a different heading for verification.

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the claude label Jun 2, 2026
@robobun

robobun commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 2:36 PM PT - Jul 28th, 2026

@robobun, your commit 4d5399c has some failures in Build #84278 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 31689

That installs a local version of the PR into your bun-31689 executable, so you can run:

bun-31689 --bun

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Found 3 issues this PR may fix:

  1. Bun.write(Bun.file(path), s3file) does not truncate a larger existing destination file #31682 - PR fixes O_TRUNC being dead code in FileSink, directly addressing the S3→file destination truncation bug
  2. FileSink does not truncate existing file #25968 - PR wires Options.truncate into flags() so FileSink now properly truncates existing files via O_TRUNC
  3. Enable writing ReadableStreams and AsyncIterables to Bun.write #17459 - PR implements exactly what this enhancement requests: Bun.write(dest, readableStream) and async iterable support

If this is helpful, copy the block below into the PR description to auto-close these issues on merge.

Fixes #31682
Fixes #25968
Fixes #17459

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. Bun.write: stream pending Response/Request bodies to disk #30133 - Also streams pending Response/Request bodies to disk via pipeReadableStreamToBlob with O_TRUNC and byte count fixes; explicitly supersedes fix: Bun.write with new Response(req.body) no longer hangs #28112 and Bun.write: pump the stream when Response/Request body is already a ReadableStream #28988
  2. Bun.write: pump the stream when Response/Request body is already a ReadableStream #28988 - Also pumps ReadableStream in Bun.write's Locked branch via pipeReadableStreamToBlob with byte count fix
  3. fix: Bun.write with new Response(req.body) no longer hangs #28112 - Also fixes Bun.write with ReadableStream-backed Response/Request bodies by piping through pipeReadableStreamToBlob with O_TRUNC

🤖 Generated with Claude Code

Comment thread src/runtime/webcore/Blob.rs
Comment thread src/runtime/webcore/Blob.rs
Comment thread src/runtime/webcore/Blob.rs
Comment thread test/js/bun/io/bun-write.test.js Outdated
Comment thread packages/bun-types/bun.d.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/runtime/file-io.mdx (1)

91-99: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Add async iterable and async generator types to the supported inputs list.

The layer description and PR objectives indicate that Bun.write should accept AsyncIterable, AsyncGenerator, and async generator factory functions in addition to ReadableStream. The documentation currently only mentions ReadableStream (line 98).

📝 Suggested additions to complete the documentation
 - `TypedArray` (`Uint8Array`, et. al.)
 - `Response`
 - `ReadableStream`
+- `AsyncIterable`
+- `AsyncGenerator`
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/runtime/file-io.mdx` around lines 91 - 99, Update the supported-inputs
list for Bun.write to include async iterables and async generators: add
"AsyncIterable" and "AsyncGenerator" (and note factory functions that return
async generators / async iterables) alongside ReadableStream as accepted
second-argument types; ensure the prose around the second argument clarifies
Bun.write accepts AsyncIterable/AsyncGenerator/async generator factory functions
in addition to the existing types and matches the layer/PR objectives for
Bun.write.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/runtime/file-io.mdx`:
- Around line 281-284: The write(...) signature currently lists ReadableStream
but omits supported async input types; update the input union for the write
function signature to also include AsyncIterable and AsyncGenerator forms (e.g.,
AsyncIterable<...> and AsyncGenerator<...>) to match the runtime and docs; when
adding them, match the exact generic constraint used in your TypeScript defs
(confirm whether the project uses Uint8Array, ArrayBufferView, or an
unconstrained generic in packages/bun-types/bun.d.ts) and ensure the symbol is
the same write(destination: string | number | BunFile | URL, input: ...)
declaration so consumers see the full supported async types.

In `@src/runtime/webcore/Blob.rs`:
- Around line 5436-5466: Add a preflight check for an already-errored
ReadableStream like the existing disturbed/locked guards: after obtaining
`stream` from `ReadableStream::from_js` and before calling
`destination_blob.pipe_readable_stream_to_blob`, check
`stream.is_errored(global_this)` || `stream.value.get(global_this,
"errored")?.is_some_and(|v| v.to_boolean())`; if true call
`destination_blob.detach()` and return an error via
`global_this.throw_invalid_arguments(format_args!("ReadableStream has
errored"))`. This mirrors the `stream.is_locked` logic and prevents truncation
when the stream has already errored.
- Around line 4552-4570: The current mkdirp_parent_of(dest_path: &[u8]) -> bool
hides all errors from NodeFS::mkdir_recursive and always yields false on
failure, causing callers (like pipe_readable_stream_to_blob/open code paths) to
report earlier ENOENT instead of the real error (EACCES, ENOTDIR, etc.); change
mkdirp_parent_of to return a Result<(), bun_sys::Result> (or the same
result/error type used by mkdir_if_not_exists), propagate and return the exact
error from node_fs.mkdir_recursive rather than collapsing it to false, and
update callers to handle the Result (mirroring the existing mkdir_if_not_exists
error propagation behavior).

In `@test/js/bun/io/bun-write.test.js`:
- Line 842: The test is asserting an async stream error with a synchronous
matcher; update the assertion to use the promise rejection matcher for
Bun.write. Replace the current expect wrapper around an async function (the one
calling Bun.write with filePath and stream) with a promise-based assertion using
expect(Bun.write(filePath, stream)).rejects.toThrow("stream go boom") so the
asynchronous stream rejection is correctly tested.

---

Outside diff comments:
In `@docs/runtime/file-io.mdx`:
- Around line 91-99: Update the supported-inputs list for Bun.write to include
async iterables and async generators: add "AsyncIterable" and "AsyncGenerator"
(and note factory functions that return async generators / async iterables)
alongside ReadableStream as accepted second-argument types; ensure the prose
around the second argument clarifies Bun.write accepts
AsyncIterable/AsyncGenerator/async generator factory functions in addition to
the existing types and matches the layer/PR objectives for Bun.write.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a3ec7f6e-7dee-4010-b2a8-d43661e7a4a2

📥 Commits

Reviewing files that changed from the base of the PR and between 561eb8f and 2cd624a.

📒 Files selected for processing (6)
  • docs/runtime/file-io.mdx
  • packages/bun-types/bun.d.ts
  • packages/bun-types/s3.d.ts
  • src/runtime/webcore/Blob.rs
  • src/runtime/webcore/FileSink.rs
  • test/js/bun/io/bun-write.test.js

Comment thread docs/runtime/file-io.mdx
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs
Comment thread test/js/bun/io/bun-write.test.js Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside current diff — PR may have been updated during review):

  • 🔴 src/runtime/webcore/Blob.rs:5436-5459 — The locked-stream pre-check added in f0ed335 doesn't cover streams that are already in the errored state — new ReadableStream({ start(c) { c.error(...) } }) has disturbed=false and locked=false, so it passes both guards here, then pipe_readable_stream_to_blob opens the destination with O_TRUNC before assign_to_stream observes the errored state and rejects, irreversibly truncating an existing destination to 0 bytes. This was noted as a remaining gap in the (now-resolved) locked-stream thread; the new "rejects when the stream errors" test uses pull (not start) on a non-pre-existing path so it doesn't observe the truncation. Fix: either inspect the stream's [[state]] before opening, or defer the O_TRUNC open until after assign_to_stream has acquired a reader (which would also harden the locked case).

    Extended reasoning...

    What the bug is

    Commit f0ed335 added pre-checks in write_file_internal (Blob.rs:5437-5458) so a ReadableStream source rejects before pipe_readable_stream_to_blob opens the destination with O_TRUNC. It guards on is_disturbed and locked. But a stream whose start() synchronously calls controller.error(...) is in the WHATWG errored state with [[disturbed]] = false (only read/cancel set that) and locked = false (no reader acquired) — so it passes both guards and enters pipe_readable_stream_to_blob, which opens the destination with O_TRUNC first (POSIX: FileSinkOptions { truncate: true }Options::flags() returns … | O_TRUNC; Windows: bun_sys::open(..., O::TRUNC, ...)) and only then calls JSSink::assign_to_stream, which rejects on the first read(). The destination is already 0 bytes by the time the promise rejects.

    The code path

    1. write_file_internal (Blob.rs:5436): ReadableStream::from_js succeeds; is_disturbedfalse; is_locked and the locked getter → false (an errored stream has no reader). Both guards pass.
    2. pipe_readable_stream_to_blob opens the destination — POSIX: (*sink).start(&stream_start) with truncate: trueFileSink::setupopen_for_writing with O_NONBLOCK|O_CLOEXEC|O_CREAT|O_WRONLY|O_TRUNC; Windows: bun_sys::open(path, WRONLY|CREAT|TRUNC|NONBLOCK, mode). The kernel truncates the existing file to 0 bytes on open.
    3. Only then does JSSink::assign_to_stream run. readStreamIntoSink calls getReader() (allowed on errored streams), and the first reader.read() returns a rejected promise. After drain_microtasks the Rejected arm fires and Bun.write returns a rejected promise — but the file on disk is already empty.

    Why existing code/tests don't catch it

    The two new pre-checks only cover disturbed and locked; per the Streams spec, ReadableStreamDefaultControllerError sets [[state]] = "errored" and [[storedError]] but neither disturbs nor locks the stream. The PR's new "rejects when the stream errors" test errors from pull (the stream is still "readable" at the time of the guard check) and writes to a path inside a fresh tempDir that does not pre-exist, so the truncation has no observable effect there. This gap was explicitly called out in the extended reasoning of the now-resolved locked-stream thread ("The synchronously-errored-stream case is harder to pre-validate without inspecting [[state]]"), but f0ed335 only addressed the locked case before the thread was resolved.

    Step-by-step proof

    await Bun.write("data.bin", Buffer.alloc(1_000_000));            // 1 MB on disk
    const s = new ReadableStream({ start(c) { c.error(new Error("boom")); } });
    // s.locked === false, ReadableStream is not disturbed
    await Bun.write("data.bin", s).catch(e => console.log(e.message)); // "boom"
    console.log((await Bun.file("data.bin").stat()).size);             // 0 — original 1 MB destroyed
    1. s is constructed; start runs synchronously, controller.error(new Error("boom")) transitions [[state]] to "errored". [[disturbed]] stays false, no reader → locked === false.
    2. write_file_internal: is_disturbed → false (guard passes); is_locked → false and stream.locked → false (guard passes).
    3. pipe_readable_stream_to_blob opens data.bin with O_TRUNC → file is now 0 bytes.
    4. assign_to_streamreadStreamIntoSinkgetReader() succeeds → read() rejects with Error("boom") → after drain_microtasks, the Rejected arm returns a rejected promise.
    5. The user's await rejects with "boom", but data.bin is permanently empty.

    Impact

    Irreversible data loss on an existing destination file, triggered by a programmer error in the source argument. Both halves of the destructive ordering (the O_TRUNC wiring and the user-reachable ReadableStream call site) are introduced by this PR, so this is PR-introduced. The trigger (passing a synchronously pre-errored stream) is narrower than the locked-stream case, but it's the same data-loss class that justified fixing the locked case in f0ed335.

    Fix

    Two options:

    • Pre-validate: before entering pipe_readable_stream_to_blob, inspect the stream's [[state]] (e.g. via the internal @state slot or by reading stream.value's state) and reject with the stored error if it's already "errored".
    • Defer the open (more robust): move the O_TRUNC open until after assign_to_stream has successfully acquired a reader. This covers the errored case, hardens the locked case without needing the dual is_locked/locked-getter check, and protects against any future synchronous-failure mode in assignToStream.

    The "rejects when the stream errors" test should also be extended (or a new test added) that errors from start instead of pull, pre-populates the destination, and asserts its contents survive — mirroring the updated "throws on a locked stream without touching the destination" test.

@robobun

robobun commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author

Re the last review's 🔴 on pre-errored streams: that landed in 37aa69a (pushed while the review was running) — write_file_internal preflights via ReadableStream__getStoredError (reads the private $state/$storedError slots) and rejects with the stream's stored error before any open/truncate. The rejects an already-errored stream without touching the destination test errors from start, pre-populates the destination, and asserts its contents survive.

Comment thread src/runtime/webcore/FileSink.rs Outdated
Comment thread src/jsc/bindings/webcore/ReadableStream.cpp Outdated
Comment thread packages/bun-types/bun.d.ts
Comment thread src/runtime/webcore/FileSink.rs
Comment thread src/runtime/webcore/Blob.rs
@robobun

robobun commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator Author

CI status for maintainers, build 84278 (final) at 4d5399ca80: 51 jobs green. The two red darwin lanes (14-x64, 14-aarch64) and every parallel-batch failure across lanes are all classified [flaky] by bun run ci:errors (each test passed when re-run alone; zero [new] entries). None are in files this PR touches; bun-write.test.js, s3.test.ts, filesink.test.ts, and streams.test.js passed on every lane.

Review follow-ups on top of the rebased branch: hermetic mock-S3 test for S3File.write(ReadableStream), swapped hand-rolled surrogate scanner for simdutf's le_with_replacement, #[inline(never)] on mkdirp_parent_of, tightened doc comments. All review threads resolved; claude[bot]'s final pass: no findings, deferring to a maintainer for API-surface sign-off.

See "Rebase notes" in the PR description for the resolutions against #33193 and #33538. Ready for review/merge.

@robobun
robobun force-pushed the farm/f575fa0a/bun-write-readable-stream branch from 15d9e74 to 344ab84 Compare June 5, 2026 11:38
Comment thread src/runtime/webcore/Blob.rs Outdated
@robobun

robobun commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator Author

I root-caused S3File.write(readableStream) writing the literal "[object ReadableStream]" independently and landed on the same diagnosis and the same three fixes: the write_file_internal fallthrough to Blob::get, the hardcoded 0 byte count in pipe_readable_stream_to_blob's file arms, and the rejected assignToStream promise never being marked handled (which flips the exit code to 1 on a fully caught stream error). This PR also fixes the missing O_TRUNC stale-trailing-bytes problem, which I had not caught. Not opening a competing PR.

One gap: the fix covers S3 destinations (it is all in the shared write_file_internal), but the test changes are test/js/bun/io/bun-write.test.js only, so S3File.write(stream) has no coverage. That was the headline symptom reported for this bug, so it is worth a test that fails against the stringify behavior specifically. My branch has one that stands up an in-process mock S3 server and asserts the PUT body is the stream contents rather than "[object ReadableStream]":

main...farm/0b76023f/fix-s3-write-readablestream

The relevant piece is the S3File.write(ReadableStream) describe block at the bottom of test/js/bun/s3/s3.test.ts; it needs no credentials or docker and fails before the write_file_internal change and passes after. Feel free to fold it in, it should apply on top of this branch without conflict since this PR does not touch that file.

The locked-stream preflight consulted the public locked getter, which
user code can shadow with Object.defineProperty, letting a locked
stream reach the O_TRUNC open before the pipe's internal reader
acquisition throws. Add ReadableStream__hasReader, which reads the
$reader private slot the way $isReadableStreamLocked does, and drop
the shadowable fallback.
@robobun
robobun force-pushed the farm/f575fa0a/bun-write-readable-stream branch from c5eae7e to b7846b9 Compare July 28, 2026 19:06
Comment thread src/jsc/bindings/webcore/streams/WebStreamsExports.cpp Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/Blob.rs Outdated
Comment thread src/runtime/webcore/FileSink.rs Outdated
Comment thread src/runtime/webcore/FileSink.rs Outdated
Comment thread src/runtime/webcore/FileSink.rs Outdated
Comment thread src/runtime/webcore/FileSink.rs Outdated
Comment thread src/runtime/webcore/ReadableStream.rs Outdated
@robobun

robobun commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main at 789be97 (523 commits). No conflicts; git merge-tree --write-tree was clean.

Verification after rebase (b7846b9f97):

  • bun bd test test/js/bun/io/bun-write.test.js: all 19 ReadableStream source tests pass; 18/19 fail under USE_SYSTEM_BUN=1 (the one passing case is the createPath: false ENOENT check, which rejects for a different reason on the unfixed build). Four unrelated tests in the same file (copyFileRange on large files, Bun.file -> Response, Response -> Bun.file -> Response -> text, Bun.file(0) survives GC) time out at ~7-8s under debug+ASAN locally against a 5s ceiling; they pass in ~750ms under release and are unchanged by this diff.
  • test/js/bun/util/filesink.test.ts: 50/0
  • test/integration/bun-types/bun-types.test.ts: 14/0 (the lib.es2020.d.ts ENOENT that produced the red types check on build 70919 is fixed on main)

Repro from the report, with this build:

ReadableStream ret=16 content="real-stream-data"
asyncIterable  ret=24 content="real-async-iterable-data"
plainObject    ret=15 content="[object Object]"

The plain-object case is unchanged: ReadableStreamTag__tagged returns -1 for an object with no Symbol.asyncIterator, so it still reaches Blob::get's ToString coercion (the Blob-constructor USVString path). Rejecting non-BlobPart inputs would be a behavior change on top of this fix; leaving it out here so this PR stays the stream/iterable fix only.

Comment thread packages/bun-types/s3.d.ts
Comment thread src/runtime/webcore/FileSink.rs Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline nit, two things were checked and ruled out this pass: (1) the Options::truncate default flip from truefalse is behavior-preserving for Bun.file().writer() because flags() previously discarded the field entirely (let _ = self), so the three ..Default::default() sites in streams.rs get no O_TRUNC before or after; (2) the early-reject paths in write_file_internal (disturbed/locked/errored) not cancelling the stream — they run before any reader is acquired, so there is nothing to cancel.

Extended reasoning...

The one open finding is a nit; the PR itself is a substantial feature addition (new Bun.write/BunFile.write/S3 input types, FileSink open-flags change, new C++ FFI export, cross-platform mkdirp retry) that has been through several review rounds with all prior threads resolved. Not approving given the API-surface and core-runtime scope.

Comment thread src/runtime/webcore/Blob.rs

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new findings this pass; every prior inline comment (locked/errored preflights, createPath/mode threading, sync-direct-stream byte count, simdutf UTF-16 length, S3 coverage, #[inline(never)]) has been addressed. Deferring to a maintainer because this adds user-facing API surface and touches refcounted native I/O across POSIX/Windows arms — worth a human sign-off.

What was reviewed:

  • Options::truncate default flip (true→false) is safe: flags() previously ignored self, so no existing caller changes behavior; only the new pipe path opts into O_TRUNC.
  • received_bytes counting vs #33538's bytes_accepted — independent counters, no double-credit; Done(n) correctly credits 0.
  • ReadableStream__getStoredError reads C++ m_state/m_storedError directly (tamper-proof), and is_locked now goes through isReadableStreamLocked — the shadowed-getter bypass is closed.
  • write_file_with_source_destination's existing pipe_readable_stream_to_blob call site updated with the two new args; no other callers.
Extended reasoning...

Overview

Adds ReadableStream / async-iterable as accepted Bun.write sources by routing them through the existing pipe_readable_stream_to_blob. Touches Blob.rs (~130 lines: stream detection in write_file_internal, mkdirp_parent_of helper, two new params on pipe_readable_stream_to_blob, byte-count reads from FileSink.received_bytes), FileSink.rs (received_bytes counter, Options::truncate wired into flags(), default flipped to false), ReadableStream.rs + WebStreamsExports.cpp (stored_error accessor), five .d.ts signatures, docs, 19 new file-destination tests, and a hermetic mock-S3 test.

Security risks

None identified. The locked/disturbed/errored preflights read internal C++ stream state (not user-overridable getters), so a hostile stream cannot trick the code into truncating a destination it shouldn't. mkdirp_parent_of uses bun_core::dirname on the caller-supplied destination path — same trust model as the sibling mkdir_if_not_exists.

Level of scrutiny

High. This is native runtime code on a hot user-facing path with raw *mut FileSink derefs, intrusive refcount management, and separate Windows/POSIX open arms. It also changes the semantics of a previously-dead Options field and adds public API surface (types + docs). None of that is mechanical.

Other factors

The PR has been through ~14 rounds of inline findings (mine and CodeRabbit/comment-cop), all resolved and all marked so on the timeline. Test coverage is thorough for file destinations (fetch body, plain/pull/direct/sync-direct/empty streams, async iterable + generator fn, disturbed/locked/shadowed-locked/pre-errored preflights, createPath true/false, mode option, truncation) plus a hermetic S3 body assertion. The rebase notes explain the interaction with #33193 (webstreams C++ rewrite) and #33538 (backpressure accounting) cleanly. The remaining reason to defer is scope, not any specific concern: new API + native memory-management changes across platforms is the kind of thing a maintainer should approve.

@robobun

robobun commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Still needed as of main 165dc9f: Bun.write(path, new ReadableStream(...)) on a main debug build still resolves with 23 and writes [object ReadableStream]. With only this PR's bun-write.test.js hunk applied to main, 18 of the 19 ReadableStream source tests fail (the createPath: false ENOENT one passes for unrelated reasons). The merged Bun.write fixes (#36006, #36809) only touch Response/Request sources, so this does not overlap with them or with #32906. The branch currently conflicts with main and needs a rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

2 participants