Skip to content

node:fs: chunk writev/readv at IOV_MAX to match libuv - #33695

Merged
Jarred-Sumner merged 3 commits into
mainfrom
farm/c9d9681c/fs-writev-iovmax
Jul 16, 2026
Merged

node:fs: chunk writev/readv at IOV_MAX to match libuv#33695
Jarred-Sumner merged 3 commits into
mainfrom
farm/c9d9681c/fs-writev-iovmax

Conversation

@robobun

@robobun robobun commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

Every vectored-I/O entry point in node:fs (writevSync / writev / promises.writev / FileHandle.writev / FileHandle.readv / readvSync) throws EINVAL the moment the buffer array crosses IOV_MAX (1024 on Linux and macOS). Node.js handles any count:

import * as fsp from "node:fs/promises";
import * as fs from "node:fs";
for (const n of [1024, 1025, 2000]) {
  const fh = await fsp.open(p, "w+");
  await fh.writev(Array.from({ length: n }, (_, i) => Buffer.from([i & 255])));
  // ...
}
node v26.3.0:  n=1024 writev:1024 readv:1024 | n=1025 writev:1025 readv:1024 | n=2000 writev:2000 readv:1024
bun (before):  n=1024 writev:1024 readv:1024 | n=1025 writev:EINVAL readv:EINVAL | n=2000 writev:EINVAL readv:EINVAL
bun (after):   n=1024 writev:1024 readv:1024 | n=1025 writev:1025 readv:1024 | n=2000 writev:2000 readv:1024

Cause

NodeFS::{writev,pwritev,readv,preadv}_inner pass the full iovec array to a single writev(2) / preadv(2). POSIX kernels reject iovcnt > IOV_MAX with EINVAL.

Node's libuv handles this in uv__fs_write_all (loops IOV_MAX-sized batches, accumulates bytes, returns the partial total on a mid-loop error) and uv__fs_read (caps nbufs at IOV_MAX, single syscall).

Fix

Mirror libuv in the node:fs layer (src/runtime/node/node_fs.rs):

  • writev_inner / pwritev_inner: loop IOV_MAX-sized slices of the iovec array, accumulating bytes_written. An error or short write after the first batch returns the accumulated total; pwritev advances the position by bytes written each batch.
  • readv_inner / preadv_inner: slice the iovec array to at most IOV_MAX entries and issue one syscall.

IOV_MAX is taken from libc::UIO_MAXIOV on Linux and libc::IOV_MAX elsewhere (both 1024 on supported targets). Windows has no kernel iovec limit; the constant is c_uint::MAX there so the loop degenerates to a single call into sys_uv, which already batches.

A small bun_sys::platform_iovec_len helper is added so the chunk-capacity sum compiles on both the libc::iovec (unix) and uv_buf_t (windows) field layouts.

Tests

test/js/node/fs/fs.test.ts gains a writev/readv with more than IOV_MAX buffers block covering all entry points with 2000 one-byte buffers: writevSync, writevSync with position, callback fs.writev, FileHandle.writev, readvSync, readvSync with position, and FileHandle.readv. Each writev case asserts the full byte count and file contents; each readv case asserts bytesRead == 1024 on POSIX (the libuv cap). All seven fail with EINVAL on stock bun and pass with this change.

bun run rust:check-all passes on every target.

Related

#31764 overlaps on the writev side (it batches inside bun_sys::writev/pwritev as part of a larger fs.WriteStream change) but does not cover readv/preadv. This PR places the chunking in the node:fs layer where the libuv semantics belong and leaves the raw bun_sys wrappers as single-syscall primitives.

writevSync / writev / promises.writev / FileHandle.writev / FileHandle.readv
all threw EINVAL when given more than IOV_MAX (1024) buffers because the
whole iovec array was handed to a single writev(2)/readv(2).

Node's libuv (uv__fs_write_all) writes in IOV_MAX-sized batches and loops
until every buffer is written; uv__fs_read caps nbufs at IOV_MAX and issues
one syscall. Mirror that in the node:fs *_inner helpers so readv/preadv are
capped and writev/pwritev loop to completion.
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: fe9796ad-b3fb-4eff-9a6a-2ee0f8e7c5d1

📥 Commits

Reviewing files that changed from the base of the PR and between 023a8f0 and a1c3a52.

📒 Files selected for processing (2)
  • src/runtime/node/node_fs.rs
  • test/js/node/fs/fs.test.ts

Walkthrough

This PR adds an IOV_MAX limit for vectored I/O, clamps vectored reads to that limit, batches vectored writes to that limit, and adds tests covering writev/readv behavior with 2000 buffers.

Changes

IOV_MAX-aware vectored I/O

Layer / File(s) Summary
IOV_MAX constant and platform iovec length helper
src/runtime/node/node_fs.rs, src/sys/lib.rs
Adds a cross-platform IOV_MAX constant and a platform_iovec_len helper for platform-specific iovec length access.
Clamp preadv/readv to IOV_MAX
src/runtime/node/node_fs.rs
preadv_inner and readv_inner now cap the iovec slice to IOV_MAX entries before syscall entry.
Batch pwritev/writev by IOV_MAX
src/runtime/node/node_fs.rs
pwritev_inner and writev_inner process iovecs in IOV_MAX-sized batches, accumulate bytes written, and adjust partial-write/error handling.
Tests for over-limit writev/readv
test/js/node/fs/fs.test.ts
Adds a concurrent suite that exercises over-limit writev and readv paths across sync and async variants.

Sequence Diagram(s)

sequenceDiagram
  participant NodeFS
  participant Syscall
  loop IOV_MAX-sized batches
    NodeFS->>Syscall: pwritev/writev(batch)
    Syscall-->>NodeFS: bytes written or error
    NodeFS->>NodeFS: accumulate bytes_written and advance position
  end
Loading
sequenceDiagram
  participant NodeFS
  participant Syscall
  NodeFS->>NodeFS: cap iovec slice to IOV_MAX
  NodeFS->>Syscall: preadv/readv(clamped iovecs)
  Syscall-->>NodeFS: bytes read
Loading

Related issues: None specified
Related PRs: None specified
Suggested labels: node.js, fs, vectored-io
Suggested reviewers: None specified

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change to chunk node:fs vectored I/O at IOV_MAX.
Description check ✅ Passed The description includes the change, rationale, tests, and verification, even though it uses different headings than the template.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the claude label Jul 7, 2026
@robobun

robobun commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 7:20 PM PT - Jul 7th, 2026

@robobun, your commit a1c3a52 has some failures in Build #70065 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 33695

That installs a local version of the PR into your bun-33695 executable, so you can run:

bun-33695 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No bugs found and the implementation looks correct, but this changes error-handling semantics on a runtime syscall path (mid-loop errors now return partial success) and overlaps with #31764's competing placement of the chunking — worth a maintainer sign-off on where this should live.

Extended reasoning...

Overview

This PR fixes EINVAL from fs.writev/readv (and their sync/promise/FileHandle variants) when passed more than IOV_MAX (1024) buffers, by mirroring libuv: writev/pwritev loop in IOV_MAX-sized batches accumulating bytes, and readv/preadv cap the iovec slice at IOV_MAX for a single syscall. It touches src/runtime/node/node_fs.rs (four *_inner functions plus a new IOV_MAX cfg-gated constant), adds a platform_iovec_len helper in src/sys/lib.rs, and adds seven tests in test/js/node/fs/fs.test.ts.

I verified the type asymmetry between the two write paths is intentional and correct: pwritev_inner iterates PlatformIoVecConst (which has a .len field on both unix and windows), while writev_inner iterates PlatformIoVec (libc::iovec on unix with iov_len, uv_buf_t on windows with len) — hence the new platform_iovec_len helper only on the latter.

Security risks

None. Input is a user-provided array of buffers already validated upstream; the change only slices/loops over an existing iovec array. No new untrusted-length arithmetic — chunk_len is bounded by remaining.len(), and chunk_capacity sums usize lengths that were already accepted before this change.

Level of scrutiny

Medium-high. This is production runtime code on a syscall path used by every node:fs vectored-I/O call, with per-platform #[cfg] gates (Linux UIO_MAXIOV, other unix IOV_MAX, Windows c_uint::MAX). The change is not mechanical: it introduces a loop with new error semantics — an error or Ok(0) after the first successful batch is now swallowed and the accumulated total returned, whereas previously the (single) syscall error propagated directly. That matches libuv's intent, but it's a behavioral change a maintainer should confirm.

Other factors

  • The PR description explicitly flags overlap with #31764, which places the same chunking inside bun_sys::writev/pwritev instead. Which layer owns this is a design call for a maintainer.
  • The short-write handling here breaks rather than advancing iovecs and retrying (which libuv's uv__fs_write_all actually does). That's a strict improvement over the prior single-syscall behavior and probably fine, but it's a deliberate simplification worth a human glance.
  • Test coverage is good: all seven entry points, sync/callback/promise, with-position and without, and a Windows-aware readvCap.
  • No CODEOWNERS entries match the touched files.

@robobun

robobun commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

On the two review points:

Mid-loop error semantics. Returning the accumulated total when a batch after the first fails is libuv's uv__fs_write_all verbatim (if (result <= 0) { if (total == 0) total = result; break; }), so this is Node's existing contract for fs.writev, not a new one. Previously we never reached a second batch at all, so the case simply did not arise.

Layer placement vs #31764. The chunking is placed in node_fs.rs because the looping/capping is a libuv uv_fs_* semantic that only node:fs promises. bun_sys::{writev,pwritev,readv,preadv} stay as single-syscall wrappers so other callers (RuntimeTranspilerCache) keep their current short-write contract. #31764 also does not cover readv/preadv, which this PR does. Happy to move it if maintainers prefer the other layer.

The short-write break is deliberate: libuv advances bufs += nbufs regardless of bytes written, which would skip unwritten bytes if a regular-file writev ever returned short mid-batch. Breaking instead returns an honest bytesWritten for the caller to resume from and is never less correct than libuv.


CI status (build 70065): 281 jobs passed, cargo clippy passed. The only hard failure is one :darwin: 26 aarch64 - test-bun shard that hit buildkite-agent artifact download timed out after 120s before running any tests (same agent failed the same way on build 70024). The [warning] annotations are unrelated known flakes (bun-install.test.ts EBADF, rm.test.ts node_modules timeout, postgres startup, napi, spawn) that all passed on retry. test/js/node/fs/fs.test.ts passed on every lane that ran it.

This is ready for a maintainer; the red is infrastructure, not the diff. The failed darwin shard can be retried individually from the Buildkite UI.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/js/node/fs/fs.test.ts`:
- Around line 1683-1792: The fs I/O cases in the writev/readv IOV_MAX suite are
independent and should be run concurrently. Update the enclosing describe block
around the writev/readv tests to use concurrent execution, since each test uses
its own tempDir, fd, or FileHandle with no shared state. Keep the existing test
bodies and locate the change in the writev/readv with more than IOV_MAX buffers
describe block.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 37884b86-156f-4ced-b112-b11b22df6234

📥 Commits

Reviewing files that changed from the base of the PR and between 3f5d816 and 023a8f0.

📒 Files selected for processing (3)
  • src/runtime/node/node_fs.rs
  • src/sys/lib.rs
  • test/js/node/fs/fs.test.ts

Comment thread test/js/node/fs/fs.test.ts Outdated
@Jarred-Sumner
Jarred-Sumner merged commit e8404c1 into main Jul 16, 2026
77 of 78 checks passed
@Jarred-Sumner
Jarred-Sumner deleted the farm/c9d9681c/fs-writev-iovmax branch July 16, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants