Skip to content

Fix IDBBatchAtomicVFS writes that do not match the stored blocks - #351

Open
lalexdotcom wants to merge 2 commits into
rhashimoto:masterfrom
lalexdotcom:fix/idb-sparse-write
Open

lalexdotcom wants to merge 2 commits into
rhashimoto:masterfrom
lalexdotcom:fix/idb-sparse-write

Conversation

@lalexdotcom

Copy link
Copy Markdown

What happens

jWrite's overwrite branch fetches one block and writes the whole buffer through it:

const block = await blocks.get(range);
block.data.subarray(iOffset + block.offset).set(data);

That assumes a block starts exactly at iOffset and holds at least as much as is being written. Neither is guaranteed, and both fail in practice.

Nothing may start at iOffset. The keys are -offset, so IDBKeyRange.bound([path, -iOffset], [path, Infinity]) yields the block with the greatest offset ≤ iOffset — the block before the one wanted, when nothing starts there. subarray(iOffset + block.offset) then lands past its end. SQLite reaches this by writing a transient file out of order, which it does when a statement is interrupted while materialising a result: measured on such a file, the write at 1843200 found the block at 1839104, a single 4096-byte page missing from an otherwise contiguous run of 2305 blocks.

A block may be shorter than the write. SQLite writes a 512-byte journal header, then a full page at the same offset. The write runs past the end of the block that covers its start.

The symptom depends on the build: offset is out of bounds on Chromium, source array is too long on Firefox, and on the asyncify build the rejection is swallowed and the call never settles. When the failure happens inside the queued IndexedDB operation rather than at the call, the file is left in a state where every later read returns SQLITE_IOERR_READ and zeroes — corruption with nothing raised at the write.

The change

jRead already walks the blocks it finds, and already checks that each one reaches the offset asked for:

if (!block || block.data.byteLength - block.offset <= fileOffset) {  }

jWrite now does the same walk with the same test. It writes into each block that covers part of the range, and stores what no block covers as a new block — which is what the branch directly above it already does for a file being extended. The asymmetry between the two was the defect: the reader knew the blocks could be of any size and arranged with gaps, the writer did not.

Tests

test/vfs_sparse_write.js, wired into test/IDBBatchAtomicVFS.test.js. Both cases go through jWrite and jRead directly, so neither depends on how SQLite happens to order its writes:

  • a gap — two blocks written with a hole between them, then the hole filled. Against master every later read of the file returns SQLITE_IOERR_READ and zeroes.
  • a short block — 512 bytes written, then a full page over them. Against master the write runs past the end of the block.

With the change, the file's 48 tests pass, and the full suite is 2910 passing on Chromium.

Checklist

  • I grant to recipients of this Project distribution a perpetual,
    non-exclusive, royalty-free, irrevocable copyright license to reproduce, prepare
    derivative works of, publicly display, sublicense, and distribute this
    Contribution and such derivative works.
  • I certify that I am legally entitled to grant this license, and that this
    Contribution contains no content requiring a license from any third party.

lalexdotcom and others added 2 commits September 18, 2026 13:13
jWrite's overwrite branch fetched a single block and wrote the whole
buffer through it, assuming a block starts exactly at iOffset and holds
at least as much as is being written. Neither holds in general.

Nothing may start at iOffset. The range then yields the next key, which
is the PREVIOUS block in the file, and subarray(iOffset + block.offset)
lands past its end: a RangeError on Chromium, "source array is too
long" on Firefox, and on the asyncify build the rejection is swallowed
and the call never settles. SQLite reaches this by writing a transient
file out of order, which it does when a statement is interrupted while
materialising a result.

A block may also be shorter than the write. SQLite lays down a 512-byte
journal header and later writes a full page at the same offset; the
write then runs past the end of the block that covers its start, with
the same result.

jRead already walks the blocks it finds and checks that each one reaches
the offset asked for. jWrite now does the same: it writes into each
block that covers part of the range, and stores what no block covers as
a new block, which is what the branch above it already does for a file
being extended.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two cases, both through jWrite and jRead directly, so neither depends on
how SQLite happens to order its writes.

A gap: two blocks are written with a hole between them, then the hole is
filled. Against the previous behaviour the fill lands past the end of
the block before it, and every later read of the file returns
SQLITE_IOERR_READ and zeroes - a silent corruption, with no error raised
at the write.

A short block: 512 bytes are written, then a full page over them. The
write runs past the end of the block that covers its start.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant