fix: materialize typed-array vectors on store fetch paths#945
Draft
gorkem2020 wants to merge 4 commits into
Draft
fix: materialize typed-array vectors on store fetch paths#945gorkem2020 wants to merge 4 commits into
gorkem2020 wants to merge 4 commits into
Conversation
gorkem2020
force-pushed
the
fix/typed-array-vectors-on-fetch
branch
from
July 17, 2026 21:37
c9bc86a to
54749f5
Compare
…al LanceDB round-trip fetchForCompaction, bm25Search, and lexicalFallbackSearch each return a typed array (Float32Array/Arrow vector) from LanceDB disguised as number[] via an unchecked cast, or silently collapse it to [] when an Array.isArray guard fails on it. All three tests store a real vector into a temp LanceDB table and fail against current code before the fix.
fetchForCompaction used Array.isArray(row.vector) to decide whether to keep or discard the vector; LanceDB returns vector columns as typed arrays (Float32Array) or Arrow vectors, for which Array.isArray is always false, so every fetched vector silently became []. bm25Search and lexicalFallbackSearch had the same underlying problem in a different shape: a bare 'row.vector as number[]' cast that never actually converts the value, leaving a typed array masquerading as number[] at runtime. consolidate is the first consumer of fetchForCompaction that actually needs the vectors (compaction's own clustering ignored them), which is why this went unnoticed until a live dry-run returned zero clusters. Fixed all three call sites to use the existing toNumberVector helper (src/store.ts:370), which vectorSearch's primary path already relies on: it accepts any ArrayLike (typed arrays, Arrow vectors, plain arrays), converts every element with Number(), and validates the result is all finite before returning it, falling back to [] only for genuinely absent or malformed vectors. getById and the other row-mapping sites in this file already used the equivalent Array.from(row.vector as Iterable<number>) pattern and were unaffected.
gorkem2020
force-pushed
the
fix/typed-array-vectors-on-fetch
branch
from
July 18, 2026 15:53
54749f5 to
9da228c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LanceDB returns vector columns as typed arrays (Float32Array) or Arrow vector objects, not plain JavaScript arrays. Three row-fetch paths in
src/store.tsassumed otherwise:fetchForCompactionusedArray.isArray(row.vector) ? row.vector : []to decide whether to keep the vector. SinceArray.isArrayis always false for a typed array, every fetched vector silently collapsed to an empty array.bm25SearchandlexicalFallbackSearchused a barerow.vector as number[]cast, which does not actually convert anything: the returnedMemoryEntry.vectoris a typed array pretending to benumber[]at the type level, but not a real array at runtime.This went unnoticed because the one existing consumer of
fetchForCompaction, the background compactor, ignores the returned vectors entirely (its own clustering step never reads them). The bug surfaced through a downstream consolidation command that clusters rows by embedding similarity and got zero clusters back from a live dry-run, because every vector it saw had length zero.getByIdand the other row-mapping call sites in this file were already unaffected, since they useArray.from(row.vector as Iterable<number>), which correctly converts a typed array via iteration.Fix
All three affected sites now use the existing
toNumberVectorhelper (already the pattern the primaryvectorSearchpath relies on), which accepts any array-like input (typed arrays, Arrow vectors, plain arrays), converts every element withNumber(), and validates the result is entirely finite before returning it, falling back to[]only for genuinely absent or malformed vectors.Test plan
test/typed-array-vector-fetch.test.mjsstores a real vector into a temporary LanceDB table and asserts each of the three affected paths returns it intact. All three failed against the current code (fetchForCompactionreturned a zero-length vector;bm25SearchandlexicalFallbackSearchreturned a non-Array value) and passed after the fix.npm run build(tsc) clean.npm testchain green (one test skipped: a known host-side port 11434 conflict unrelated to this change).test/typed-array-vector-fetch.test.mjsregistered in bothpackage.json'stestscript andscripts/ci-test-manifest.mjs(storage-and-schema group); manifest verification passes.