feat(sqlite): vector search index - #244
Draft
LeeroyHannigan wants to merge 22 commits into
Draft
Conversation
LeeroyHannigan
changed the title
…kends
Models DynamoDB's vector index surface in core and makes it implementable
by a storage backend, without implementing it for any: both in-tree
backends still refuse, and the refusals are proven over the wire.
The contract
CreateTable and UpdateTable accept VectorIndexes and VectorIndexUpdates,
DescribeTable reports them, and SearchVectors is a first-class operation
with its own consumed-capacity shape (VectorSearchRequestBytes, a byte
count with no units field). Item writes validate the vector attribute and
the search-schema attributes against the index definition.
Vector search is an OPTIONAL capability, expressed as a separate
VectorSearchEngine trait with no default bodies, reached through one
accessor on DataEngine that returns Option<&dyn VectorSearchEngine> and
defaults to None. Declaring support therefore requires handing over an
implementation: a backend cannot claim the capability and omit the method,
which an earlier boolean flag allowed. The accessor deliberately sits on
DataEngine rather than StorageEngine, because StorageEngine comes from a
blanket impl over the six focused traits and a defaulted method there
could never be overridden by a backend.
This is the first optional feature on a trait surface where every other
method is mandatory, so it is also the template: peel the feature into its
own trait with no defaults, add one accessor to a trait backends already
implement, gate in core so the refusal never reaches storage, and return
StorageError::Unsupported, added here, so declining is not reported as an
internal fault.
Capability is not the same as having acted
The capability gate proves a backend CAN serve vector indexes. It cannot
prove the backend acted on a given request, and that gap is not
theoretical: measured against the SQLite backend before it implemented the
UpdateTable path, a Create returned 200 and created nothing, discoverable
only on the first search, and a Delete returned 200 while the index stayed
ACTIVE, stayed in DescribeTable, and kept returning hits. A backend that
declares the capability and never reads the field is indistinguishable from
one that succeeded.
Both paths now check a post-condition against the description the backend
itself returned, so no backend can opt out: an index asked to be created
must be present, and one asked to be deleted must not be present and
ACTIVE. Deliberately tolerant about which post-state is correct, since that
is unmeasured, so doing nothing is caught without asserting a lifecycle
this contract has not observed. Reported as an internal fault rather than a
validation error, because it is a bug in the backend and not something the
caller did wrong.
The delete direction is the one that motivated this. Deleting a vector
index to stop serving a set of embeddings is something people do for
reasons that are not performance, and being told it worked while the
vectors remain queryable is the kind of failure that is discovered by
someone else.
Measured against the live service
Most of the specifics were measured against real DynamoDB rather than
inferred, and in nearly every case the measurement contradicted the
inference:
- the DistanceFunction enum order is [DOT_PRODUCT, COSINE, EUCLIDEAN],
which is neither alphabetical nor declaration order
- a search schema accepts at most one HASH element, and the inline
filter cap is 18, not the 20 that follows from the query-side limit
- a HASH element is OPTIONAL; SearchConditionExpression becomes
required only when one is declared
- a component must be representable as a 32-bit float; f32::MAX exactly
is accepted, and excess decimal precision is accepted, not rejected
- N never comes back in scientific notation, and the 38-digit limit
bounds significant digits rather than characters
- an absent vector attribute is accepted; an empty list draws the size
message with Actual: 0
- TableThroughputMode is not a member of CreateTable at all, so an
alias for it made ExtendDB accept a request AWS ignores
Error messages are pinned by whole-string equality, not by fragment. That
matters because fragment assertions beginning after the "One or more
parameter values were invalid" prefix hid three divergences: a colon where
the service uses a full stop, twice, and one message missing the prefix
entirely.
The refusal suite cannot silently stop running
Because the capability is optional, a suite that adapts to whatever the
backend reports never asserts WHICH backend is under test. The refusal
tests would then self-skip the moment any backend gained vector support,
and the contract that non-participating backends refuse would stop being
checked anywhere while still reporting green.
EXTENDDB_EXPECT_VECTORS lets a run state its expectation, with three states
rather than two: 0 means the backend must refuse, so a skipped refusal
suite is an error; 1 means it must support, so a backend that refuses is an
error; unset stays adaptive so a plain local cargo test works without
ceremony. An unrecognised value panics. The Rust integration job pins 0,
since no in-tree backend implements vector search, and the first backend
that does sets 1 in its own job.
The mechanism lives here rather than with the first implementation on
purpose. Whoever implements vector search inherits the guard instead of
inventing one, which is where the hole would otherwise be introduced.
All four states were exercised against a live backend, which found two
defects in the guard itself. The expectation was read inside a
short-circuiting &&, so an invalid value was never validated on a backend
without vector support, meaning the typo guard was dead exactly where it
was needed. And only one direction of the contradiction was asserted, so a
run claiming support the backend does not have passed unchecked, there
being no positive suite yet to notice.
Only the tests asserting a REFUSAL are gated on the capability. Three tests
in the same file assert behaviour true of every backend, that a plain
CreateTable still succeeds, that an empty vector list is not a vector
request, and that SearchVectors is a known operation requiring auth, and
they stay on the endpoint guard so they keep running once a backend
implements vector search.
Deliberately absent
No backend implements this, so no vector index can yet be created in tree.
The backfill state and the UpdateTable create and delete paths are modelled
but unvalidated, because validating them requires a backend that performs a
backfill; expect that portion to move. Async propagation of index writes is
not modelled. Errors report the bare field name where the service reports a
positional path such as vectorIndexes.1.member.distanceFunction, which
cannot be closed by a serde deserializer that does not know its own
position.
Verification
Refusals are asserted over the wire against a live server rather than by
calling the validator, using a SigV4 raw-request helper because no SDK
version carries the vector types. That suite immediately found a real
defect the unit tests could not see: UpdateTable's at-least-one check
omitted VectorIndexUpdates, so a request carrying only vector index
changes was rejected as empty, contradicting what the live service accepts.
The post-condition guard was verified over the wire too, against a backend
that declared the capability and dropped the field: both the create and the
delete now fail where they previously returned 200, and an ordinary
UpdateTable and a CreateTable carrying vector indexes are unaffected.
427 Rust integration tests and 742 workspace tests, 0 filtered out, fmt
and clippy -D warnings clean on both feature sets.
…ed capability First working slice of the SQLite implementation: a vector index can be created, is persisted, is reported back, and can be searched. Write maintenance, the UpdateTable path and backfill follow. Storage is one row per vector, which is a measured decision. A packed blob per partition reads 2 to 4x faster at 256 dimensions and 1.3 to 2.5x at 1024, and loses at 4096, but it makes every write O(partition): inserting one vector rewrites the whole blob, 390 MB for a 100k-vector partition at 1024 dimensions. Vector indexes are maintained on every write touching an indexed attribute, so that trade is not available at any read speed. Row-per-vector also streams, has no blob ceiling, and reuses the existing per-index table pattern. Exact scan rather than approximate, also measured rather than assumed. No SQLite vector extension meets this backend's constraints: a static-musl FROM scratch build cannot dlopen a loadable extension, the only extension with a compatible licence and an in-database index is brute force in every stable release anyway, and every real-ANN option stores its index in a sidecar file, forbids transactions, or is not open source. Measured throughput on one core, warm: 213k to 334k vectors/sec at 256 dimensions, 94k to 103k at 1024, 39k to 43k at 4096. The scan is bound by getting bytes out of SQLite, not by the arithmetic: a zero-copy &[f32] view of the blob measured no faster than decoding per element, so optimising the distance loop would be wasted effort. Notes on specific choices: * Vector metadata lives in its own `vector_indexes` catalog table rather than in `indexes`. A vector index is not described by a key schema, so reusing that table would mean storing something meaningless in a NOT NULL column. Two CHECK constraints encode the measured wire contract, including that an ACTIVE index must not carry the `backfilling` member at all. * Vector data tables are named with the base table_id as well as the index_id. That is what lets `drop_data_table` clean them up: it runs after the catalog rows have been cascade-deleted in the same transaction, so the index ids are no longer readable, and the names are instead discovered from sqlite_master. Without this, dropping a table would orphan its vector tables permanently. * Top-k consults the distance function rather than assuming one direction. Cosine and Euclidean are distances so smaller wins, dot product is a similarity so larger does; a single ordering would silently return the worst matches for one of the three. Tested both directions. * An index with no HASH element uses a reserved partition value no real key can produce, so an unscoped search is one partition rather than a second code path. Three defects found and fixed on the way: * The catalog version seed was `INSERT OR IGNORE` with a hardcoded literal, so `extenddb migrate` would create new objects and still leave the recorded version stale, and the server would keep refusing to start on a mismatch. Now an upsert, with two tests pinning the literal to CATALOG_VERSION and forbidding a regression to IGNORE. Verified by migrating a live deployment 0.0.2 to 0.0.3. * `IndexStatus` was not re-exported from core, so a backend could not name the type it is required to set on every VectorIndexDescription. Same class of gap as the unnameable BoxFuture. * CreateTable's response did not echo the vector indexes it had just created. The negative wire suite now self-skips when the backend supports vectors, probing by attempting the smallest real vector CreateTable. The probe distinguishes a refusal from any other failure and panics on the latter, because treating "not a 200" as unsupported would make the suite skip silently the first time an unrelated error appeared. Verified against a live SQLite server: 427 integration passed / 0 failed / 0 filtered out, workspace 746 passed / 0 failed / 0 filtered out, fmt clean, clippy -D warnings clean on both feature sets. Persistence confirmed by inspecting the database directly: catalog row ACTIVE with no backfilling member, plus the table-scoped data table and its partition index.
…r the wire Vector search now works end to end: an item written to a table with a vector index is indexed, findable, replaceable and removable, and a scoped index isolates partitions. Nine wire tests against a live server cover it. Maintenance is synchronous, inside the base write transaction, so a vector row cannot survive a rolled-back item write or be lost to a crash between the two. That is STRICTER than the service, which makes vector indexes eventually consistent like a GSI, and it is a deliberate first step rather than the final answer: being fresher than required cannot give a wrong answer, whereas being staler can. The asynchronous path should reuse the existing gsi_pending queue, which already provides crash recovery, per-key FIFO and a configurable delay. Until it does, a search immediately after a write sees the new item where the service might not. Applied at all six write sites, not just the obvious three. PutItem, UpdateItem and DeleteItem plus the three TransactWriteItems branches; missing the transactional ones would have meant a transactional write silently skipping the index. Deliberately placed OUTSIDE the existing `if !indexes.is_empty()` guard, because that guard is about GSIs and LSIs and a table may have a vector index and neither. Two things the design turns on: * Rows are keyed by the base item, not by the partition. An item whose HASH attribute changes must MOVE between partitions, and keying by partition would leave the old row behind so the item was findable under two tenants at once. The write path therefore deletes by base key before inserting. Tested directly. * Sort keys bind through parse_sk + sk_bound, the same D2 representation the GSI and LSI tables use: order-preserving text for numbers, BLOB for binary. My first version used pk_to_text, which would have been self-consistent but divergent from every other index table for the same item. Extraction and the norm live in core beside validate_vector_write rather than in the backend, because the two must agree on what a vector attribute is and separating them invites a backend storing something the validator would reject. Five tests pin them together, including that a component overflowing f32 is rejected rather than becoming infinity and poisoning every distance. One defect found by the failing tests: table_key_info never populated vector_indexes, so the write-path gate was always false and nothing was ever indexed. Now populated, which also lets core validate vector attributes on writes. Noted while doing it: VectorIndexKeyInfo still carries no distance function and no projection, so a search reads the catalog for those; widening that type would remove the last per-search catalog read. Also removed two `..Default::default()` spreads that clippy correctly identified as dead now that every field is populated. For a site that builds a complete value, a new core field SHOULD break it and force a decision rather than defaulting silently. Verified against a live SQLite server: 436 integration passed / 0 failed / 0 filtered out (9 new), workspace 751 passed / 0 failed / 0 filtered out, fmt clean, clippy -D warnings clean on both feature sets.
…or suites ran Without this, the vector implementation is untested in CI while reporting green. The existing Rust integration job runs against Postgres, which does not implement vector search, so `vector_index_search` self-skips there and every assertion in it is silently never executed. This job is the one that actually exercises it, and it re-runs the whole suite on the second backend, which has caught backend-specific drift before. The job also guards against the failure mode that self-skipping suites invite. Both vector suites skip when the backend is the wrong kind, so a green run proves nothing on its own: a broken probe, or a backend that stopped declaring vector support, would skip everything and still pass. The final step re-runs the positive suite and fails unless it reports a non-zero pass count above a floor. A skipped suite reports zero passes, which the guard catches. One difference from the Postgres job, found by running it: SQLite's `init` does not print an Account ID, so the account is read back from the catalog with `list-accounts | jq` rather than scraped from the init output. Both the derivation and the pass-count extraction were verified against a live deployment rather than assumed: the account resolved and the guard read 9 passes against its floor of 5. Throttling enforcement is enabled as the Postgres job does, because the suite's capacity_throttling tests fail without it; I reproduced that locally before wiring it in. The aggregator now gates on this job too, so it cannot go red unnoticed.
ADR-0004. Records why vector search is an exact scan over one row per vector, so a future contributor does not have to reverse-engineer it from the code or re-run the elimination. Nine options considered and named individually, each eliminated by a hard constraint rather than by preference: static-musl FROM scratch cannot dlopen a loadable extension, the project is Apache-2.0 so a non-OSI source-available licence cannot be carried, and the index must live in the database file because the backup and restore path would not capture a sidecar. That eliminates every SQLite vector extension, including the one with an in-file index (libSQL's LM-DiskANN, which would also require replacing sqlx and cannot pre-filter a partition). The ADR is explicit that the only viable extension would not have bought an index anyway: sqlite-vec is brute force in every stable release, so adopting it meant a C toolchain, an unverified static-musl build and a pre-1.0 dependency in exchange for a constant factor on the same asymptotic scan, while still lacking dot product. Two measured findings are recorded because they are the sort of thing that otherwise gets repeated: * An earlier modelled estimate of ~36,000 vectors at 1024 dimensions inside 10 ms was wrong by 10 to 30x. Measured effective throughput is 0.2 to 1.2 GB/s, not the ~15 GB/s the model assumed, so the scan is not memory-bandwidth bound and any figure derived from that model is void. The ADR states the void explicitly rather than quietly replacing the number. * The cost is the SQLite read path, not the arithmetic. A zero-copy &[f32] view of the blob, which should have let the dot product vectorise, measured no faster. Optimising the distance loop would be wasted effort until the read path changes. The layout decision is recorded as turning on write amplification rather than read speed: a packed blob per partition reads 2 to 4x faster at 256 dimensions but rewrites the whole blob to insert one vector, 390 MB for a 100k-vector partition at 1024 dimensions, on every write touching an indexed attribute. The trigger for revisiting is stated as a measurement rather than a judgement: an index declared with no HASH element searches the whole table, and past the recorded figures that leaves an interactive budget. Named candidates for that case, gated on a measured need. Status is Proposed; the repo's process marks an ADR Accepted on merge. Indexed in docs/adr/README.md as the process requires.
…, and two weak assertions An independent adversarial review of the vector test suites found one blocking hole and mutation-proved a weak assertion. This addresses both, plus one severity correction where the review overstated the risk. BLOCKING, now fixed: the two wire suites self-skip symmetrically with no anchor. `vector_index_search` skips unless the backend supports vectors; the refusal tests in `vector_index_unsupported` skip when it does. Nothing asserted WHICH backend was under test, so if the shipping backend silently lost vector support, the positive suite would skip all nine assertions (green) while the refusal tests started passing (also green) and the entire positive contract would evaporate unnoticed. `EXTENDDB_EXPECT_VECTORS` now lets a run state its expectation, deliberately with three states rather than two: `1` makes a skipped positive suite a failure, `0` makes a skipped refusal suite a failure, and unset keeps both adaptive so a plain local `cargo test` works against either backend without ceremony. My first version failed whenever the variable was absent, which broke local runs for no safety gain. All four states were verified against a live backend, including that an invalid value panics rather than being read as one of them. Both CI jobs now pin their expectation. This also answers the review's second blocking question, whether a non-vector backend exercises the refusal tests anywhere: it does, in the Postgres Rust job, and that is now enforced rather than incidental. The ad-hoc pass-count scraping step added earlier is removed, since the in-suite anchor supersedes it and fails at the assertion rather than by parsing output afterwards. SEVERITY CORRECTED. The review reported the unscoped-partition sentinel as a cross-tenant leak: `pk_to_text` stores an `S` attribute verbatim, so a caller could supply the sentinel string as a partition key. The finding is real about the test but wrong about the risk, and I checked rather than accepting it. The partition is chosen from the INDEX's schema, not from the item, and each index has its own data table, so within one table either every row is keyed by a real hash value or every row uses the sentinel. They never coexist, so there is nothing for a collision to leak into. The constant now documents that the guarantee is structural and that the leading NUL is defence in depth rather than the thing correctness rests on, because a reader who believed otherwise would feel free to weaken it. The test is still fixed, because the review was right that it asserted nothing of the sort. It compared the sentinel against one example value and passed unchanged when the sentinel was weakened to the ordinary string "unscoped" (the reviewer proved this by mutation). It now asserts the actual invariant, that the partition comes from the index schema and never from the item, across several values including the sentinel itself. Mutation-checked in both directions: weakening the sentinel now fails, restoring it passes. Also strengthened `overwriting_an_item_replaces_its_vector`, which asserted the row count only. A delete-then-insert that reinserted the OLD image would leave exactly one row and pass, while the stored vector was stale. It now asserts the score, which must be ~0 against the new vector where the old one was its exact opposite. Verified: workspace 752 passed / 0 failed / 0 filtered out, integration suite green with the expectation pinned, clippy -D warnings clean, my test files formatted (the repo has pre-existing rustfmt drift elsewhere that I deliberately did not touch). Remaining review findings, not in this commit: coverage gaps for BatchWriteItem and TransactWriteItems, UpdateItem removing a vector attribute, composite base keys, the scoped-index condition requirement, inline filters, and wrong-dimension requests over the wire.
…ee of them measured An independent adversarial review of the contract layer found nothing blocking but four should-fix items, three of which needed the live service to settle. I probed all three rather than accept or dismiss them, and in every case the measured answer differed from what either of us had inferred. SEARCH SCHEMA WAS NOT STRUCTURALLY VALIDATED, and the gap was a real contradiction rather than a missing nicety. A create or update could declare more than one HASH element; the query side then requires a condition for EVERY declared HASH while a backend resolving the scope takes the first and demotes the rest to filters. So the contract accepted a schema it could not honour. Measured caps and messages: HASH count <= 1 INLINE_FILTER count <= 18 The filter cap is the interesting one. The obvious inference from the query-side limit (MAX_SEARCH_CONDITIONS, one HASH plus twenty filters) gives twenty, which is what the review suggested hardcoding, and it is wrong. A test now pins 18 explicitly so a later edit "tidying" it to match the query cap breaks rather than silently diverging. Boundary tested at the cap and one over. DISTANCE FUNCTION ENUM ORDER was a guess and was wrong. The failure message listed [COSINE, DOT_PRODUCT, EUCLIDEAN] (alphabetical); the review guessed the enum's declaration order; the service says [DOT_PRODUCT, COSINE, EUCLIDEAN], which is neither. Now corrected and asserted over the wire. The same probe exposed a second divergence in that message which I have NOT fixed and have documented instead: the service reports the positional path 'vectorIndexes.1.member.distanceFunction' where this reports the bare 'distanceFunction'. A serde deserializer for the enum cannot know its index within the request, so closing it means deserialising the field permissively and validating positionally, exactly as the required-Projection check already does. That is a separate change with its own risk, not something to slip in here. TableThroughputMode ALIAS REMOVED. It arrived with my port of the internal branch carrying an unverified comment claiming "some clients send the billing mode under that name". aws-sdk-dynamodb 1.119.0 has no such member anywhere: CreateTable's request shape is BillingMode only. Accepting the alias meant a request that produced a PAY_PER_REQUEST table here would be ignored by AWS and produce a PROVISIONED table there, so code written against ExtendDB would break against the real service. The two tests that asserted the alias now assert the opposite, that an unknown member is ignored, which is the AWS JSON 1.0 behaviour. RestoreTableFromBackup now applies validate_vector_index_readiness, matching create and describe. Currently harmless, since a restored index is CREATING and a non-vector backend could never hold a vector-index backup, but it was the one description-returning path that omitted an invariant whose whole stated purpose is to guard those paths. Two stale doc comments fixed: one still named an error from the deployment flag removed in 79897d4, and one described the same function twice. The surviving version also states plainly that multi-fault parity is not attempted, since the service aggregates faults with its own ordering while this returns the first and hardcodes a count of one. Verified: workspace 758 passed / 0 failed / 0 filtered out, integration 438 passed / 0 failed / 0 filtered out with the capability expectation pinned, fmt clean, clippy -D warnings clean on both feature sets. Both new validations asserted over the wire, not only in unit tests.
…e metric claims to exclude
An independent adversarial review of the SQLite implementation found nothing
blocking and confirmed the property that mattered most: every item-mutating site
maintains the index inside the write transaction, including all three
TransactWriteItems branches, BatchWriteItem transitively through put/delete, and the
TTL sweep. It also found one real divergence with a billed consequence.
THE STORED ROW IGNORED THE INDEX PROJECTION. The GSI sibling stores
`project_item_for_index(...)`; the vector path stored the whole item and applied no
projection at write or read. That is an unexplained divergence from the sibling, and
a search returned attributes the index does not declare. Now projected, mirroring the
sibling.
THE BILLED METRIC WAS INFLATED BY A COMMENT THAT WAS SIMPLY UNTRUE. The capacity
computation in the SearchVectors handler read:
// ... excluding the vector component (the stored item already omits the
// vector attribute).
let non_vector_bytes = hits.iter().map(|h| item_size_bytes(&h.item)).sum();
No backend omitted it, so `VectorSearchRequestBytes` was over-reported by the
vector's serialized size on every hit, roughly 10 to 15 KB at 1024 dimensions. I
wrote both the storage and that comment and never connected them, which is exactly
what an independent read is for. The vector attribute is now subtracted explicitly,
so the figure no longer depends on an assumption about how a backend stores rows.
I did NOT strip the vector from the stored row, and the reason is recorded in the
code rather than left implicit: the vector attribute is an attribute of the item, so
a Projection of ALL includes it, and whether the service returns it in a search
result is unverifiable from here. `SearchVectors` is not served by the standard
DynamoDB endpoint at all ("This operation is not supported by this endpoint"), which
I found by probing. So the duplication between the `vec` column and the row payload
is deliberate pending an endpoint that can answer the question, not an oversight.
The `LIKE` pattern for the table-cleanup sweep is now `GLOB`. The old comment
justified the wrong half: it noted that a UUID contains no LIKE metacharacters, which
is true and beside the point, because the pattern's own underscores are
single-character wildcards. It could only ever over-match and no other table can
share the UUID, so it was safe in practice but not for the stated reason, and a
comment that justifies the wrong half is worse than none. GLOB treats `_` literally.
Verified the sweep actually works rather than assuming: after the full suite the
database holds 21 vector data tables against exactly 21 catalog rows, with zero
orphans and zero missing. Dropping a table takes its indexes with it, which is why
the `_vidx_part_*` index needs no separate handling.
The review also flagged, out of scope, that no code drops a single vector index's
data table. Correct, and it is not a leak today because the UpdateTable vector paths
are not implemented in this backend yet. It is a prerequisite for that work rather
than an existing defect.
Verified: workspace 758 passed / 0 failed / 0 filtered out, integration 438 passed /
0 failed / 0 filtered out, fmt clean, clippy -D warnings clean on both feature sets.
…n names it The vector attribute is returned by SearchVectors only when the caller explicitly asks for it in ProjectionExpression, even under a Projection of ALL. This was the one part of the previous commit I could not settle: SearchVectors is not served by the standard DynamoDB endpoint, so no probe from here could answer it, and I recorded the uncertainty rather than guessing. Lee confirmed the behaviour. The change is narrow because the existing projection path already does most of it. When a ProjectionExpression is supplied, the compiled projection restricts the item to the paths it names, so naming the vector keeps it and not naming it drops it, with no special case needed. Only the no-projection branch had to change, and it now removes the vector attribute. That also makes the earlier capacity comment right in intent, though for a different reason than it claimed. It asserted the STORED item omits the vector, which was false: the SQLite path holds the projected item verbatim, exactly as the GSI path does. What is true is that the RETURNED item omits it. The subtraction stays, because the capacity figure is computed from what the backend handed over, before the response projection runs, so it must not assume how a backend stores its rows. The comment now says that rather than the thing that was wrong. I deliberately did not strip the vector at write time, which was the other way to satisfy this. Storing it once and reconstructing it on demand would save roughly 16 KB per row at 4096 dimensions, but it would return an f32-narrowed value rather than what the client wrote, since the wire type is arbitrary-precision decimal and the index narrows to f32. Keeping the projected attributes verbatim matches the GSI model and preserves exactly what was written; the storage cost is the price of that fidelity, and it is now a considered trade rather than an accident. The test asserts both directions, because either half alone passes against a broken implementation: "absent by default" passes if the attribute is never returned at all, and "present when named" passes if it is always returned. The default case also asserts the rest of the item IS present, so it cannot pass merely because nothing came back. Verified: integration 439 passed / 0 failed / 0 filtered out, with the new test confirmed to have executed rather than skipped. Workspace 758 passed / 0 failed / 0 filtered out, fmt clean, clippy -D warnings clean on both feature sets.
A vector index holds 32-bit floats. The service's own validation is the
evidence: it rejects a component outside [-3.4028235E38, 3.4028235E38],
exactly f32::MAX, and names the expected type "32-bit floating point
number". Reading a narrowed value back could not be measured directly,
because SearchVectors is not served by the standard DynamoDB endpoint.
The index row was therefore carrying the vector twice: once as f32 in the
`vec` column, and once as the client's verbatim decimal text inside the
projected row payload, with searches returning the latter. That was wrong
in two ways. It returned more precision than the index stores, which no
client could get from the service, and it duplicated the vector at roughly
five times the size of the blob it duplicated.
Searches now reconstruct the attribute from the stored bits via a new
core `vector_attribute()`, the inverse of `vector_components()`, so the
returned value is the one that was actually indexed. Measured at 1024
dimensions: the row payload falls from 19,862 bytes to 37, against a
4,096-byte blob.
Three details worth recording.
The reconstruction happens for the k surviving hits, not per candidate
scanned: the components ride along inside TopK. Rebuilding during the scan
would allocate a decimal string per component per row examined, which at
4096 dimensions over a large partition would cost more than the scan
itself. A test asserts each retained hit keeps its own vector, since an
off-by-one in the insert position would pair a neighbour's vector with
this item's attributes: wrong data, no error.
Two formatting exceptions to f32's Display, which otherwise gives the
shortest round-tripping decimal. Negative zero is normalised, because N
has one zero, and the sign of zero is unobservable through any of the
three distance functions. And a magnitude outside [1e-6, 1e21) is written
with an exponent, because Display never uses one and would expand f32::MAX
to 39 digits, past the 38 that N carries. A test pins the round-trip
through vector_components for f32::MAX, MIN_POSITIVE and a subnormal.
The stored `vector_attribute` column holds serialized JSON, not a bare
name. Reading it as a plain string produced the key
{"AttributeName":"emb"}, so the vector was silently absent from every
projection. The unit tests could not see this; the wire test caught it
immediately, which is the second time this suite has paid for itself.
Verified against a live SQLite deployment: 440 Rust integration tests
(0 filtered out, up from 436 with the new case), 764 workspace tests,
fmt and clippy -D warnings clean on both feature sets. The base item is
asserted to keep the client's own precision alongside the narrowed index
value, so the test cannot pass against an implementation that merely
corrupted the item on write.
…moDB
Everything below was measured on 2026-08-07 against the live service in
us-east-1, using raw signed requests because the installed CLI predates
vector indexes. Three of the four were wrong by inference, and none of the
existing tests could see it.
Number formatting. The service NEVER returns an exponent in an N value,
whatever form it was sent: 3.4028235E+38 reads back as
340282350000000000000000000000000000000, and 1E-40 as
0.0000000000000000000000000000000000000001. Both are exactly what f32's
Display produces, so returning a reconstructed vector is now plain decimal
always. The previous commit wrote extreme magnitudes in scientific
notation on the reasoning that a 39-digit expansion would exceed the 38
digits N carries. That reading was wrong: the limit is on SIGNIFICANT
digits, confirmed by a separate error for a 39-significant-digit value
("Attempting to store more than 38 significant digits in a Number") and by
a 38-digit value being accepted and returned verbatim. An f32's shortest
round-tripping form carries at most 9 significant digits, so no stored
component can ever approach the limit however long the expansion runs.
Three error messages diverged.
The non-list message was invented rather than measured, and was wrong in
three ways at once: a colon after "invalid" where the service uses a full
stop, the phrase "a list of numbers" where the service says "32-bit
floating point number list", and a full stop before IndexName that the
service omits. N, S and NS in the vector position all produce the same
text.
The out-of-range message omitted the "One or more parameter values were
invalid. " prefix entirely.
The wrong-type message used a colon after "invalid" where the service uses
a full stop.
The tests could not catch any of this because they asserted fragments that
began after the prefix, which is exactly where the drift was. All three now
assert the whole string by equality. The one message that was already
pinned whole, the dimension mismatch, was correct.
Confirmed while measuring, and unchanged: the value in the out-of-range
message IS normalised to scientific notation whatever form it arrived in
(a plain 39-digit integer drew "Value: 4E+38"), a component at exactly
f32::MAX is accepted, excess decimal precision is accepted rather than
rejected, an absent vector attribute is accepted, an empty list draws the
size message with Actual: 0, and negative zero reads back as 0, matching
the normalisation in format_component. N also trims one trailing zero,
which the new wire test had to account for.
440 Rust integration tests (0 filtered out), 764 workspace tests, fmt and
clippy -D warnings clean on both feature sets. Probe tables deleted.
Closes the hole the engine post-condition guard exposed. The backend declared vector support and never read vector_index_updates, so a Create returned 200 and created nothing, and a Delete returned 200 while the index stayed ACTIVE and kept returning hits. Structured as the GSI sibling is, two-phase: the catalog row commits first at CREATING, then the data table and backfill run, then the row flips to ACTIVE. A crash in between leaves a CREATING row for the startup reconciler rather than an ACTIVE index over a missing or partial table. The status sequence is the one measured against the service on 2026-08-06 by seeding 3000 items of 1024 dimensions so the backfill took 8.5 minutes and could be observed: CREATING with Backfilling false, then CREATING with true, then ACTIVE with the member absent. Writing false first rather than jumping to true matters because presence does not imply backfilling, so a client must read the value; and the flag is set outside the backfill transaction, since a flag that only exists inside the transaction doing the work cannot be observed by anyone while the work happens. The row writer is now shared, not duplicated. insert_vector_row is called by both the write path and the backfill, because they are the only two producers of a vector row and a second copy would be free to drift: a backfilled row shaped differently from a live-written one searches correctly right up until the difference matters, with nothing to catch it. A test writes one item before the index exists and one after, and requires both to be found, which is what actually pins the two paths together. Also adds drop_vector_data_table_by_id, which an earlier review flagged as missing. The table-drop path sweeps sqlite_master because the catalog rows have already cascade-deleted by then and the index ids are unreadable; here the id is known, so the name is derived directly. Crash recovery reconcile_incomplete_vector_indexes runs at startup beside the GSI reconciler, as a separate pass rather than a shared one: the two live in different catalog tables and are built by different code, so a failure to reconcile one must not skip the other. It drops and rebuilds rather than resuming, and there is a test for exactly that, because the crash that actually happens leaves the data table holding SOME of the rows. Resuming would index those items a second time and a search would return the same item twice. The simpler test, whose simulated crash leaves no data table at all, cannot catch this: the drop is a no-op there. Both were mutation-checked, removing the drop fails the partial-table test and restoring it passes. Verification The two backfill wire tests were mutation-checked as well: with the backfill neutered to write nothing, both fail, and the second reports only the later-written item, so it genuinely distinguishes backfilled rows from live-written ones. Five new wire tests: backfill of pre-existing items including one without a vector that must be skipped rather than break the scan, later writes still indexed, delete stops both serving and reporting, delete leaves the base items untouched, and duplicate-create plus missing-delete rejected. Three reconciler unit tests, one pre-existing. 445 Rust integration tests and 773 workspace tests, 0 filtered out, fmt and clippy -D warnings clean on both feature sets.
The contract commit emitted a top-level `Count` in `SearchVectorsOutput`. The
service does not, and nothing in the suite could have caught it: a generated
client ignores unknown top-level members, so an extra field is invisible unless
something asserts its absence.
Measured in sandbox 964157134968 us-east-1 on 2026-08-10 across five parameter
variations, none of which produced a `Count`: no projection; with
ReturnConsumedCapacity=INDEXES; with a ProjectionExpression naming the vector;
with TopK larger than the item count; and with a projection naming a single
non-key attribute. The botocore 1.43.64 model agrees, declaring exactly
`SearchResults` and `ConsumedCapacity`.
The test asserts three cases rather than one. The plain search covers the normal
path. TopK exceeding the match count is included because that is precisely when a
`Count` field is most tempting to add. The third case requests ConsumedCapacity
and then asserts the response's top-level member set contains nothing beyond the
two legal members, so any future extra field fails here rather than being
discovered by a client.
Negative control: restoring the `Count` field fails the test with
`{"Count":2,"SearchResults":...}` in the message, so it discriminates. With the
field removed, the vector suites are 26/26 with 0 filtered out.
The vector-attribute projection rule Lee described (returned only when named in
ProjectionExpression, even under Projection ALL) was re-confirmed live in the same
probe run and is already asserted in both directions by
`the_vector_attribute_is_returned_only_when_named`.
LeeroyHannigan
force-pushed
the
feat/sqlite-vector-search
branch
from
August 10, 2026 20:44
c37f379 to
6031daf
Compare
…nd and per-table index limits
Three divergences from the documented service behaviour, found by checking the
implementation against the developer guide and the quota table rather than
against the shapes alone.
1. KEYS_ONLY dropped the inline filter attributes, breaking filtered search.
A vector index does NOT use GSI KEYS_ONLY semantics. The documented rule is
that KEYS_ONLY projects the base table primary key, the vector attribute AND
any inline filter attributes declared in the SearchSchema. The write path
called the shared GSI projection helper with an empty index key schema, so
under KEYS_ONLY only the base keys survived.
This was not merely a reporting difference. The inline filter is evaluated in
`vector_search.rs` against the stored payload, so a missing filter attribute
made `item.get(name)` return None for every row, every row failed the
predicate, and a filtered search returned zero results. Reproduced before
fixing: the new test failed with `{"SearchResults":[]}`, 0 against an expected
1.
The SearchSchema attribute names are now carried on `VectorIndexMeta` and
retained regardless of ProjectionType. The vector attribute itself is still
excluded from the payload deliberately, because it lives in the `vec` column
at f32 width and the search path rebuilds it from those bits.
It escaped notice because `create_vector_table` hardcodes ProjectionType ALL,
so all 18 pre-existing search tests exercised the single projection under
which the distinction cannot appear.
2. A vector index was accepted on a PROVISIONED table.
Vector indexes are supported only on on-demand tables. BillingMode defaults to
PROVISIONED when absent, so an omitted BillingMode is rejected too, and the
test asserts both spellings because an implementation checking only the
explicit value would pass one and fail the other.
3. No limit on vector indexes per table.
The documented default quota is 5. Asserted in both directions so an
off-by-one cannot hide: five is accepted, six is refused.
Ordering note: per-index shape validation runs before the two table-level checks.
Which the service reports first is unobservable from outside, because botocore
rejects a malformed index client-side before the request is sent, so the order
that preserves the already-measured per-index messages is the one kept. One
existing unit test built its positive fixture without a BillingMode and now needs
PAY_PER_REQUEST; that fixture was updated rather than the check weakened.
Verification: negative controls for both changes fail as required, the KEYS_ONLY
one with an empty result set and the quota one with a 200 where a 400 is owed.
Vector suites 28/28 with 0 filtered out. Full Rust integration suite 449 run,
448 passed; the single failure was `restore_active_completeness` throttling out
because this deployment still had `throttling_enabled` set from an earlier
capacity run, and it passes 1/1 with that setting cleared. 433/433 core unit
tests. fmt exit 0, clippy -D warnings exit 0.
Vector indexes are eventually consistent, the same model the service gives them
and the same model a GSI has here. Maintenance was applied synchronously inside
the base write transaction, which is stricter than the service and made a search
immediately after a write return the new item where the service might not.
Maintenance now runs on the existing gsi_pending queue rather than a queue of its
own, which is what makes the asynchronous path correct rather than merely
deferred. None of these properties would come free otherwise:
* Crash safety. The pending row is inserted in the base write transaction, so
the item is never committed with its index work not yet durable. The worker
claims and applies in one transaction, so a crash mid-apply rolls back and
retries. At-least-once is safe because an apply is idempotent: it deletes the
base key's row and reinserts it from the snapshotted item.
* Per-key ordering across index kinds. The row's partition is a hash of the
base key, so a vector row and a GSI row for one item share a partition,
ready_at is clamped monotonic within it, and the worker drains in id order.
* Snapshot semantics. The row carries its own VectorApplyContext, so the worker
needs no catalog read and an index dropped or redefined between enqueue and
apply cannot make a queued write unapplicable.
maintain_vector_indexes is the single entry point and owns the sync/async choice,
so the seven write paths each have one call site. A delay of 0 still applies
inline; anything else enqueues. A write whose new item carries no vector still
enqueues, because the removal is the work in that case.
One queue carrying two kinds of work needs a discriminant. PendingApplyContext is
untagged, which is load-bearing rather than stylistic: a GSI context serializes to
exactly the bytes it did before, so rows already on disk still deserialize. That
matters because an unparseable index_context is treated as a poison row and
DROPPED, so a tagged representation would have silently discarded every in-flight
GSI update across an upgrade. The variants are unambiguous by shape, a GSI context
requires `index` and a vector context requires `vector`, and a test pins a
verbatim legacy payload.
Also fixes a latent defect in the queue that the ordering guarantee above depends
on. Apply order was resting on the order of DELETE ... RETURNING output, which
SQLite defines as undefined and which demonstrably ignores the subselect's
ORDER BY (a DESC subselect still returns ascending). Per-key FIFO was therefore
accidental: two writes to one item claimed in the same batch could be applied
newest-first, and because each apply overwrites the row wholesale, the earlier
write would win and the later be lost. Claimed rows are now sorted by id in code.
Verification. 9 new tests, each with a negative control:
* Forcing the synchronous path fails the asynchrony test.
* Reversing the apply-order sort fails the write-order test with exactly the
lost-update symptom (the earlier write surviving).
* Skipping the enqueue for a vectorless item fails the removal test.
* Making the worker's vector apply a no-op fails the converted integration
tests with "index never converged", proving they depend on the async path
rather than on residual synchronous behaviour.
The integration suite asserts convergence rather than a single search, because
under eventual consistency one search proves nothing in either direction: a
missing item may not have propagated and a removed item may not have been removed
yet. Bounded polling, not fixed sleeps, which are simultaneously too slow when
propagation is immediate and too short on a loaded machine. Where an absence is
the subject, the test first waits for something ordered behind it in the same
apply, so the absence cannot pass merely because the write was late.
Honest note on one control that did not discriminate: the missing-table tolerance
in apply_vector_context is log hygiene, not data safety. Removing it leaves the
tests passing, because the pre-existing per-row savepoint already contains the
error and reaches the same end state. What it changes is that a routine
DeleteTable race stops emitting an ERROR line. Both comments say so rather than
claiming more.
449/449 rust integration tests live over HTTP with throttling enabled and 0
filtered out, 40/40 storage-sqlite unit tests, fmt and clippy -D warnings clean.
…, and close review gaps Two things, both consequences of vector maintenance now sharing the GSI propagation queue: the setting that governs it was named for only one of the two index kinds, and an independent review of the previous commit found three test gaps and three comments that claimed more than they proved. ## The rename, and why it needs a fallback rather than a migration `gsi_propagation_delay_ms` now governs vector indexes as well as GSIs, so the name actively misleads: an operator reading it would reasonably conclude that vector search is unaffected by it. Renamed to `index_propagation_delay_ms`. A bare rename would have been a silent data-loss bug, which is why this is more than a search and replace. The server refuses to start on a catalog-version mismatch rather than migrating in place, and this change does not bump the catalog version, so a pre-rename catalog starts normally and keeps the operator's value under the old key. Reading past that row would reset a deliberately configured delay to the 10ms default, and because 0 means synchronous, the silent change would be from strict to eventually consistent: tests that assert steady state without waiting would start failing for a reason nowhere near the change. So reads prefer the canonical key and fall back to the legacy one, with an explicit `ORDER BY key = 'index_propagation_delay_ms' DESC` so the preference is deterministic when both rows exist rather than resting on row order. Writes to the legacy name are redirected to the canonical key, so a deployment converges on one row instead of accumulating two that disagree. `settings set gsi_propagation_delay_ms 0` therefore keeps working for anyone with it in a runbook. Both key strings and the resolver live in one place, `extenddb-core`'s `settings_keys`, because the literal is read by both backends, written by the management API, seeded by both schemas, and documented. Scattering it is what let the name drift out of step with its meaning in the first place. Two tests, each with a stated failure mode: `a_pre_rename_catalog_still_honours_its_configured_delay` reshapes a catalog to look as it did before the rename and asserts the value survives; `the_canonical_key_wins_when_both_are_present` pins the precedence. ## Review response `deleting_an_item_removes_it_from_the_index` had a real soundness gap: it never established that the doomed item was present before deleting it, so converging on the survivor was also satisfied in the window where the doomed item's write had not yet applied. It could pass without exercising the delete at all. It now converges on both items, deletes, then converges on the survivor. Two properties were claimed by the previous commit message and untested. `applying_the_same_row_twice_is_idempotent` covers the replay that crash safety rests on, and it discriminates: an earlier version of it did not, because under a conditional delete the replay collided and the end state was identical either way, so it was replaced with one that exercises a path where the difference is observable. `a_gsi_row_and_a_vector_row_for_one_item_share_a_partition` covers per-key FIFO across index kinds, which no test touched. Three comments corrected to say only what is true. The claim that exactly one context variant can ever match was wrong for a hand-corrupted blob carrying both discriminant fields; unreachable from any writer, but the comment should not assert it. Two test docs promised more than their assertions prove. `insert_vector_row` keeps a plain `INSERT` where the GSI sibling uses `INSERT OR REPLACE`, and now records why: the unconditional delete always precedes it, and a primary key violation is the desired outcome if a future refactor makes that delete conditional, where `INSERT OR REPLACE` would silently paper over it. Deliberately not fixed here: a stale queued row can clobber a newer inline write when the delay flips from non-zero to 0 with rows still in flight. It is pre-existing in the GSI sibling rather than introduced by vectors, so a vector-only fix would leave the two index kinds inconsistent. Tracked separately, to be fixed for both together. ## Docs `differences-from-dynamodb.md` documented GSI propagation and said nothing about vector indexes, which became a real gap once their consistency model changed. Adds a row stating that vector search is eventually consistent like a GSI, rides the same queue and setting, has no per-index override, and that 0 is stricter than the service. ## Verification fmt --check exit 0. clippy --workspace --all-targets -D warnings exit 0. 787 unit tests, 0 failed, 0 filtered out. Python integration on a fresh catalog, run as CI invokes it: 937 passed, 5 skipped, 1 xfailed, plus 327 comprehensive, 0 failures. Rust integration live over HTTP, single-threaded with throttling enabled and EXTENDDB_EXPECT_VECTORS=1 so a skipped vector suite fails rather than passes: 449 passed, 0 failed, 0 filtered out, 28 of them vector tests. A fresh catalog seeds only the canonical key.
…ainst DynamoDB
Ran the vector index surface side by side against real DynamoDB, both systems
driven through the same boto3 client shape with 1000 identical 128-dimension
items, and judged results against brute-force exact nearest neighbours rather
than only against each other.
Search quality needed nothing: 10/10 recall with exact ordering on every query,
for COSINE, DOT_PRODUCT and EUCLIDEAN, scores agreeing to float32 precision,
holding under filters, under KEYS_ONLY and INCLUDE projections, and for a
backfilled index versus an inline-built one. What follows is everything that did
not match. After these changes 12 of 13 re-probed cases agree, and all six
create-time messages are byte-identical.
## The one that mattered: a silent wrong answer
A search against a HASH-scoped index with no `SearchConditionExpression` returned
HTTP 200 with ZERO results. The service refuses it. The caller was told "no
matches" for an invalid request, indistinguishable from an empty table.
The code asserted in a comment that "validation upstream guarantees it is present
here". No such validation existed. The check is keyed off the resolved hash key
rather than off whether an expression was supplied, because an expression that
omits the HASH attribute leaves the search just as unscoped.
The integration test seeds two tenants and asserts the SCOPED search finds its
rows FIRST, so "zero results" cannot pass as correct. Control: reverting the
check gives `200 {"SearchResults":[]}` where 400 is owed.
## Requests accepted that the service refuses
`ProjectionType: INCLUDE` with no `NonKeyAttributes`. The rule and its exact
message already existed in `validate_index_projections`, which iterated GSIs and
LSIs and never vector indexes, so the fix is to iterate them rather than to
restate the rule.
A `VectorAttribute` naming an attribute also present in `AttributeDefinitions`,
including the table's own partition key. This rule was not modelled at all. It
follows from the shape: `VectorAttribute` carries no type because a vector is not
a scalar type AttributeDefinitions can express. Checked before the surrounding
rules because the attribute may simultaneously be a legitimate key, which is
exactly how it slipped through: `pk` satisfied both definition-exists and
definition-is-used.
A reserved keyword as a bare identifier in `SearchConditionExpression`
(`bucket = :b`). The check existed for `ProjectionExpression`, byte-identical to
the service, and was simply never wired into this expression type. Applied to the
bare form only, with a test asserting the aliased form still works, since
aliasing is the documented escape hatch and breaking it would trade one defect
for another.
## Cosine distance left its domain
An exact self-match returned -1.19e-07. Cosine distance has domain [0, 2]; the
f32 quotient can exceed 1 when the vectors are identical. The similarity is now
clamped before subtraction.
The existing test could not catch this: it asserted `s.abs() < 1e-6`, which the
negative value satisfies. Taking the absolute value discarded the sign, which was
the only thing wrong. The new test asserts non-negativity over 2000 generated
vectors, and its control reproduces exactly -0.00000011920928955078125.
## ConsumedCapacity was about 40% low and ignored returned vectors
`VectorSearchRequestBytes` reported 2748.8 where the service reported 6196.0. The
previous constant of 17.6 bytes per dimension came with a comment excusing
imprecision by a documented 1.176 bimodal spread, but the observed ratio was 2.25,
so the constant was wrong rather than the service noisy.
Re-derived by sweeping 16, 64, 128, 256, 512, 768, 1024 and 2048 dimensions:
* 30.6875 bytes per dimension, exact at every point in the low sweep
(intercepts 491, 1964, 3928, 7856, 15712).
* 72 bytes per returned result, flat across dimensions. The previous model had
no per-result term at all, so TopK did not move the figure.
* 4 bytes per dimension for each returned item carrying the vector, one float32
per dimension. This was absent entirely: projecting the vector doubled the
service's figure and changed nothing here.
Two things the sweep established that the old comment denied. The figure DOES
depend on how many items the index holds, about 19.8 bytes per item at 512
dimensions, which is why two sweeps disagreed by 3% at the same dimension. And
the old recorded observations, 18067 at 1024 dimensions and 36058 at 2048, are
not reproducible: the service returns 31206 and 61692 for the same shape today.
The constant is set from the 60-item observations so the model errs high on small
indexes rather than low on realistic ones. Also recorded, not fixed: a HASH-scoped
search costs LESS than an unscoped one (3675 against the 3928 the dimension term
alone accounts for), leaving the model roughly 38% high on a scoped search. That
term needs a sweep over partition counts that has not been run.
## Message wording, measured rather than paraphrased
Billing mode, index count cap, and SearchSchema attributes with no definition. The
last was reusing the GSI key-attribute message, which names the attribute and
lists every definition where the service says only that one element is undefined.
The unindexed-attribute message now names WHICH attribute, as the service's does,
reproducing its grammar slip ("attributes that is not") deliberately: parity means
matching what clients receive, not correcting it.
## Verification
fmt --check exit 0. clippy --workspace --all-targets -D warnings exit 0.
799 unit tests, 0 failed, 0 filtered out (12 new).
450 integration tests live over HTTP, single-threaded, throttling enabled,
EXTENDDB_EXPECT_VECTORS=1, 0 failed, 0 filtered out.
Seven negative controls, one per fix, each reverting only that fix and each
failing the intended test with the intended symptom.
Re-probed against the live service afterwards: 12 of 13 cases agree, the
remaining one being the scoped-search capacity term recorded above.
Two gaps the report flagged were closed and came back clean, needing no change:
the 18 inline-filter cap is already enforced with a byte-identical message, and
DOT_PRODUCT and EUCLIDEAN both rank identically to exact ground truth.
Not addressed here, as a deliberate design question rather than a defect: the
backfill is awaited inline, so `IndexStatus: CREATING` and `Backfilling` are
written to the catalog but flip to ACTIVE before the response is built, while the
service holds CREATING for over eight minutes on an empty table and refuses
searches throughout. Making that asynchronous would change what an index does
immediately after UpdateTable returns.
…s until it is ACTIVE
Adding a vector index held SQLite's write lock for the whole backfill inside one
transaction, so every write to the base table stalled until the index finished,
and `UpdateTable` returned with the index already ACTIVE. The service behaves the
opposite way: the table stays ACTIVE and writable throughout, the index reports
CREATING with a `Backfilling` member for over eight minutes on an EMPTY table when
measured, and a search against it is refused until it is done.
The backfill is now detached and commits per batch, which is what lets writes
proceed while it runs.
## Searches are refused while the index builds
Measured four times against DynamoDB on 2026-08-11: a search against a CREATING
index returns `The table does not have the specified index`, naming no status.
That is byte-identical to the message the engine already produces for an index
that is absent, so the gate is a filter on the resolution step rather than a new
string. `DescribeTable` still reports the index and its status, as the service
does; only the search path treats it as absent.
This is load-bearing now rather than cosmetic. While the backfill was one
transaction a partially populated index was unobservable. It is reachable now, so
without the gate a search mid-build answers from incomplete data: the control
returns `200 {"SearchResults":[]}` where 400 is owed.
## Writes during a backfill are held, then applied
The queue worker no longer claims rows for a table whose vector index is CREATING,
so writes that land mid-build accumulate and are applied once it goes ACTIVE.
The hold is per TABLE rather than per index. Holding only the vector rows would let
a GSI row and a vector row for the same item be applied out of order relative to
each other, which is the cross-kind FIFO property this queue is documented to
provide.
Worth stating plainly, because it changes what the hold is FOR: it is not what
prevents a stale snapshot from overwriting a newer write. A control with the hold
removed still converges correctly, because each batch reads and writes inside one
transaction under the write lock, so the backfill always indexes the CURRENT base
value and can never write a stale one. The hold guarantees the drain ordering and
preserves cross-kind FIFO; the atomicity is what makes staleness impossible.
## Pagination is by key, because batching made OFFSET unsafe
Removing an already-scanned row shifts every later position by one, so the next
batch skips a row entirely. That row is then missing from the index permanently and
no queue entry can repair it, because the skipped row was never written to: only
the removed one was. Reproduced before the fix, one removal during a backfill left
the row at the batch boundary absent, and the control with `OFFSET` restored fails
the new test with the boundary row never appearing.
This was unreachable while the backfill was a single transaction, since no
concurrent write could interleave. Batching created it, so it is fixed here rather
than left as a regression.
## Testability
The ordering property cannot be observed unless a write is guaranteed to land
mid-backfill, and a backfill over a test-sized table finishes faster than a client
can issue its next request. `vector_backfill_batch_delay_ms` pauses between batches,
outside the write lock, defaulting to zero and bounded at 60s by its validator. It
exists for the same reason `index_propagation_delay_ms` does.
The rust integration job gains `EXTENDDB_TEST_MGMT_PASSWORD` so the tests can set
it. Deliberately NOT named `EXTENDDB_ADMIN_PASSWORD`: that name also un-skips
`batch_transact_authz`, which hardcodes account 123456789012 while init generates a
random one, so all ten of its tests fail with `ResourceNotFoundException`. That
suite silently skipping and reporting green is a real hole, and a separate one.
Two existing tests searched immediately after `UpdateTable` and now correctly race
the build. Both wait for the index instead, which is what a real client must do.
One deliberately keeps its write BEFORE the wait so it still covers the hold, and
`wait_for_vector_index_active` exists because `wait_for_active` only waits on
`TableStatus`, which is ACTIVE throughout an index build.
`CreateTable` is untouched: it has its own path, and an index on a brand-new empty
table has nothing to backfill, so reporting ACTIVE immediately is correct there.
Crash recovery is unchanged and already covered: a crash mid-backfill leaves the
index CREATING, which `reconcile_incomplete_vector_indexes` rebuilds at startup,
including a partially-backfilled case it already tests.
## Verification
fmt --check exit 0. clippy --workspace --all-targets -D warnings exit 0, with the
eight-argument batch function refactored into `BackfillPlan` rather than suppressed.
799 unit tests, 0 failed, 0 filtered out.
Integration live over HTTP, single-threaded, throttling on, EXPECT_VECTORS=1:
451/452, 0 filtered. The one failure is `restore_active_completeness`, which passes
1/1 in isolation, is a known load-related flake, and sits in a path this change does
not touch.
Four negative controls, each reverting one thing: the status gate (search returns
200 with an empty result set from a partial index), keyset pagination (boundary row
never appears), and the queue hold (converges anyway, which is why the claim above
is narrowed).
Also verified end to end against a live server with the delay set: `UpdateTable`
returns CREATING/Backfilling=true, DescribeTable agrees, a search is refused, a
write lands mid-build, and after ACTIVE the index reflects the NEW value and no
longer the old one.
…tests stop skipping `batch_transact_authz` has reported green in every CI run since it landed in #232 on 2026-07-28 while executing nothing. All ten of its tests call `skip_no_admin()`, which returns early when `EXTENDDB_ADMIN_PASSWORD` is unset, and neither rust integration job set it. The suite is not at fault and neither is its hardcoded account. Its own comment says to run it via `devtools/run-tests --extenddb --rust-integration`, and that path is correct: `devtools/provision-test-credentials` creates account 123456789012 with an IAM user, access key and full-access policy, then exports the credentials. The suite targets exactly that account. What was wrong is that both jobs invoked `cd tests/rust && cargo test` directly, bypassing the harness, so neither the account nor the password ever existed. Each job then hand-rolled its own IAM provisioning against whatever random account `init` generated, which is enough for the suites that only need a data-plane caller and not enough for one that needs a known account. So both jobs now call the harness, and their bespoke provisioning steps are deleted rather than kept alongside it. Confirmed locally against a clean deployment: ten of ten authz tests execute and pass, and the run reports zero SKIP lines where it previously printed ten. The vector backfill tests go back to reading `EXTENDDB_ADMIN_PASSWORD`. They briefly used a separate `EXTENDDB_TEST_MGMT_PASSWORD` on the theory that the standard name would un-skip a broken suite and turn CI red. That theory was wrong: the suite passes once the harness provisions its account, so the workaround is removed rather than left in place. Their guard still hard-fails when the variable is absent and `EXTENDDB_EXPECT_VECTORS=1`, and now names the harness in the message. Verification, run exactly as the rewritten job does (`devtools/run-tests --extenddb --rust-integration --release`): 451 passed, 0 filtered out, 0 SKIP. The single failure is `restore_active_completeness`, which passes 1/1 in isolation, fails the same way under the previous bare `cargo test` invocation, and is unrelated to this change.
The MongoDB backend landed on main while this branch was in flight, so main now carries a third `DataEngine` implementation that predates the vector index contract. Merging without adapting it breaks the build, which is what CI on a15e388 was reporting: ten failing checks, all downstream of four struct-initializer errors and one non-exhaustive match in `extenddb-storage-mongodb`. MongoDB needs no vector code. `DataEngine::as_vector_search` defaults to `None`, so a backend that has never heard of vector search refuses `CreateTable` with vector indexes, `UpdateTable` changing them, and `SearchVectors` by omission. What it needed was to stop assuming the shape of types the contract widened: - `TableDescription` (create and describe paths) and `TableKeyInfo` now take `..Default::default()`, matching what the Postgres backend already does at the same sites and for the same stated reason: a field added for a feature this backend does not implement should not break its build. - The restore path's `CreateTableInput` likewise, mirroring `storage-postgres/src/backup_engine.rs`. - `table_key_info` classified indexes with a local `match` over `IndexType`, which a new variant breaks by construction. It now calls `core::types::partition_indexes`, which exists for this and which Postgres already uses, taking the groups it serves and ignoring the rest. `index_info_from_doc` still rejects any kind other than GSI or LSI, so a vector index cannot reach this path in the first place. Verified: `cargo fmt --check` and `cargo clippy --workspace --all-targets -D warnings` both exit 0, and 859 workspace unit tests pass with 0 filtered out. The MongoDB integration suite needs a Mongo container and was not run locally; CI covers it.
…row-bound
Adds an ignored timing harness for the vector scan, parameterised by item count,
dimensions, payload width and TopK. It is scaffolding for a decision rather than
an assertion, which is why it is `#[ignore]`d and why its module docs carry the
numbers it produced.
It exists because the obvious optimisation turned out to be wrong, and the
reasoning that made it look obvious should not be repeated. The scan selects
`item_data` for every row in the partition and deserialises it before the TopK
heap has decided the candidate is irrelevant. That looked expensive: the stored
item carries the vector as decimal strings, so parsing it per row means parsing
`dimensions` numbers per row, which is far more work than the distance itself.
Measured, at 10,000 items and 384 dimensions with a 2KB non-indexed attribute,
the scan takes roughly 30ms, about 3us per row, and none of that is the work
above:
- Dropping `item_data` from the projection entirely: no change.
- Skipping the blob decode and the distance computation entirely: no change.
34ms doing no per-row work at all, against 25 to 28ms doing all of it.
- Replacing the indexed distance loops with iterator `zip`, on the theory that
a reused buffer defeats auto-vectorisation: no change.
So the cost is per-row overhead in the row-streaming layer, which is shared with
every other scan path in this backend and is not vector-specific. Two changes
were implemented and reverted rather than kept: deferring the item parse until a
candidate can enter the retained set, and scoring straight from the stored bytes
into a reused buffer. Interleaved against unmodified code over six alternating
pairs, that version measured 48.6ms against 29.5ms with no overlap in range. It
was reproducibly slower while doing strictly less work, and this harness did not
isolate why.
A note on method, because the first attempt at this measurement was invalid. A
median of seven repetitions on an eight-core machine at load average 3 could not
discriminate: the unmodified code measured 59.1ms and then 32.7ms on two
consecutive runs of the identical binary, a wider spread than any effect under
investigation. The numbers above come from building both binaries, alternating
between them, and taking the minimum of many repetitions.
…ory on Postgres Two defects in a15e388, both invisible locally because a workflow expression is only evaluated by GitHub. Single braces. Both rust jobs passed `${ steps.init.outputs.admin_password }` rather than `${{ ... }}`, so GitHub never interpolated it and the literal string was sent as the password. devtools/provision-test-credentials then failed with `create account failed: 401 Invalid credentials` before a single test ran, which is what CI reported on d49c844. The two pytest jobs were already correct, so this was introduced by the edit rather than copied from them. Local verification could not catch it: running the harness by hand means exporting the password directly, which is the one thing CI does differently. Missing expectation on Postgres. `EXTENDDB_EXPECT_VECTORS: "0"` was described as present on the Postgres job but never landed, lost when the workflow was reverted after an earlier edit deleted the SQLite job wholesale and redone by line range. Postgres implements no vector search, so it is the only job where the wire refusal tests can execute, and without the expectation that suite could skip every assertion and still report green. Now "0" there and "1" on SQLite, so a skip on either side is a failure rather than a silent pass. Verified: the workflow parses, all six jobs are present, and both rust jobs resolve to the harness invocation with the expectation each one needs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements vector indexes and
SearchVectorson the SQLite backend, making it the first backend to declare the optionalVectorSearchEnginecapability added by the contract #243 this is stacked on.Working end to end:
CreateTablewithVectorIndexes, andUpdateTablecreate/delete with a real backfill of items already in the tableDescribeTablereports the indexes and their statusSearchVectorsforCOSINE,EUCLIDEANandDOT_PRODUCT, withper-metric ordering, partition scoping,
TopK, inline equality filters andProjectionExpressionTransactWriteItemsbranchesCREATINGis dropped and rebuilt at startupStorage layout is one row per vector, chosen on write amplification rather than read speed. A packed per-partition blob reads 2 to 4 times faster but would rewrite the whole blob to insert one vector: roughly 390 MB per
PutItemfor a100k-vector partition at 1024 dimensions. Recorded in the ADR.
The backfill status sequence follows what the service was measured to report, not what seemed reasonable:
CREATINGwithBackfilling: false, thenCREATINGwithtrue, thenACTIVEwith the member absent. Observed on 2026-08-06 by seeding 3000 items of 1024 dimensions so the backfill took 8.5 minutes and could be sampled.falsecomes first, so presence does not imply backfilling and a client must read the value rather than test for the member.Why
The contract PR models the vector index surface but no backend implements it, so no vector index can be created in tree. This makes it real on SQLite, which is the backend that ships in the dev/test image.
It also closes a hole the contract's post-condition guard exposed: this backend declared the capability while ignoring
vector_index_updates, so anUpdateTablecreate returned 200 and created nothing, and a delete returned 200 while the index stayedACTIVEand kept returning hits.Stacked on the vector contract PR. Review that one first.
Closes #
Testing done
Verified against a live SQLite-backed server over HTTP, not in process:
25 vector wire tests, covering nearest-first ordering,
TopK, overwrite, delete, unindexed items, tenant isolation in both directions, partition moves when the HASH attribute changes, the dot-product ordering inversion, backfill of pre-existingitems (including one without a vector that must be skipped rather than break the scan), later writes still indexed, index delete stopping both serving and reporting, base items surviving an index delete, duplicate-create and missing-delete rejection, and the
f32narrowing.Three things were mutation-checked rather than trusted green:
Also verified directly against the database rather than inferred:
Measured scan throughput is roughly 1,000 vectors per 10 ms at 1024 dimensions on one core. Worth stating plainly because an earlier modelled figure of 36,000 was wrong by 10 to 30 times: the cost is the SQLite read path, not the arithmetic, and a zero-copy
&[f32]view measured no faster.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC:
docs/adr/0004-vector-search-exact-scan.md(added here), RFC #236Breaking changes
On-disk format. The SQLite catalog version moves from
0.0.3to the version carrying thevector_indexestable, so an existing deployment must runextenddb migratebefore the server will start. The migration is the idempotent schema apply; it was exercised on a live deployment rather than reasoned about.No wire-protocol or trait changes here: those are all in the contract PR.
Known gaps, stated rather than discovered
gsi_pendingqueue, which already has crash recovery and per-key FIFO. Not in this PR yet.BatchWriteItem, composite base keys, the scoped-indexSearchConditionExpressionrequirement, and wrong-dimension requests.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.