feat(engine): Vector index and SearchVectors contract for storage backends

[LeeroyHannigan]

What this does

Models DynamoDB's vector index surface in extenddb-core and makes it implementable by a storage backend, without implementing it for any. Both in-tree backends still refuse every vector operation, and those refusals are proven over the wire rather than by calling the validator directly.

CreateTable and UpdateTable accept VectorIndexes / VectorIndexUpdates, DescribeTable reports them, SearchVectors is a first-class operation with its own consumed-capacity shape, and item writes validate the vector attribute and the search-schema attributes against the index definition.

Design docs: docs/adr/0004-vector-search-exact-scan.md (added on the stacked implementation branch), RFC #236.

Vector search is an optional capability

This is the first optional feature on a trait surface where every other method is mandatory, so the shape is worth a look:

// A separate trait, with NO default bodies.
pub trait VectorSearchEngine: Send + Sync {
    fn search_vectors(&self, req: VectorSearch<'_>) -> BoxedFuture<'_, VectorSearchResult>;
}

// One accessor on DataEngine, defaulting to None.
fn as_vector_search(&self) -> Option<&dyn VectorSearchEngine> { None }

Declaring support therefore requires handing over an implementation. An earlier revision used a supports_vector_indexes() -> bool, which let a backend return true and never implement the method: an honour system where the type system should be doing the work.

The accessor deliberately sits on DataEngine, not StorageEngine. StorageEngine comes from a blanket impl over the six focused traits, so a defaulted method there could never be overridden by a backend. It would have looked correct and been unoverridable.

StorageError::Unsupported is added so a backend declining a feature is not reported as an internal fault. If we ever want a backend that declines transactions or streams, this is the template: peel the feature into its own trait with no defaults, add one accessor to a trait backends already implement, gate in core so the refusal never reaches storage.

Measured against the live service

Most specifics were measured against real DynamoDB rather than inferred, and in nearly every case the measurement contradicted the inference:

Fact	Inferred	Measured
DistanceFunction order	alphabetical, or declaration order	[DOT_PRODUCT, COSINE, EUCLIDEAN], neither
Inline filter cap	20, from the query-side limit	18
SearchSchema HASH elements	exactly one, required	at most one, and optional
SearchConditionExpression	always required	required only when a HASH is declared
Component range	-	representable as f32; f32::MAX exactly is accepted
Excess decimal precision	rejected	accepted
N output form	may use an exponent	never; the 38-digit limit bounds significant digits
TableThroughputMode	a CreateTable member	not a member at all

That last one mattered: an alias for it made ExtendDB accept a request AWS ignores.

Error messages are pinned by whole-string equality, not by fragment. Fragment assertions that began after the One or more parameter values were invalid prefix hid three divergences: a colon where the service uses a full stop (twice), and one message missing the prefix entirely.

Deliberately absent

Please read these as scope, not omissions:

• No backend implements this, so no vector index can be created in tree yet. The SQLite implementation is a stacked branch and will follow.
• Backfill state and the UpdateTable create/delete paths are modelled but unvalidated, because validating them needs a backend that actually performs a backfill. This is the portion most likely to move, and reviewer time is probably better spent elsewhere.
• Async propagation of index writes is not modelled.
• Errors report the bare field name where the service reports a positional path such as vectorIndexes.1.member.distanceFunction. A serde deserializer for the enum cannot know its position, so closing this means deserialising permissively and validating positionally, as the Projection check already does. Its own change.

Verification

Refusals are asserted over the wire against a live server, using a SigV4 raw-request helper because no SDK version carries the vector types. That suite immediately earned its place: it found that UpdateTable's at-least-one check omitted VectorIndexUpdates, so a request carrying only vector index changes was rejected as empty, contradicting what the live service accepts. The unit tests call the validator directly and never reach that check, so it was invisible to them.

Both vector suites self-skip on the wrong kind of backend, and the expectation is pinned per CI job (EXTENDDB_EXPECT_VECTORS), so a backend that silently lost or gained the capability fails rather than skipping green.

• 427 Rust integration tests, 0 filtered out
• 735 workspace tests, 0 filtered out
• cargo fmt --check clean, clippy --all-targets -D warnings clean on both feature sets

What I would most like reviewed

1. The optional-capability shape, since it becomes the precedent for every future optional feature.
2. Whether the accessor belongs on DataEngine or somewhere better.
3. Wire parity: anything modelled that the service does differently.