[RFC] MongoDB Storage Backend

[diegotoledano95]

What

Adds docs/rfcs/0000-mongodb-backend.md, a draft RFC for adding MongoDB as a optional ExtendDB storage backend.

Why

MongoDB is a natural fit as an additional database target: data model alignment; high read/write throughput through horizontal scalability; infrastructure fit.

DynamoDB and MongoDB share the same data model approach - documents stored as schema-less JSON-like data. MongoDBs document model maps directly to the approach taken by DynamoDB with each item stored as a MongoDB BSON document with no impedance mismatch at the data model level. Unlike relational databases, the translation from JSON to BSON is direct without complicated relational mapping techniques required.

This PR proposes the RFC tracked by the below issue.

Closes #206
Related forked implementation code

Testing done

git diff --check
python docs/build-docs.py

Checklist

• I have read CONTRIBUTING.md
• Code is formatted (cargo fmt --check) (No Rust code was changed)
• I have updated documentation if behavior changed - [x] This PR is the RFC for the proposed MongoDB storage backend

ADR / RFC: This PR

[LeeroyHannigan]

Hi @diegotoledano95,

Thank you for the contribution RFC. The RFC looks great, but I did find some gaps while reviewing the reference implementation. Below are the findings, ranging from critical to minor.

Please let us know if you need any of the below clarified, we'd be happy to help.

Critical Findings

C1. GSI index collection uses GSI keys as document _id, silent data loss on duplicate index keys

Where: data_engine.rs:1215-1234 (sync_indexes / sync_indexes_in_session); _id construction in data/mod.rs:51.

DynamoDB: GSIs (and LSIs) allow multiple items with identical index pk+sk. A Query on the index returns all of them. The index entry is uniquely identified by index keys plus base table keys, the PostgreSQL reference stores base_pk/base_sk_* columns and deletes filter on them (storage-postgres/src/data/index.rs:234-299).

This code: Index documents are built with item_to_document(&projected, &idx_key_schema, ...), so _id and the replace filter derive solely from the index pk/sk. Two base items with the same GSI key produce the same _id; the second replace_one(...upsert) silently overwrites the first.

Failure scenario: Table Orders with GSI on customer_id + order_date. A customer places two orders on the same date -> only the second appears in GSI queries. Worse: deleting the first item afterwards deletes the surviving index entry that now represents the second item (delete_one matches on index keys only), so both vanish from the index.

C2. TransactWriteItems drops stream records and GSI updates entirely

Where: data_engine.rs:1876-1902 (OwnedTransactWriteOp has no stream field), :1904-1957 (clone_transact_write_op discards stream and return_values_on_ccf via .. on every variant), :1618-1865 (execute_transact_write_op_in_session never calls sync_indexes_in_session or write_stream_inline_in_session).

Trait contract: storage/src/lib.rs:313-315: "When stream is Some, stream records for each write operation are inserted in the same transaction as the data writes." PostgreSQL does exactly this (storage-postgres/src/data/transactions.rs:145-168) and syncs GSIs in-transaction (:342-468).

Failure scenario: Any application using TransactWriteItems on a streams-enabled table (the canonical outbox/CDC pattern) gets zero stream records, consumers silently miss every transactional write. Any GSI query after a transactional write returns permanently stale results; there is no async repair queue in this backend.

C3. MongoDB WriteConflict -> InternalServerError (500) under normal contention; TransactionConflictException never produced

Where: every Mongo error maps to StorageError::Internal(e.to_string()), put commit data_engine.rs:445-448, delete commit :560-563, update replace/commit :698-702, 734-737, transact op writes :1685, 1741, 1814, transact commit :1610-1613. No check for TransientTransactionError / UnknownTransactionCommitResult labels or WriteConflict (code 112) anywhere; no retry loop around any transaction.

DynamoDB: two concurrent PutItems on the same key both succeed (last-writer-wins). A write conflicting with a transaction returns retryable TransactionConflictException. The PostgreSQL backend matches via row-level locks (second writer waits, then succeeds).

This code: every single-item PutItem/DeleteItem opens a snapshot multi-document transaction (:340-359, :484-503), so WiredTiger aborts one side of any same-document race with WriteConflict -> HTTP 500. MongoDB's contract is "abort-and-retry"; the backend implements neither retry nor error mapping. Under sustained contention the backend sprays 500s where DynamoDB and the PostgreSQL backend absorb the contention silently.

C4. Sequence numbers assigned outside the transaction -> gap-read: stream records permanently skipped

Where: stream_engine.rs:430-453 (next_sequence_number: find_one_and_update $inc on a counters doc, no session, runs outside any transaction); data_engine.rs:1388-1391 (called from write_stream_inline_in_session without the session; record inserted with .session() but the counter increment is immediately visible while the data transaction commits later).

Failure scenario: Writer A draws seq 5, its transaction commits at T+50ms. Writer B draws seq 6, commits at T+10ms. A consumer polling at T+20ms with sequence_number $gt <cursor> reads seq 6 and advances its iterator past 5 (the engine encodes AFTER_SEQUENCE_NUMBER|<last_seq> in the next iterator, engine/src/streams.rs:236-247). When A commits, seq 5 lands behind the cursor and is never returned. Permanent record loss for the consumer. The update_item retry loop (:630-742) re-draws a fresh sequence number per conflicted attempt, widening the window. PostgreSQL assigns nextval() inside the data transaction (storage-postgres/src/data/tx_helpers.rs:284-289).

C5. UpdateTable GSI creation performs no backfill (and reports ACTIVE immediately)

Where: table_engine.rs:610-653, the create branch inserts only a catalog document with "index_status": "ACTIVE" (:636). No scan of the existing base collection, no index-entry writes, no CREATING->ACTIVE transition.

DynamoDB: adding a GSI backfills all existing items while the index is CREATING, then flips to ACTIVE. PostgreSQL: backfill_gsi(...) inside the DDL transaction (storage-postgres/src/update_table.rs:365-374).

Failure scenario: add a GSI to a table with 1M items, the index reports ACTIVE immediately and permanently contains only items written after the UpdateTable. Queries silently return incomplete results with no error.

C6. UpdateItem old-image not captured in the common case -> stale GSI entries on every index-key change

Where: slow path data_engine.rs:672-677, need_old = return_old || stream.is_some(); when the client didn't request ReturnValues=ALL_OLD and streams are off (the overwhelmingly common case), old_item = None, so sync_indexes_in_session's "delete old index entry" branch (:1295-1300) never runs. Fast path :587-612 is worse: explicitly calls sync_indexes(key_info, None, Some(&item)) (:608), non-transactionally, and it handles native $set/$unset, precisely the updates that change or remove GSI key attributes.

DynamoDB: changing a GSI key attribute removes the entry under the old key and adds one under the new key; removing the attribute removes the entry (sparse).

Failure scenario: UpdateItem SET status = :done on a status-keyed GSI -> the item remains indexed under pending and done forever. REMOVE status leaves it in the GSI forever. PostgreSQL always fetches the old image and deletes the old index row in the same transaction (storage-postgres/src/data/index.rs:138-180).

C7. Shard IDs collide across accounts and table re-creations, cross-tenant stream record leakage (SECURITY)

Where: stream_engine.rs:42 (shard_id = format!("shardId-{table_name}-{i:012}"), embeds table name, not table_id); lib.rs:231 (single shared extenddb_data database for all accounts); stream_engine.rs:114-121, 505-515 (get_stream_records / latest_sequence_number filter only by shard_id); table_engine.rs:415-470 (delete_table_impl never deletes stream_shards/stream_records rows); stream_engine.rs:44-52 (no unique index on stream_shards.shard_id; plain insert_one).

Confirmed exploitable end-to-end: engine/src/streams.rs::handle_get_records calls get_stream_records(shard_id, …) with no account_id and no validate_shard, only GetShardIterator validates ownership, and the attacker never needs to call it.

PostgreSQL baseline: one data database per account (extenddb_account_<id>), shard_id PRIMARY KEY with table_id REFERENCES tables ON DELETE CASCADE and cascading stream_records (storage-postgres/migrations/001_schema.sql:85-107).

Failure scenarios:

• Account A and Account B both create table users with streams -> both get shard shardId-users-000000000000 in the shared stream_records collection. Account B's consumer receives Account A's items (keys + full images). Cross-tenant data disclosure.
• Delete and recreate a table with the same name -> the new stream replays the deleted table's history (records are never cleaned, see M11), and init_stream_shards inserts duplicate shard docs so DescribeStream returns duplicated shards.

C8. UpdateItem that creates an item emits MODIFY with a fabricated OldImage instead of INSERT

Where: data_engine.rs:648-679, when existing_doc is None, existing_item = key.clone(); with streams enabled old_item = Some(<key attributes>) even though no item existed. write_stream_inline_in_session (:1346-1351) then matches (Some(_), Some(_)) => Modify and populates OldImage with a phantom key-only item for OLD_IMAGE/NEW_AND_OLD_IMAGES view types (:1367-1370).

DynamoDB: UpdateItem on a non-existent key is an insert -> eventName: INSERT, no OldImage. PostgreSQL is correct here (storage-postgres/src/data/update_item.rs:89-94).

Failure scenario: consumers that branch on eventName ("on INSERT, send welcome email") never fire for upsert-created items; OLD_IMAGE consumers receive an old image for an item that never existed.

C9. _id = "{pk}#{sk}" composite is collision-prone, two distinct items cannot coexist

Where: data/mod.rs:51 (doc.insert("_id", format!("{pk_text}#{sk_text}"))); pk_to_text (storage/src/util/key.rs:77-86) does not escape #, only multi-HASH-attribute composites get netstring encoding.

Failure scenario: item A {pk:"a#b", sk:"c"} and item B {pk:"a", sk:"b#c"} both produce _id = "a#b#c". The second insert fails E11000, which put_item_impl:401 misclassifies as ConditionalCheckFailedException, or an unconditional replace silently targets the wrong document. Two legitimately distinct DynamoDB items cannot both exist. Scan pagination (_id $gt start_id, data_engine.rs:964-968) is ambiguous across the same collision. (Note: N-value normalization in core means "1" vs "1.0" is not a collision source, the # delimiter is the real problem.)

C10. Numbers with >34 significant digits: Decimal128 rounding / f64 fallback collides distinct keys

Where: data/mod.rs:62-79 (item_to_document), :152-164 (pk_filter), data_engine.rs:2032-2043 (sk_to_bson).

DynamoDB: 38 significant digits, all values distinct. Decimal128 holds 34; the fallback stores an f64 (~15-17 digits), and sk_n then mixes BSON double and Decimal128 across documents.

Failure scenario: two sort keys differing only in digits 35-38 map to the same sk_n -> E11000 on the unique (pk, sk_n) index, or pk_filter equality matches the wrong document, GetItem returns / PutItem replaces a different item. PostgreSQL uses exact BigDecimal.

(Verification status: spec-grounded, Decimal128 = 34 sig digits by IEEE-754 decimal128; DynamoDB Number = 38 sig digits. Live us-east-1 confirmation with a 39-digit sort key available on request.)

C11. Idempotency tokens are not account-scoped in the shared data DB, cross-account ClientRequestToken collision (ADDED)

Where: data_engine.rs:1549-1583 (token check/insert: find_one({token}) / insert_one({token, fingerprint, created_at}), no account_id); the shared compute_fingerprint (HMAC over TransactItems) also omits account; lib.rs:231 (single shared extenddb_data database for all accounts). Distinct from M4 (which is the missing unique index / concurrent double-execution); this is the cross-account collision on the same collection.

DynamoDB: ClientRequestToken idempotency is scoped per-account (+region); two accounts may reuse the same token string independently.

PostgreSQL baseline: per-account data database (extenddb_account_<id>) inherently scopes tokens; the canonical fix keys idempotency on (account_id, token) (SQLite commit 2738c4e; PR #208 does the same for Postgres, still open).

Failure scenario: Account B issues a TransactWriteItems reusing a ClientRequestToken string that Account A used within the 10-minute window -> B's write is silently deduped as a replay (or rejected with IdempotentParameterMismatchException if the fingerprint differs). Cross-tenant correctness violation. Fix: key idempotency_tokens on (account_id, token) and include account_id in the fingerprint (same shape as C7's per-account data-layout decision).

Major Findings

M1. ReturnValuesOnConditionCheckFailure ignored inside transactions

Where: data_engine.rs:1667-1671 (Put), :1731-1735 (Delete), :1792-1796 (Update), :1856-1860 (ConditionCheck), all hardcode condition_check_failed_with_item(None); the field is dropped in clone_transact_write_op.

DynamoDB: with ALL_OLD, CancellationReasons[i].Item contains the existing item. PostgreSQL implements this (storage-postgres/src/data/transactions.rs:505-528). Per-item reason ordering and ConditionalCheckFailed-at-the-right-index are otherwise correct.

M2. UpdateItem OCC retry loop guards a near-impossible failure; real conflicts are not retried

Where: data_engine.rs:630-747. The read, condition eval, and versioned replace_one all execute inside one snapshot transaction, so the _v mismatch path (matched_count == 0, the only retried path, :704-710) is nearly dead code. Actual concurrent conflicts surface as WriteConflict on replace_one or commit_transaction, mapped to Internal and returned immediately, the 50-attempt loop never engages. Concurrent-create upserts (:711-719) both take the insert branch; the loser gets a 500. Exhaustion returns Internal("too many version conflicts") -> 500, an error DynamoDB has no analogue for.

Failure scenario: an ADD requests :one rate-limiter at ~50 concurrent updates/sec on one key returns a stream of 500s; DynamoDB and PostgreSQL (lock waits) succeed. The native $inc fast path explicitly gives up on numeric ADD (:1074-1094), so every ADD takes the conflict-prone transaction path.

M3. Conditional-insert race: E11000 -> ConditionalCheckFailed mapping is dead code inside a transaction

Where: data_engine.rs:399-413. The insert_one runs inside a snapshot transaction: a competing insert that committed after the snapshot manifests as WriteConflict (code 112), not E11000, the E11000 branch (:401, also fragile string matching on e.to_string()) almost never fires; the race lands in the Internal arm -> 500.

Failure scenario: two clients race PutItem with attribute_not_exists(pk). DynamoDB: one OK, one ConditionalCheckFailedException. Here: one OK, one InternalServerError. Static (non-racing) condition combinations were verified correct.

M4. No unique index on idempotency token -> duplicate ClientRequestToken can execute twice

Where: token check/insert data_engine.rs:1549-1583; bootstrapper.rs:84-106 creates only a TTL index on created_at. Two concurrent requests with the same token start snapshot transactions; neither sees the other's uncommitted token doc; both insert (no unique constraint) and both execute. PostgreSQL avoids this with ON CONFLICT (token) (storage-postgres/src/data/tx_helpers.rs:338-368). (See also C11: even with a unique index, the key must be (account_id, token).)

Failure scenario: double-applied ADD inside a TransactWriteItems Update, violates DynamoDB's exactly-once guarantee for ClientRequestToken.

M5. Binary sort key ordering diverges from DynamoDB

Where: data/mod.rs:82-90 (sk_b stored as BSON Binary), data_engine.rs:820-836 (native range filters/sort). MongoDB compares BinData by length first, then subtype, then bytes; DynamoDB orders binary by unsigned lexicographic byte order. Example: DynamoDB says [0x01,0xFF] < [0x02]; MongoDB says [0x02] < [0x01,0xFF]. The code patches begins_with with a post-fetch filter (:881-904) but not range conditions (BETWEEN, <, >) or sort order.

Failure scenario: any table with Binary sort keys returns Query results in the wrong order and wrong result sets for range conditions. (Verification status: spec-grounded, BSON BinData comparison order is length->subtype->bytes; DynamoDB binary is unsigned byte order. Live confirmation available on request.)

M6. Condition-pushdown compiler is dead code with multiple latent correctness bugs; RFC describes it as the design

Where: condition.rs, condition_to_filter is imported at data_engine.rs:27 but never called. Every runtime condition path (put :373, delete :516, update :660, all four transact ops) does read-then-extenddb_core::expression::evaluate_condition inside a transaction, matching the PostgreSQL baseline. #![allow(unused)] at lib.rs:10 suppresses the warning that would have exposed this.

The RFC's headline claim, "condition expressions compiled to MongoDB filters and executed via atomic findOneAndReplace, no separate fetch, no race window", does not describe the implementation as built. Worse, the compiler contains latent bugs that go live the moment someone wires it up:

• Numeric comparisons compile to BSON string comparisons (condition.rs:61-73): {"item_data.price.N": {"$lt": "100"}} is lexicographic, "9" > "10", "2" > "100", negative ordering inverted. The unit test at condition.rs:548 enshrines the bug. BETWEEN and IN inherit it.
• Eq/Ne on sets, lists, maps compile to Bson::Null (:71), matches nothing where DynamoDB does deep equality.
• size() in a comparison is rejected (:359-366, :115) with a ValidationException where DynamoDB evaluates it.
• IN with a mixed-type list uses the first literal's type for all entries (:415-427): a IN (:s, :n) can match across types where DynamoDB says S"7" ≠ N"7".
• begins_with on Binary rejected (:279-281); DynamoDB accepts B operands.
• attribute_type type-name injected into the field path unvalidated (:346-356).

Either wire the compiler up after fixing these (with $expr/$toDecimal or typed storage for numerics), or delete the module and correct the RFC text.

M7. Parallel Scan can silently drop items

Where: data_engine.rs:974-1028, fetch window capped at (limit+1) * total_segments, CRC32 segment filter applied post-fetch, last_evaluated_key = None whenever fewer than limit+1 items survive. With key-hash skew, a segment's items beyond the fetch window are silently dropped and the scan terminates early. DynamoDB guarantees every item is returned by exactly one segment. (The RFC discloses full-collection-scan-per-segment as a performance tradeoff, but not this correctness gap.)

M8. gsi_cache incoherent across processes, indexes silently never maintained

Where: lib.rs:213 (per-process DashMap<table_id, bool>); negative-cache early return data_engine.rs:1165-1169, 1252-1256; populated on any write (:1239, :1327); invalidated only in-process on UpdateTable (table_engine.rs:652, 669). If process A cached false and process B adds a GSI, process A skips index maintenance for every write until restart. Combined with C5 (no backfill), these gaps are unrecoverable.

M9. Query pagination clobbers the sort-key range condition (all queries, not just index queries)

Where: data_engine.rs:810-835, build_sk_filter inserts e.g. {sk_n: {$gte: 5, $lte: 10}} into the filter (:813-816), then the ExclusiveStartKey block does filter.insert(sk_f, doc! { "$gt": sk_bson }) (:829/:831). Document::insert replaces the existing entry under the same key.

Failure scenario: Query sk BETWEEN 5 AND 10, Limit 2 -> page 1 correct (LEK at sk=7); page 2's filter is just sk > 7, the <= 10 bound is gone -> returns items 11, 12, 15... outside the requested range. Same for begins_with (the upper prefix bound is lost -> page 2 escapes the prefix) and backward pagination ($lt replaces the lower bound). For index queries the resume additionally discards the base-key half of the ESK that DynamoDB requires to disambiguate duplicate index keys (masked today by C1). The engine layer builds the combined base+index LEK correctly (engine/src/query.rs:397-407); the storage resume logic discards it.

M10. No index-key type validation on write

Where: data_engine.rs:2089-2093 (item_has_index_keys checks contains_key only); the backend never calls extenddb_core::validation::validate_index_keys (PostgreSQL does, storage-postgres/src/data/put_item.rs:44-57).

Failure scenario: an item whose GSI key attribute has the wrong scalar type produces a malformed index document (typed sk_* field silently skipped, data/mod.rs:61-79) that later deletes can't match, another stale-entry source; non-scalar values surface as InternalServerError mid-write. DynamoDB returns a clean ValidationException up front.

M11. 24-hour stream retention never enforced, cleanup worker never spawned

Where: lib.rs:124-130, spawn_workers spawns only the TTL worker. cleanup_expired_stream_records (stream_engine.rs:365-380) has no caller; no Mongo TTL index on stream_records.created_at (the bootstrapper does create one for idempotency_tokens, so the omission is streams-specific). PostgreSQL spawns an hourly cleanup with RETENTION_HOURS = 24 (storage-postgres/src/workers.rs:92-118).

Failure scenario: stream_records grows without bound; TRIM_HORIZON replays the table's entire lifetime instead of ≤24h; combined with C7, deleted tables' records persist forever.

M12. UpdateTable stream enable is not idempotent, duplicate shards and broken stream ARNs

Where: table_engine.rs:578-596, whenever stream_enabled == true, unconditionally generates a new stream_label and re-runs init_stream_shards; no existing-shards check, no unique index. PostgreSQL checks for existing shards first (storage-postgres/src/update_table.rs:149-201). DynamoDB rejects enabling an already-enabled stream with ValidationException.

Failure scenario: a redundant UpdateTable with StreamEnabled: true (idempotent IaC re-apply) (a) inserts 4 duplicate shard docs -> DescribeStream returns 8 shards; consumers enumerate shards and process every record twice; (b) rewrites stream_label, so the previously issued stream ARN stops resolving, in-flight consumers get ResourceNotFoundException mid-stream.

M13. Scan on a GSI builds the pagination cursor from the base-table key schema

Where: data_engine.rs:936-969, the collection is the index collection (:940), but composite_pk_to_text(start_key, &key_info.key_schema) and sk_info(&key_info.key_schema, ...) (:950-952) use the base table schema, while index documents' _id is index-pk#index-sk. The resume filter compares against the wrong _id shape -> paginated index Scans skip or repeat arbitrary ranges. LEK extraction (:1036-1038) likewise uses the base schema on index items.

M14. Binary begins_with post-filter drops matches and loses pagination

Where: data_engine.rs:2010-2016 returns an empty filter for binary begins_with (fetches the whole partition), fetch capped at limit+1 (:847), items retain-ed by prefix post-fetch (:890-901), LEK computed only if items.len() > limit (:907-917).

Failure scenario: a partition with 1000 items where 50 match the binary prefix but none appear in the first limit+1 docs -> returns 0 items with no LastEvaluatedKey -> the 50 matching items are silently unreachable. DynamoDB computes LEK from the last item read, never dropping matches.

M15. UpdateItem fast path bypasses version control, concurrent CAS update can silently discard its write

Where: data_engine.rs:587-612, the native $set/$unset fast path doesn't bump _v, so a concurrent slow-path update (:694-709) can pass its version check against the pre-fast-path version and overwrite the fast path's committed write. Lost update where DynamoDB serializes.

Minor Findings

• m1. Unconditional single-item writes needlessly wrapped in snapshot multi-document transactions (data_engine.rs:340-359, 484-503), the direct cause of C3's contention 500s and added latency; a plain find_one_and_replace with majority write concern would match DynamoDB last-writer-wins in the no-condition/no-GSI/no-stream case.
• m2. Decimal128 sort keys: 34 significant digits vs DynamoDB's 38; keys with 35-38 sig digits lose precision (collide or misorder). The f64 fallback (data/mod.rs:68-77) mixes BSON double and Decimal128 in one field.
• m3. A user-supplied connection string containing readPreference=secondaryPreferred would silently break ConsistentRead=true; nothing validates/strips it (lib.rs:216-241). Read-your-writes otherwise holds (primary reads, majority writes).
• m4. Idempotency window is ~10-11 min (Mongo TTL monitor runs every 60s) and there is no created_at check on read; PostgreSQL explicitly allows token reuse after 10 min.
• m5. Single global sequence counter ({"_id": "stream_seq"}) shared across all shards/tables/accounts, cluster-wide write hotspot; get_i64("value").unwrap_or(1) silently resets sequencing to 1 on a type mismatch instead of erroring (stream_engine.rs:450).
• m6. No index (or uniqueness) on stream_records (shard_id, sequence_number), GetRecords is a sorted full-collection scan that degrades as records accumulate (unbounded per M11). Zero-padded {:021} formatting is safe for i64, for the record.
• m7. GSI/LSI data collections never get MongoDB indexes (every index query is a collection scan; the "simple" collation in query_impl:849-857 couldn't use one anyway) and are leaked on DeleteTable / UpdateTable-delete (table_engine.rs:443-455, 655-670).
• m8. Stream label uses nanosecond ISO8601 with Z (table_engine.rs:151-162) vs PostgreSQL's second precision and DynamoDB's millisecond-no-zone format, may break clients that parse the label out of the ARN.
• m9. No-op UpdateItem still emits a MODIFY record, but the PostgreSQL baseline does the same, so this is a pre-existing project-level divergence from DynamoDB, not a Mongo regression.
• m10. TransactGetItems is correct (snapshot session, op order, None for missing); it's more permissive than DynamoDB only in that snapshot reads never conflict, benign.
• m11. Dead code: non-session write_stream_inline (data_engine.rs:231-322) and the non-transactional StreamEngine::write_stream_record (stream_engine.rs:59-101) are unused within the crate, masked by #![allow(unused)] (lib.rs:10), should be deleted to prevent future misuse; the allow(unused) itself hides real issues (M6).
• m12. String begins_with upper bound uses prefix + char::MAX with $lt (data_engine.rs:2064-2069), wrongly excluding stored keys of exactly prefix + '\u{10FFFF}'. Pathological but real.
• m13. <> on a missing attribute evaluates TRUE in the core evaluator (evaluator.rs:41), and Mongo $ne matches missing fields the same way, backend and PostgreSQL baseline are mutually consistent, but real DynamoDB evaluates all comparators (including <>) to false on a missing attribute (hence the canonical attribute_not_exists(a) OR a <> :v pattern). Project-level divergence, not a Mongo regression; deserves an integration test against real DynamoDB.

Verified Correct

• Plugin registration: four inventory::submit! blocks; no changes to engine/server/auth/core crates; cmd_serve backend gate generalized cleanly.
• ConsistentRead=true on GSI queries is rejected with the correct ValidationException at the engine layer (engine/src/query.rs:64-73), despite the RFC's misleading "GSI reads are strongly consistent" claim, there is no runtime divergence here. LSI consistent reads pass through, matching DynamoDB.
• Sparse-index omission for items missing GSI key attributes.
• Index projections (ALL / KEYS_ONLY / INCLUDE) at write time, always including base table keys.
• LSIs structurally cannot be added post-creation (matches DynamoDB).
• Stream image population per view type (INSERT no OldImage, REMOVE no NewImage, MODIFY both; gated by KEYS_ONLY/NEW_IMAGE/OLD_IMAGE/NEW_AND_OLD_IMAGES), modulo C8's phantom old image.
• Per-key shard affinity: CRC32 of the partition key only, so same pk / different sk lands on the same shard.
• TTL worker: deletes route through delete_item with a condition re-check and a StreamCapture carrying UserIdentity {Service, dynamodb.amazonaws.com} -> REMOVE records written in-transaction. Matches PostgreSQL and DynamoDB.
• put/delete/update (non-fast-path) write the stream record and GSI updates on the same ClientSession transaction as the data write, with snapshot read concern + majority write concern.
• Number normalization: validate_and_normalize_number in core canonicalizes N values before storage, so "1" vs "1.0" key-identity is handled project-wide.
• Shard iterator semantics (TRIM_HORIZON / LATEST / AT / AFTER, 15-min expiry) live in the shared engine layer, identical for both backends.
• FilterExpression / Limit ordering: the storage trait's query/scan take no filter, the engine applies FilterExpression post-read (engine/query.rs:430, engine/scan.rs:381) with Limit applied to items read and LEK from the last read item, matching DynamoDB (except where M7/M9/M13/M14 corrupt the LEK).
• String sort-key collation: the (pk, sk_s) index and Query both use locale "simple" (table_engine.rs:212-215, data_engine.rs:849-857), MongoDB simple collation is code-point comparison, which for UTF-8 matches DynamoDB's byte ordering.
• Cross-type comparisons in live condition evaluation (e.g. filter on a.N vs stored a.S) correctly don't match, since the typed path is absent.

[diegotoledano95]

The fixes for the gaps detailed in the comment above have been done and are ready for review. You can find them in the Related forked implementation code link in the PR description.

The RFC presented in this PR has also been changed to reflect those changes.

The Related forked implementation code link has also been updated in the Github issue related to this PR.

[LeeroyHannigan]

Thanks for the substantial revision.

All 11 Critical and 15 Major findings from the first pass are addressed. The core data-plane design is sound: transactional GSI + stream propagation, the netstring composite _id, in-transaction stream sequencing, and WriteConflict retry all hold up.

Conformance (verified live)

Suite	Result
Python pytest	844 / 844
Python comprehensive	330 / 330
Rust integration	382 / 384
Total	1,556 passing, 2 failing

Blocking

1. Table CREATING state is not implemented. table_engine.rs:212 writes table_status: "ACTIVE" unconditionally, and control_plane_delay_seconds appears nowhere in the crate (Postgres uses it in four files). This fails two existing conformance tests that pass on Postgres: put_item_on_creating_table_returns_not_found
(PutItem wrongly succeeds) and restore_table_from_backup. GSI-level CREATING is implemented correctly; only table status is missing.

2. Binary begins_with returns wrong results. data_engine.rs:3283-3286 derives the range upper bound by incrementing the prefix bytes then hex-encoding, which is not the next prefix in fixed-width hex space. Reproduced at the wire level:

• begins_with([0xFF]) → returns nothing (all matches dropped)
• begins_with([0x2F, 0xFF]) → also returns the unrelated key [0x30] (false positive)

The string path already handles this correctly via next_string_prefix; applying the same logic in hex space fixes it.

3. Field-vs-field conditions evaluate backwards. pushdown.rs:179 admits Field <op> Field for all types, and condition.rs:181 then compares untyped tagged subdocuments, so Numbers compare lexically. Reproduced both directions:

• "counter_a < counter_b" with a=42, b=9 → write allowed (should reject)
• same expression with a=9, b=42 → write rejected (should allow)

Fix: mark Field vs Field non-pushable and fall back to the in-Rust evaluator, consistent with the existing N/B literal exclusions.

4. Rebase required The branch is based on 3ad1dcc; main is now ecc69e3. Our commit db0baba added account_id to Storage::get_stream_records after you forked, so the crate will not compile after rebase. You'll also need the account-ownership guard the trait now expects, reference implementation at storage postgres/src/stream_engine.rs:133+. Apologies for the moving target.

5. Steering violations in devtools/run-mongodb-tests. :214 starts the server with the & background operator; :109 uses kill instead of extenddb stop.

Non-blocking (worth tracking at merge)

• 35–38-digit numeric sort keys are rejected (Decimal128 caps at 34); Postgres accepts them, same key, different answer per backend.
• Multi-node GSI cache staleness can drop index entries during backfill (bounded ~60s; benign single-node).
• Per-index MongoDB collections are never dropped on DeleteTable/index-delete, orphans accumulate.
• BETWEEN bound validation compares via f64, so an inverted range with tiny low-order differences escapes validation.
• Stream writes read the catalog DB inside the data DB session, valid only when both share one deployment; please assert and document.
• The shared StorageError enum gained a TransactionConflict variant, additive and reasonable, but it's a shared type, so flag it for an owner decision.
• Doc inconsistencies: min version 7.0 (RFC) vs 6.0+ (design), neither enforced; the design describes a single shared client, the code builds six.
• differences-from-dynamodb.md isn't updated, notably "GSI reads are strongly consistent" (stronger than DynamoDB) and the replica-set requirement.

[diegotoledano95]

@LeeroyHannigan Acknowledging the feedback, will start working on the blocking issues. Quick question, do you want to continue reviewing the code as we have on the forked branch?

Or do you want to start adding the code here in the current PR or a new PR?

[LeeroyHannigan]

Thanks @diegotoledano95

Would be great to get it here, along with your intended CI. #218 does change how backends register, so if you want to wait until we merge that in, make those changes on your fork and then push here, might be cleanest.

[diegotoledano95]

@LeeroyHannigan Will do thanks! Would you have an ETA on #218 ?

[LeeroyHannigan]

@diegotoledano95 #218 has just landed. That should unblock you.

[diegotoledano95]

@LeeroyHannigan Have pushed the changes for the blocking issues above, and put the code in this branch and PR as requested too. Please let me know what you think, thanks!

[LeeroyHannigan]

Thanks @diegotoledano95 for turning these around quickly, and for the mutation-checked tests that came with them. I rebuilt from d9fe4da and re-ran everything against MongoDB 7 through devtools/run-mongodb-tests. Every number below came out of a log file with a captured exit code.

Gates

• cargo fmt --all -- --check: fails (exit 1). 13 diffs, all in crates/storage-mongodb/src/ (catalog_store.rs:38, data/mod.rs:5,:14, data_engine.rs:21,:1709,:2284,...). This is what CI run #429 reports. Mechanical: one cargo fmt --all clears it.
• cargo clippy --workspace --all-targets -- -D warnings: clean (exit 0).
• cargo test --workspace --lib: 691 passed, 0 failed (exit 0).
• cargo build --release: clean (exit 0).
• Rust integration vs MongoDB 7: running 413 tests ... 412 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 370.04s. The single failure is a new test I added for the restore issue below; all 412 pre-existing tests pass.
• Python suite: 890 passed, 7 failed, 24 errors. Every failure and error is in the console tests and traces to CERTIFICATE_VERIFY_FAILED, which is a self-signed-cert artifact in my runner rather than anything in this PR. No
  data-plane, auth, or isolation failures.
• clippy -W clippy::pedantic: the crate is net-new, so it adds roughly 268 pedantic warnings against a baseline of 0. Mostly missing backticks (94) and missing # Errors doc sections (40), plus about 40 lossy or wrapping casts worth a look. Not a blocker, just a heads-up if you ever turn pedantic on.

Confirmed fixed, and re-proven live with the two wire-level tests from last round:

• Binary begins_with: the hex-string-space bound is correct. binary_sk_to_hex is fixed-width, two hex chars per byte, and order-preserving, and the bound uses the same space as stored keys, so hex-lexicographic ordering equals bytewise-unsigned ordering. Hand-checked the empty prefix, [0x2f,0xff], [0xff], and all-0xFF.
• Field-vs-field pushdown: (Field, Field, _) => No, no other arm leaks a Yes, the catch-all is No, and the new test spans all six operators and fails if the fix is reverted.
• CreateTable CREATING state: CREATING plus status_transition_at, data-plane ops on a CREATING table return ResourceNotFoundException, and the transition worker is idempotent across restarts.
• run-mongodb-tests lifecycle: self-daemonizing serve, extenddb stop, isolated run_dir. Works.

One functional bug I would like fixed before merge

Restore reports ACTIVE before the data copy finishes (crates/storage-mongodb/src/backup_engine.rs).

restore_table_from_backup calls create_table at :488, which is what schedules the CREATING to ACTIVE transition, and only then starts the $out copy at :505-513. The transition is a wall-clock timer, not a callback on the
copy. In table_engine.rs:211-217 the transition time is computed as now + control_plane_delay_seconds, and the worker in worker_store.rs:66-69 flips any table whose status_transition_at has simply elapsed:

"table_status": "CREATING",
"status_transition_at": { "$lte": now },

with a 250 ms poll cadence. Nothing in that path references the copy, so ACTIVE does not imply the restore is complete.

The comment at :521-527 states the opposite, and I think this is the crux of it: it reasons that "the data was just copied above, so it is in place before the table becomes ACTIVE." That holds only if the copy finishes inside the
delay window. It is an assumption about timing, not an ordering the code enforces.

Empirically it is load-dependent, which is what makes it easy to miss:

• Under a busy mongo, a 40,000-item restore reports ACTIVE with 0 items readable, 10 out of 10 consecutive runs (96 to 107 seconds each). The $out copy takes tens of seconds while the flip fires roughly 50 ms after create_table.
• On an idle mongo the same test passes, because $out finishes inside the window. Those passes are real, not short-circuits: the seed loop is unconditional and the assertion is a live Select::Count scan, so reaching ok proves all 40,000 items were seeded, backed up, and restored.

So a green run here is evidence of a lucky schedule rather than of correctness, and the existing conformance test cannot catch it because a tiny backup always finishes inside the window. A client doing wait-for-ACTIVE against a production-sized restore reads an empty table.

Suggested fix: set ACTIVE, or schedule the transition, only after $out drains. I have the reproducing test (a concurrent observer that counts at first-ACTIVE) and am happy to contribute it.

Two gaps worth closing in the same pass

• Nothing in CI exercises this backend. Every integration job runs Postgres, and devtools/run-mongodb-tests is never invoked by a workflow, so the backend has compile-only coverage in CI and none of the fixes above are protected from regression.
• No mongo-specific Rust integration tests. tests/rust/ gained no cases for the behaviours fixed here. The restore test above is a natural first one.

One related note if you do wire the harness into CI: run-mongodb-tests accepts --mongo-port but hardcodes CONTAINER_NAME="extenddb-runtests-mongo" at :76 and runs an unconditional docker rm -f "$CONTAINER_NAME" at :118. Two concurrent invocations therefore destroy each other even on different ports, and they also share an output path. I hit this while reviewing: one run tore down another run's mongo mid-test and overwrote its log. In CI that will present as unexplained flakes the moment two jobs land on one runner. Deriving the container name and output dir from the port or a run id would fix it.

Smaller items

• Missing encryption key is handled differently to Postgres, and it can panic. lib.rs:117 loads the key with .unwrap_or_default(), so a missing encryption_key setting silently becomes an empty string, which is then passed to both the catalog store and the credential store. encrypt_secret base64-decodes it to zero bytes and calls aes_gcm::Key::<Aes256Gcm>::from_slice, which panics when the length is not 32. Postgres handles the same case at storage-postgres/src/lib.rs:527-534 by returning BackendError::MissingEncryptionKey and refusing to start. That error variant already exists, so this looks like an oversight rather than a deliberate divergence.
• MongoStorageConfig derives Serialize over connection_string (config.rs:9-12), so user:pass@ can leave the process on any serialize path. PostgresStorageConfig derives only Debug, Clone, Deserialize. The Debug derive matches the Postgres precedent so I am not flagging that half, but the added Serialize is new and the repo already has config::redact_password for this.
• The connection-string guards only cover one of five clients. The readPreference=primarycheck is a hard error, and the TLS check a warning, but both live inMongoEngine::newand apply only to the client built withwith_options. The catalog, auth, and two further clients are built with with_uri_str (lib.rs:102,:127,:169,:185`) and bypass them. The guard's own message says non-primary read preferences "silently break ConsistentRead=true", which would apply to catalog metadata and credential reads served from a lagging secondary.
• GSI backfill can leave an index in CREATING. On the done=false with last_id=None path (ttl_worker.rs:178-183) the early return correctly avoids an infinite loop, but it returns without transitioning the index, so the index stays in CREATING with nothing scheduled to finish it. The comment says it "shouldn't happen in practice", which is probably right, though a warn log or an explicit failure state would make it diagnosable if it does.
• Sort-key BETWEEN inversion detection parses both bounds as f64 (data_engine.rs:3321-3325), so bounds that differ beyond f64 precision are not detected as inverted. DynamoDB numbers are 38 significant digits.
• begins_with tests could use an empty-prefix case and a multi-byte all-0xFF case, and it is worth documenting that an empty binary prefix matches everything.
• Docs drift: the design doc says v6.0 while the RFC says 7.0, and the CI and live-test claims in the RFC do not match the repo yet.
• The CREATING commit message says DeleteTable has no DELETING state, but the code sets and returns DELETING. The behaviour is right, the message is not.
• Backup lookup in restore_table_from_backup (:415) has no account predicate of its own. The handler does enforce ARN ownership upstream (engine/src/backup.rs:24-47, returning AccessDeniedException on mismatch, with tests), so this is not exploitable. Worth adding for defence in depth since describe and delete are scoped at the storage layer, but purely optional.

The two wire-proven bugs from last round are properly fixed, the CREATING modelling is sound for CreateTable, and the full integration suite is green apart from the restore case. Requesting changes on the restore race and the fmt gate, with the CI and test-coverage gaps strongly recommended alongside.

[diegotoledano95]

@LeeroyHannigan Thanks for the thorough second pass — the load-dependent restore repro in particular was exactly the kind of thing a green run hides. Rebased onto latest main and pushed. Point by point:

Gates

• fmt: fixed — cargo fmt --all -- --check is now clean (commit "style(mongodb): satisfy cargo fmt check"), verified with the exact workspace command CI runs.
• pedantic (~268): left as-is, intentionally. The crate is clean under our actual gate, cargo clippy --all-targets -- -D warnings (0 warnings). pedantic/nursery aren't enabled anywhere in the workspace, so turning them on for just this crate would make it inconsistent with core/engine/storage postgres rather than more consistent. Happy to open a separate issue for a workspace-wide pedantic pass if the team wants that direction.

Functional bug — restore reports ACTIVE before the copy finishes

Fixed (commit "set restored table ACTIVE only after the data copy completes"). Restore no longer relies on the wall-clock transition: create_table is called with the transition deferred, the $out copy runs, and only then is the row set ACTIVE directly (no timer). ACTIVE now implies the copy has drained, by ordering rather than by timing assumption. The misleading comment is gone, and your reproducing scenario is covered by the committed test "cover restore reporting ACTIVE before the data copy completes" — thanks for offering it; the tree includes an equivalent concurrent-observer test.

Gap — nothing in CI exercises the backend

Added .github/workflows/integration-mongodb.yml (commit "ci(mongodb): add MongoDB integration workflow"). It mirrors integration.yml: a pytest job and a rust-integration job run in parallel, each building with --features mongodb and delegating to devtools/run-mongodb-tests, joined by a gate job. The orchestrator does the replica-set bootstrap (rs.initiate + wait-for-PRIMARY) that GitHub services: can't express, so CI runs the exact path used locally.

Gap — no mongo-specific Rust integration tests

Added dual-target tests/rust/ cases (commits "binary begins_with edges and field-vs-field conditions" and "use a multi-byte all-0xFF begins_with prefix"): binary begins_with by unsigned byte prefix, empty-prefix (whole partition), multi-byte all-0xFF, and field-vs-field condition comparisons. These run against Postgres and Mongo, so the fixes are now regression-protected on both backends.

run-mongodb-tests container/output collision

Fixed (commit "isolate run-mongodb-tests container and output per run"): the container name and output dir now derive from the mongo port (plus PID for the dir), so concurrent invocations no longer tear each other down or share a log.

Smaller items

• Missing encryption key panic: fixed — the key now loads with .ok_or(BackendError::MissingEncryptionKey)? instead of .unwrap_or_default(), refusing to start like Postgres does (commit "harden connection and credential handling").

• MongoStorageConfig derives Serialize: removed; it now derives only Debug, Clone, Deserialize, matching PostgresStorageConfig (commit "stop deriving Serialize on MongoStorageConfig").

• Guards cover only one of five clients: fixed — all client-construction sites now go through a shared connect_guarded helper that applies the read-preference hard error to every client and the no-TLS warning to the server data client (same commit as the key fix).

• GSI backfill can leave an index in CREATING: added a warn on the done=false/last_id=None path so a stuck index is diagnosable (commit "warn when GSI backfill returns an empty, non-final batch").

• BETWEEN f64 inversion: documented rather than reworked. The comparison is monotonic under f64 rounding, so a valid range is never wrongly rejected; the only gap is a genuinely inverted range distinguishable only beyond ~15–17 significant digits, which returns an empty result instead of a ValidationException. Spelled out in the code comment and differences-from-dynamodb.md (commit "document f64-precision bound on inverted BETWEEN keys"). Glad to switch to exact decimal comparison if you'd prefer the fix over the note.

• begins_with tests + empty-binary-prefix behaviour: empty-prefix and multi-byte all-0xFF cases added (above); the empty-binary-prefix-matches-everything behaviour is documented in the test and the impl comment.

• Docs drift: design doc bumped 6.0 → 7.0, and the RFC's CI/test claims rewritten to match the real setup (commit "correct CI/test description and bump design doc to 7.0").

• CREATING commit message: reworded — it now states DeleteTable sets and returns DELETING but completes the drop synchronously, rather than claiming there's no DELETING state.

• Restore backup lookup account predicate: added (restore_table_from_backup now filters on account_id at the storage layer too), matching describe/delete — defence in depth as suggested (commit "scope restore backup lookup to the account").

Full suite re-run through devtools/run-mongodb-tests against MongoDB 7 before pushing: rust integration 414/414, comprehensive 330/330, pytest 920 passed. The one pytest failure is TestAtomicCounter::test_atomic_counter hitting ProvisionedThroughputExceededException under concurrent load with throttling enabled — pre-existing, unrelated to this PR (the branch doesn't touch that test). Happy to iterate further on the BETWEEN edge or anything else.

[diegotoledano95]

@LeeroyHannigan A note on the three red CI jobs — none are in the MongoDB backend; the cross-backend CI and my dual-target test are surfacing pre-existing issues in the other backends.

run-rust-integration (PostgreSQL) — restored_table_has_all_items_when_first_active:

The dual-target restore-completeness test I added runs against Postgres and fails (1379/40000 at first-ACTIVE). It's the same restore race you flagged for MongoDB, in the Postgres backend: storage-postgres/src/backup_engine.rs calls create_table (:454), which schedules the CREATING→ACTIVE transition on a timer, then copies items one INSERT at a time (:480-501); with 40k items the copy outlasts the delay, so the control-plane worker flips ACTIVE mid-copy (the explicit ACTIVE at :520 just races it). MongoDB passes this test after my fix.

Happy to apply the same ordering fix to Postgres in this PR or leave it to you — and let me know if you'd rather I hold the dual-target test back until Postgres is fixed.

run-integration-sqlite (pytest) — two SQLite GSI tests:

• test_query_scan.py::TestBaseKeySchemaFlow::test_index_pagination_uses_base_key_schema_for_tiebreaker — GSI pagination returns a partial set (4–6 of 12 items, varying across runs; assert len == 12 fails). Looks like a cursor/tiebreaker bug in the base-key-schema pagination path.
• test_gsi_async.py::TestGsiAsyncPropagation::test_gsi_sync_path_with_zero_delay — at gsi_propagation_delay_ms=0, the GSI query returns 0 items right after the write (assert 0 == 1).

I reproduced both SQLite tests on a clean main (d6afa1e) SQLite build in a separate worktree — they fail there independently of this PR. This PR touches neither the SQLite backend nor the shared GSI path; its only shared change is an additive StorageError::TransactionConflict variant + its engine mapping (RFC-0003 §4.3), which only MongoDB produces and is inert in the SQLite binary.

So: my PR's CI is red, but on pre-existing bugs in the Postgres and SQLite backends. Let me know how you'd like to proceed — particularly whether the Postgres restore fix belongs in this PR.

[LeeroyHannigan]

Brilliant @diegotoledano95 , thank you!

The one flake, and the one ask. The single first-run failure was restore_active_completeness. I instrumented a mongo-side status watcher and can confirm your server-side transition is atomic (CREATING,0 -> ACTIVE,40000): the implementation is correct. The defect is in the test's observer, which swallows scan errors (Err(_) => break) and reports a truncated count as missing data, so it will flake in CI on any transient scan error. Please make scan errors fail the test (or retry) instead of silently truncating the count. That's the only change I need before approving.

The red CI jobs are ours. Your dual-target tests surfaced three pre-existing bugs in our backends, and rather than asking you to gate the tests, we've fixed our side: #239 (GSI pagination tiebreaker, covers SQLite and Postgres), #245 (SQLite honored a stale cached GSI propagation delay for up to 30s, so the zero-delay synchronous path never engaged), and #246 (Postgres had the same restore race you fixed here: ACTIVE flipped on a timer decoupled from the copy; reproduced at 515/40000 items). Once those three land, a rebase should take this PR fully green with no changes on your side beyond the oracle fix above.

I thought our fixes where already in for our own backends, but they must have got lost in the noise somewhere.

[diegotoledano95]

@LeeroyHannigan Thanks so much for the response! I have pushed the requested change on the test. I will keep an eye on those PRs landing to rebase this and update branch, thank you!

[diegotoledano95]

@LeeroyHannigan Regarding the remaining CI failure.

The SQLite CI job is down to a single remaining failure, test_index_pagination_uses_base_key_schema_for_tiebreaker. #245 fixed the zero-delay GSI test (now green). #239 (cd09154) improved the pagination one but didn't fully close it — it now returns a contiguous prefix that stops early (7 of 12 items, varying 4–7 across runs), pointing at pagination terminating early / a cursor bound, with some timing sensitivity. It reproduces on clean origin/main — my branch doesn't touch storage-sqlite or the shared GSI path (byte-identical to main), so it's not introduced here. Flagging since it keeps the SQLite job red on my PR; happy to help repro but it looks like a follow-up to #239 on your side.