Skip to content

test(eventhubs): cover blob checkpoint store isolation - #5090

Draft
Johnathan W (j7nw4r) wants to merge 1 commit into
Azure:mainfrom
j7nw4r:j7nw4r/eventhubs-blob-checkpoint-live-tests
Draft

test(eventhubs): cover blob checkpoint store isolation#5090
Johnathan W (j7nw4r) wants to merge 1 commit into
Azure:mainfrom
j7nw4r:j7nw4r/eventhubs-blob-checkpoint-live-tests

Conversation

@j7nw4r

@j7nw4r Johnathan W (j7nw4r) commented Aug 20, 2026

Copy link
Copy Markdown
Member

Summary

azure_messaging_eventhubs_checkpointstore_blob had 14 tests, which covered claims, etag versions, and a stale etag failure. This change adds 17 tests. They cover record isolation across the event hub, the consumer group, and the namespace, and the error that store operations return against a container that does not exist.

Motivation

The blob checkpoint store keys every record by namespace, event hub, consumer group, and partition. A wrong key lets two unrelated processors overwrite each other's checkpoints, and events are then lost or processed twice. No test pinned that key. The four key builders in event_processor/models.rs had no unit test at all, and no test covered a container that does not exist.

Two of the existing tests look like isolation proofs but cannot fail. list_checkpoints and list_ownerships copy the namespace, the event hub, and the consumer group from the caller's own arguments onto every returned record, so an assertion on those three fields proves nothing. The new tests do not use that shape.

Changes

  • Add tests/checkpoint_store_isolation_tests.rs with 17 tests and no change to any other file.
  • Add 6 live isolation tests, one for each of the event hub, the consumer group, and the namespace, for checkpoints and again for ownership.
  • Add 4 live tests that prove list_checkpoints, list_ownerships, update_checkpoint, and claim_ownership give HTTP 404 with the ContainerNotFound error code against a container that does not exist.
  • Add 7 plain #[test] unit tests over the four public key builders, which pin the blob name layout with no service.
  • Prove isolation by presence, by absence, by partition_id, and by blob metadata, never by the three fields that the list methods copy from their arguments.
  • Randomize all three key parts in each isolation test, so the expected count of 1 is exact in the shared container.
  • Keep the file free of mod checkpoint_unit_tests;, which would compile and run the 7 existing checkpoint tests a third time.

Test plan

  • The 10 service tests use #[recorded::test(live)], not #[recorded::test]. A new recorded test name has no recording in the tag that assets.json pins, so CI fails it with header not found x-recording-id. A live test is compiled with #[ignore] unless AZURE_TEST_MODE=live, so it needs no recording asset. assets.json is unchanged, and no recording must be pushed for this change.
  • CARGO_BUILD_JOBS=1 cargo test -p azure_messaging_eventhubs_checkpointstore_blob --test checkpoint_store_isolation_tests -- --test-threads=1 gives 7 passed, 0 failed, 10 ignored.
  • The 7 unit tests carry a mutation proof. Seven single-line edits to sdk/eventhubs/azure_messaging_eventhubs/src/event_processor/models.rs each broke the predicted tests and were then reverted. A change of /checkpoint/ to /checkpoints/ at line 58 breaks the two checkpoint layout tests. A dropped + partition_id at line 73 breaks checkpoint_blob_name_appends_partition_id alone. A dropped consumer group at lines 56 and 57 breaks checkpoint_blob_name_differs_for_each_key_part. A dropped check_non_empty_parameter!(consumer_group); at line 52 breaks key_builders_reject_empty_key_parts alone. The ownership builders give the same result at lines 116, 134, 112, and 113.
  • cargo fmt --check, cargo clippy --all-features --all-targets with -Dwarnings, cargo doc --all-features --no-deps with -Dwarnings, and cspell with the repository configuration all pass.

Three findings from this work are out of scope here and need their own issues.

  • .NET lowercases the namespace, the event hub, and the consumer group before it builds the blob key, and the Rust builders do not. A Rust processor and a .NET processor that share one container write to different blobs.
  • set_checkpoint_metadata_on_blob treats every 404 from set_metadata as a missing blob and retries with an upload. For a missing container it logs "Blob ... not found, creating." and then fails on the upload. The returned error is correct, and the log names the wrong cause.
  • The assertions at checkpoint_unit_tests.rs:65-66 and ownership_unit_tests.rs:52-53 and their peers stay vacuous, because this change adds a file and edits none.

Closes #4894.

Live validation

Every test here ran against a live Event Hubs namespace on 2026-08-20: 17 passed, 0 failed.

Command: AZURE_TEST_MODE=live cargo test --package azure_messaging_eventhubs_checkpointstore_blob -- --test-threads=1.

No change was needed. The tests passed as written on the first live run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added Event Hubs Storage Storage Service (Queues, Blobs, Files) labels Aug 20, 2026
@j7nw4r Johnathan W (j7nw4r) self-assigned this Aug 20, 2026
Add a test file for the blob checkpoint store. Six live tests write
two records that differ in one key part only, then list one key and
make sure the other record does not appear. Four live tests point the
store at a container that does not exist and make sure each operation
reports HTTP 404 with the ContainerNotFound error code.

The service tests are live only. A new recorded test name has no
asset in the pinned assets tag, so a playback run would fail on a
missing recording.

Seven key-builder tests run with no service and no environment
variable. A mutation proof backs them. Seven single-line edits to
the checkpoint and ownership key builders each make the predicted
test fail.
@j7nw4r
Johnathan W (j7nw4r) force-pushed the j7nw4r/eventhubs-blob-checkpoint-live-tests branch from 46aebc7 to d677ea8 Compare August 25, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Event Hubs Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Event Hubs] Add live tests for the blob checkpoint store

1 participant