fix(eventhubs): fold checkpoint blob key to lowercase ASCII - #5109
Draft
Johnathan W (j7nw4r) wants to merge 2 commits into
Draft
fix(eventhubs): fold checkpoint blob key to lowercase ASCII#5109Johnathan W (j7nw4r) wants to merge 2 commits into
Johnathan W (j7nw4r) wants to merge 2 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
The checkpoint and ownership blob keys keep the case of the caller's namespace, event hub name, and consumer group. Two callers that spell the same Event Hub with a different case land on two key sets, so a checkpoint that one caller writes is invisible to the other. Add a unit test module to the event processor models. It pins that each of the three key fields folds to lowercase on its own, that the partition id keeps its case, that the key is stable across the input case, and that the fold is an ASCII fold. One characterization test holds the empty parameter errors in place. Add two checkpoint store tests. They store with a mixed case triple, list with a lowercase triple, and expect the record back with its stored case.
Event Hubs treats the fully qualified namespace, the event hub name, and the consumer group as case insensitive. The checkpoint and ownership blob key kept the case of the caller, so one deployment that spelled the consumer group `$Default` on one run and `$default` on the next built two disjoint key sets and reprocessed events. The four key functions on `Checkpoint` and `Ownership` now fold the three key fields with `azure_core::fmt::to_ascii_lowercase`. The partition id keeps its case. The fold applies to ASCII letters only. No call site changes, because every caller routes through these four functions and no caller parses the three segments back out of a key. Records that an older Rust client wrote stay at the old key and become unreachable. The change adds no dual read and no fallback lookup. The recorded playback assets for the blob checkpoint store pin the old casing, so that suite needs a re-record. Fixes Azure#5099
Johnathan W (j7nw4r)
force-pushed
the
j7nw4r/fix-eventhubs-blob-key-lowercase
branch
from
August 25, 2026 18:13
cb256cb to
d904c86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BlobCheckpointStorebuilt the checkpoint and ownership blob key from the fully qualified namespace, the event hub name, and the consumer group without folding their case. Event Hubs treats the consumer group as case insensitive, so one deployment that spelled the group$Defaulton one run and$defaulton the next built two disjoint key sets and reprocessed events. The four key builders now fold those three segments to lowercase ASCII, and the partition id keeps its case.Motivation
The single deployment defect stands on its own and needs no second SDK. Nothing in the crate normalized the three segments, and the service accepts both spellings of a consumer group as the same group, so two runs of one program owned two disjoint sets of checkpoints. Neither run saw the other's progress, and a restart resumed from the wrong place.
The cross language question is separate and harder, and this pull request does not settle it. Three SDKs fold all three segments: .NET (
BlobCheckpointStoreInternal.cslines 142, 204, and 429), JavaScript (blobCheckpointStore.tslines 282 to 284), and Python (_blobstoragecs.py). Three do not: Go (checkpoints/blob_store.golines 274 to 301 format the fields verbatim), Java, and Rust. Folding does not restore interoperability. It moves this crate from the second group into the first. This crate follows the Go shape elsewhere, so the parity choice is not obvious and a maintainer must make it before merge.Changes
Checkpoint::get_checkpoint_blob_prefix_nameandOwnership::get_ownership_prefix_namefold the namespace, the event hub name, and the consumer group withazure_core::fmt::to_ascii_lowercase.Checkpoint::get_checkpoint_blob_nameandOwnership::get_ownership_nameinherit the fold and append the partition id unchanged.str::to_lowercase. The standard library applies the context dependent Greek final sigma rule, which would produce a key that differs from the .NET key in exactly the cross language case this change targets.in_memory_checkpoint_store.rsandcheckpoint_store.rsare unchanged. Every call site already routed through the four functions, and neither parses the three segments back out of a key.Migration
Records that an older Rust client wrote stay at the old key and become unreachable. This change adds no dual read and no fallback lookup. A processor that starts against an existing container resumes from its configured start position.
What this breaks in CI
The recorded playback assets pin the old casing inside their request URIs, so the recorded suite fails on this branch. Measured on this branch: 22 tests, 2 passed, 20 failed. The
checkpoint_unit_testsbinary reports 0 passed and 7 failed. Theownership_unit_testsbinary reports 1 passed and 13 failed, and its one pass isclaim_ownership_empty_list, which sends no HTTP request. The doc test passes. The test proxy reports a request mismatch, for example request...%2F$default%2Fcheckpoint%2F0against record...%2F$Default%2Fcheckpoint%2F0.The repair is a re-record against tag
rust/eventhubs/azure_messaging_eventhubs_checkpointstore_blob_d7273c4b84. A re-record needs the test resources fromsdk/eventhubs/test-resources.bicepand write access toAzure/azure-sdk-assets, so it needs a maintainer. No test source and no recorded asset was edited here.Open decisions for a maintainer
checkpointstore_blobjob stays red.to_ascii_lowercasein the publishedazure_core1.1.0 mixes a char index and a byte index, soto_ascii_lowercase("\u{00e9}A")panics withstart byte index 1 is not a char boundary. Event Hubs restricts these three names to ASCII, so no supported input reaches it, but the four functions are public and take&str. An upstream fix intypespec_client_corewould remove the risk.Test plan
models.rs, and two drive the publicCheckpointStoretrait intests/eventhubs_checkpoint_store.rs.key_is_stable_across_input_casestates the defect without naming a fold direction, andtest_checkpoint_key_survives_consumer_group_case_changefailed withleft: 0, right: 1.key_functions_reject_empty_parametersis a characterization test and is green by design. Deleting theconsumer_groupguard turns it red, which was proved and then reverted.fold_is_ascii_onlyalso fails astr::to_lowercasefix, because the Greek pair must pass through unchanged.RUSTFLAGS=-Dwarnings cargo test --no-runexits 0 for both crates on the pinned 1.95 toolchain.cargo test --package azure_messaging_eventhubs -- --test-threads=1exits 0, andcargo fmt --checkandcargo clippy --all-targets --all-featuresexit 0.Closes #5099
Live validation against real storage
The recorded suite failing does not mean this change is wrong. Run in live mode, where the test proxy is bypassed and the calls reach a real storage account, the whole crate passes with the lowercased key:
checkpoint_unit_tests: 7 passed, 0 failedownership_unit_tests: 14 passed, 0 failedSo the lowercased key round-trips correctly against Azure Blob Storage, and the 20 playback failures are stale assets rather than a broken change. Re-recording the assets is the only outstanding work.