Skip to content

fix(eventhubs): fold checkpoint blob key to lowercase ASCII - #5109

Draft
Johnathan W (j7nw4r) wants to merge 2 commits into
Azure:mainfrom
j7nw4r:j7nw4r/fix-eventhubs-blob-key-lowercase
Draft

fix(eventhubs): fold checkpoint blob key to lowercase ASCII#5109
Johnathan W (j7nw4r) wants to merge 2 commits into
Azure:mainfrom
j7nw4r:j7nw4r/fix-eventhubs-blob-key-lowercase

Conversation

@j7nw4r

@j7nw4r Johnathan W (j7nw4r) commented Aug 20, 2026

Copy link
Copy Markdown
Member

Summary

BlobCheckpointStore built the checkpoint and ownership blob key from the fully qualified namespace, the event hub name, and the consumer group without folding their case. Event Hubs treats the consumer group as case insensitive, so one deployment that spelled the group $Default on one run and $default on the next built two disjoint key sets and reprocessed events. The four key builders now fold those three segments to lowercase ASCII, and the partition id keeps its case.

Motivation

The single deployment defect stands on its own and needs no second SDK. Nothing in the crate normalized the three segments, and the service accepts both spellings of a consumer group as the same group, so two runs of one program owned two disjoint sets of checkpoints. Neither run saw the other's progress, and a restart resumed from the wrong place.

The cross language question is separate and harder, and this pull request does not settle it. Three SDKs fold all three segments: .NET (BlobCheckpointStoreInternal.cs lines 142, 204, and 429), JavaScript (blobCheckpointStore.ts lines 282 to 284), and Python (_blobstoragecs.py). Three do not: Go (checkpoints/blob_store.go lines 274 to 301 format the fields verbatim), Java, and Rust. Folding does not restore interoperability. It moves this crate from the second group into the first. This crate follows the Go shape elsewhere, so the parity choice is not obvious and a maintainer must make it before merge.

Changes

  • Checkpoint::get_checkpoint_blob_prefix_name and Ownership::get_ownership_prefix_name fold the namespace, the event hub name, and the consumer group with azure_core::fmt::to_ascii_lowercase.
  • Checkpoint::get_checkpoint_blob_name and Ownership::get_ownership_name inherit the fold and append the partition id unchanged.
  • The fold is ASCII only, not str::to_lowercase. The standard library applies the context dependent Greek final sigma rule, which would produce a key that differs from the .NET key in exactly the cross language case this change targets.
  • The doc comments on all four functions state the folding rule and name the layout, because the functions are public.
  • Both changelogs carry a Breaking Changes entry that names the old key, the new key, and the migration.
  • in_memory_checkpoint_store.rs and checkpoint_store.rs are unchanged. Every call site already routed through the four functions, and neither parses the three segments back out of a key.

Migration

Records that an older Rust client wrote stay at the old key and become unreachable. This change adds no dual read and no fallback lookup. A processor that starts against an existing container resumes from its configured start position.

What this breaks in CI

The recorded playback assets pin the old casing inside their request URIs, so the recorded suite fails on this branch. Measured on this branch: 22 tests, 2 passed, 20 failed. The checkpoint_unit_tests binary reports 0 passed and 7 failed. The ownership_unit_tests binary reports 1 passed and 13 failed, and its one pass is claim_ownership_empty_list, which sends no HTTP request. The doc test passes. The test proxy reports a request mismatch, for example request ...%2F$default%2Fcheckpoint%2F0 against record ...%2F$Default%2Fcheckpoint%2F0.

The repair is a re-record against tag rust/eventhubs/azure_messaging_eventhubs_checkpointstore_blob_d7273c4b84. A re-record needs the test resources from sdk/eventhubs/test-resources.bicep and write access to Azure/azure-sdk-assets, so it needs a maintainer. No test source and no recorded asset was edited here.

Open decisions for a maintainer

  • The parity choice between the .NET group and the Go group, stated under Motivation.
  • The re-record, without which the checkpointstore_blob job stays red.
  • to_ascii_lowercase in the published azure_core 1.1.0 mixes a char index and a byte index, so to_ascii_lowercase("\u{00e9}A") panics with start byte index 1 is not a char boundary. Event Hubs restricts these three names to ASCII, so no supported input reaches it, but the four functions are public and take &str. An upstream fix in typespec_client_core would remove the risk.

Test plan

  • 8 new plain tests and no new recording. Six unit tests sit inline in models.rs, and two drive the public CheckpointStore trait in tests/eventhubs_checkpoint_store.rs.
  • Seven were proved red against the unchanged source before the fix landed. key_is_stable_across_input_case states the defect without naming a fold direction, and test_checkpoint_key_survives_consumer_group_case_change failed with left: 0, right: 1.
  • key_functions_reject_empty_parameters is a characterization test and is green by design. Deleting the consumer_group guard turns it red, which was proved and then reverted.
  • fold_is_ascii_only also fails a str::to_lowercase fix, because the Greek pair must pass through unchanged.
  • RUSTFLAGS=-Dwarnings cargo test --no-run exits 0 for both crates on the pinned 1.95 toolchain.
  • cargo test --package azure_messaging_eventhubs -- --test-threads=1 exits 0, and cargo fmt --check and cargo clippy --all-targets --all-features exit 0.
  • No live test ran.

Closes #5099

Live validation against real storage

The recorded suite failing does not mean this change is wrong. Run in live mode, where the test proxy is bypassed and the calls reach a real storage account, the whole crate passes with the lowercased key:

  • checkpoint_unit_tests: 7 passed, 0 failed
  • ownership_unit_tests: 14 passed, 0 failed

So the lowercased key round-trips correctly against Azure Blob Storage, and the 20 playback failures are stale assets rather than a broken change. Re-recording the assets is the only outstanding work.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added Event Hubs Storage Storage Service (Queues, Blobs, Files) labels Aug 20, 2026
@j7nw4r Johnathan W (j7nw4r) self-assigned this Aug 20, 2026
The checkpoint and ownership blob keys keep the case of the caller's
namespace, event hub name, and consumer group. Two callers that spell
the same Event Hub with a different case land on two key sets, so a
checkpoint that one caller writes is invisible to the other.

Add a unit test module to the event processor models. It pins that
each of the three key fields folds to lowercase on its own, that the
partition id keeps its case, that the key is stable across the input
case, and that the fold is an ASCII fold. One characterization test
holds the empty parameter errors in place.

Add two checkpoint store tests. They store with a mixed case triple,
list with a lowercase triple, and expect the record back with its
stored case.
Event Hubs treats the fully qualified namespace, the event hub name,
and the consumer group as case insensitive. The checkpoint and
ownership blob key kept the case of the caller, so one deployment
that spelled the consumer group `$Default` on one run and `$default`
on the next built two disjoint key sets and reprocessed events.

The four key functions on `Checkpoint` and `Ownership` now fold the
three key fields with `azure_core::fmt::to_ascii_lowercase`. The
partition id keeps its case. The fold applies to ASCII letters only.
No call site changes, because every caller routes through these four
functions and no caller parses the three segments back out of a key.

Records that an older Rust client wrote stay at the old key and
become unreachable. The change adds no dual read and no fallback
lookup.

The recorded playback assets for the blob checkpoint store pin the
old casing, so that suite needs a re-record.

Fixes Azure#5099
@j7nw4r
Johnathan W (j7nw4r) force-pushed the j7nw4r/fix-eventhubs-blob-key-lowercase branch from cb256cb to d904c86 Compare August 25, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Event Hubs Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Event Hubs] BlobCheckpointStore does not lowercase the blob key, so Rust and .NET processors do not interoperate

1 participant