test(eventhubs): drive both processors in the displacement test - #5105
Draft
Johnathan W (j7nw4r) wants to merge 2 commits into
Draft
test(eventhubs): drive both processors in the displacement test#5105Johnathan W (j7nw4r) wants to merge 2 commits into
Johnathan W (j7nw4r) wants to merge 2 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
The live test never made the second processor attach a receiver, so the broker never displaced the first one. A receiver reaches the broker on the first poll of `stream_events()`. It does not attach when `next_partition_client()` hands the partition client out. `EventProcessor::run()` claims partitions and builds `EventReceiver` values, but it polls nothing. The test polled processor A's streams only. Processor B therefore held no AMQP link on any partition. One receiver on a partition creates no epoch contention, so the 90 second budget expired with no error. The test also polled A's streams only after B had started. Both processors attach at owner level 0, and the broker disconnects the receiver that attached first. An A that attaches after B displaces B, so the test would still see no error on A. The test now polls A's merged stream for a bounded window before it builds B. It then drives B's partition clients on the same task, because those streams are not `Send`. The assertion on `ErrorKind::ConsumerDisconnected` does not change.
The arm that runs when every stream ends without an error returned an empty partition id. The panic then read "partition stream ended", which hides which partitions the test watched. It now returns "(all)".
Johnathan W (j7nw4r)
force-pushed
the
j7nw4r/fix-eventhubs-displacement-live-test
branch
from
August 25, 2026 18:13
44bf553 to
46107c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The live test
second_processor_displaces_first_with_consumer_disconnectednever made the second processor attach an AMQP receiver, so nothing displaced the first processor and the expectedConsumerDisconnectednever arrived. The test now attaches processor A before processor B starts, and it drives processor B's partition clients so B holds real links. The assertion onErrorKind::ConsumerDisconnecteddoes not change.Motivation
A receiver reaches the broker on the first poll of
stream_events(), and not whennext_partition_client()hands the partition client out (see #5094).EventProcessor::run()claims partitions and buildsEventReceivervalues, but it polls nothing, and the old test polled processor A's streams only. Processor B therefore held no link on any partition, and one receiver on a partition creates no epoch contention, so the 90 second budget expired with no error. The attach order was wrong as well: both processors attach atPROCESSOR_OWNER_LEVEL = 0, and the broker disconnects the receiver that attached first when a receiver with an equal or higher epoch arrives, so an A that attached after B would displace B instead. The displacement path itself is intact, and unit tests cover both the map fromamqp:link:stolentoErrorKind::ConsumerDisconnectedand the retry decider that refuses to reattach a stolen link. The partition count lead in the issue is ruled out: theat least one of 5 partitionstext printspartition_clients_a.len(), and no assertion depends on that number.Changes
PartitionClient::stream_events()is notSendand cannot move to a spawned task.futures::future::pendingafter its streams end, sotokio::select!never polls a finished future.Test plan
CARGO_BUILD_JOBS=1 RUSTFLAGS=-Dwarnings cargo test --no-run --package azure_messaging_eventhubsexits 0.CARGO_BUILD_JOBS=1 RUSTFLAGS=-Dwarnings cargo clippy -p azure_messaging_eventhubs --all-features --all-targetsexits 0.CARGO_BUILD_JOBS=1 cargo test -p azure_messaging_eventhubs --lib -- --test-threads=1exits 0, with 144 passed and 0 failed. That set holds the stolen-link tests that prove the library half of the path.cargo fmt --package azure_messaging_eventhubs -- --checkexits 0, and cspell reports 0 issues over 1 file.Live validation
The test ran against a real namespace on 2026-08-20. This change is necessary and it is not sufficient. Measured, four runs:
azure_core_amqp1.1.0 from crates.ioazure_core_amqp1.2.0-beta.1The second blocker is a dependency, not a test defect.
azure_messaging_eventhubslinks the crates.ioazure_core_amqp1.1.0, whose receiver builder calls.properties()before.name(). Infe2o3-amqp0.14.0 thename()builder method rebuilds the struct withproperties: Default::default(), so it discards them, and thecom.microsoft:epochlink property that carries the owner level never reaches the broker. With no epoch, the broker never arbitrates and nothing is displaced. The in-tree copy fixed that order in #4805.So this pull request must not merge alone and be expected to turn the live test green. It needs the dependency flip that PR #5078 already carries, or a released
azure_core_amqp1.2.0. See #5100 for the full matrix.open_receiver_on_partitionand issue [Event Hubs] A receiver attaches on the first stream poll, so open_receiver_on_partition cannot report a bad consumer group #5094 reports the same behavior from a separate investigation. Confidence that the fix makes the live test pass is medium, because attach timing against a real namespace is not proven here.AZURE_TEST_MODE=live EVENTHUBS_HOST=<namespace>.servicebus.windows.net EVENTHUB_NAME=<hub> cargo test --package azure_messaging_eventhubs --test eventhubs_processor second_processor_displaces_first_with_consumer_disconnected -- --exact --test-threads=1.Closes #5100