Skip to content

test(eventhubs): drive both processors in the displacement test - #5105

Draft
Johnathan W (j7nw4r) wants to merge 2 commits into
Azure:mainfrom
j7nw4r:j7nw4r/fix-eventhubs-displacement-live-test
Draft

test(eventhubs): drive both processors in the displacement test#5105
Johnathan W (j7nw4r) wants to merge 2 commits into
Azure:mainfrom
j7nw4r:j7nw4r/fix-eventhubs-displacement-live-test

Conversation

@j7nw4r

@j7nw4r Johnathan W (j7nw4r) commented Aug 20, 2026

Copy link
Copy Markdown
Member

Summary

The live test second_processor_displaces_first_with_consumer_disconnected never made the second processor attach an AMQP receiver, so nothing displaced the first processor and the expected ConsumerDisconnected never arrived. The test now attaches processor A before processor B starts, and it drives processor B's partition clients so B holds real links. The assertion on ErrorKind::ConsumerDisconnected does not change.

Motivation

A receiver reaches the broker on the first poll of stream_events(), and not when next_partition_client() hands the partition client out (see #5094). EventProcessor::run() claims partitions and builds EventReceiver values, but it polls nothing, and the old test polled processor A's streams only. Processor B therefore held no link on any partition, and one receiver on a partition creates no epoch contention, so the 90 second budget expired with no error. The attach order was wrong as well: both processors attach at PROCESSOR_OWNER_LEVEL = 0, and the broker disconnects the receiver that attached first when a receiver with an equal or higher epoch arrives, so an A that attached after B would displace B instead. The displacement path itself is intact, and unit tests cover both the map from amqp:link:stolen to ErrorKind::ConsumerDisconnected and the retry decider that refuses to reattach a stolen link. The partition count lead in the issue is ruled out: the at least one of 5 partitions text prints partition_clients_a.len(), and no assertion depends on that number.

Changes

  • Poll processor A's merged stream for a bounded attach window before processor B is built, so every one of A's receivers reaches the broker first.
  • Fail with a clear message when a partition errors inside that window, because nothing else holds a link yet, so the error is a failed attach and not a displacement.
  • Drain processor B's partition clients and poll their merged stream, so B attaches the epoch 0 links that displace A.
  • Run the B driver on the same task as the A race, because PartitionClient::stream_events() is not Send and cannot move to a spawned task.
  • Park the B driver on futures::future::pending after its streams end, so tokio::select! never polls a finished future.
  • State both rules in the test's doc comment: the attach happens on the first poll, and the attach order decides the direction of the steal.

Test plan

  • CARGO_BUILD_JOBS=1 RUSTFLAGS=-Dwarnings cargo test --no-run --package azure_messaging_eventhubs exits 0.
  • CARGO_BUILD_JOBS=1 RUSTFLAGS=-Dwarnings cargo clippy -p azure_messaging_eventhubs --all-features --all-targets exits 0.
  • CARGO_BUILD_JOBS=1 cargo test -p azure_messaging_eventhubs --lib -- --test-threads=1 exits 0, with 144 passed and 0 failed. That set holds the stolen-link tests that prove the library half of the path.
  • cargo fmt --package azure_messaging_eventhubs -- --check exits 0, and cspell reports 0 issues over 1 file.

Live validation

The test ran against a real namespace on 2026-08-20. This change is necessary and it is not sufficient. Measured, four runs:

test as on main test with this change
azure_core_amqp 1.1.0 from crates.io FAIL FAIL
in-tree azure_core_amqp 1.2.0-beta.1 FAIL PASS

The second blocker is a dependency, not a test defect. azure_messaging_eventhubs links the crates.io azure_core_amqp 1.1.0, whose receiver builder calls .properties() before .name(). In fe2o3-amqp 0.14.0 the name() builder method rebuilds the struct with properties: Default::default(), so it discards them, and the com.microsoft:epoch link property that carries the owner level never reaches the broker. With no epoch, the broker never arbitrates and nothing is displaced. The in-tree copy fixed that order in #4805.

So this pull request must not merge alone and be expected to turn the live test green. It needs the dependency flip that PR #5078 already carries, or a released azure_core_amqp 1.2.0. See #5100 for the full matrix.

  • Confidence in the diagnosis is high, because the lazy attach is plain in open_receiver_on_partition and issue [Event Hubs] A receiver attaches on the first stream poll, so open_receiver_on_partition cannot report a bad consumer group #5094 reports the same behavior from a separate investigation. Confidence that the fix makes the live test pass is medium, because attach timing against a real namespace is not proven here.
  • The live command to confirm it: AZURE_TEST_MODE=live EVENTHUBS_HOST=<namespace>.servicebus.windows.net EVENTHUB_NAME=<hub> cargo test --package azure_messaging_eventhubs --test eventhubs_processor second_processor_displaces_first_with_consumer_disconnected -- --exact --test-threads=1.

Closes #5100

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

The live test never made the second processor attach a receiver, so the
broker never displaced the first one.

A receiver reaches the broker on the first poll of `stream_events()`. It
does not attach when `next_partition_client()` hands the partition client
out. `EventProcessor::run()` claims partitions and builds `EventReceiver`
values, but it polls nothing. The test polled processor A's streams only.
Processor B therefore held no AMQP link on any partition. One receiver on
a partition creates no epoch contention, so the 90 second budget expired
with no error.

The test also polled A's streams only after B had started. Both
processors attach at owner level 0, and the broker disconnects the
receiver that attached first. An A that attaches after B displaces B, so
the test would still see no error on A.

The test now polls A's merged stream for a bounded window before it
builds B. It then drives B's partition clients on the same task, because
those streams are not `Send`.

The assertion on `ErrorKind::ConsumerDisconnected` does not change.
The arm that runs when every stream ends without an error returned an
empty partition id. The panic then read "partition  stream ended", which
hides which partitions the test watched. It now returns "(all)".
@j7nw4r
Johnathan W (j7nw4r) force-pushed the j7nw4r/fix-eventhubs-displacement-live-test branch from 44bf553 to 46107c5 Compare August 25, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Event Hubs] Live test second_processor_displaces_first_with_consumer_disconnected fails on main

1 participant