Flex ranks: scalable two-level layout exchange (#726) and coherence read-outcome diagnostic (#742)

[DLWoodruff]

Two flexible-ranks items in one PR, both localized to the unequal-rank communication layer.

Closes #726. Closes #742.

Scalable layout exchange (#726)

The unequal-rank path's buffer-layout exchange was an interim flat fullcomm.allgather — an O(N) startup collective that must not ship for production use, since total rank counts in the thousands are a real operating regime. It is replaced by the two-level scheme the design doc prefers (two_level_layout_exchange in spcommunicator.py):

1. allgather within each cylinder (size = that cylinder's rank count);
2. allgather across one anchor rank per cylinder (the base rank, on a temporary fullcomm.Split comm of just the anchors — a handful of ranks);
3. broadcast of the assembled result within each cylinder.

Two-level was chosen over local-compute because field registration is dynamic (extensions register extra send fields), so remote layouts cannot be reconstructed from static information alone. The change only affects how strata_buffer_layouts is populated at startup — the multi-source reader is untouched, and the equal-rank path keeps its strata_comm.allgather verbatim (SPWindow takes an optional layout_exchanger; the default is the old behavior).

Coherence read-outcome diagnostic (#742)

The unequal-rank multi-source reader now counts every multi-source read (>= 2 sources; a single source cannot straddle a publish) into one of five buckets at the reduce_source_write_ids call site:

• new_accepted — coherent, advanced, used
• not_new — coherent, but the sender has not published (slow sender)
• rejected_incoherent — strict field, sources disagreed (the fundamental coherence miss)
• rejected_cross_reader — this rank's sources agreed, the collective cross-reader check rejected
• accepted_mixed — relaxed field, blended assembly used

so an infrequently-reporting bounds cylinder can be diagnosed as coherence problem (rejected_incoherent/accepted_mixed dominate) vs. slow upstream sender (not_new dominates). Counting is always-on (two integer increments per multi-source read); each cylinder prints a per-field summary at finalization (aggregated across its ranks with one gather, rank-0-gated, only for fields that did multi-source reads — equal-rank runs print nothing). An opt-in periodic line (coherence_diagnostics_period in the spcomm options) supports live debugging, and the counters are exposed as SPCommunicator.coherence_counters. Sample from the farmer 4+2 integration test:

coherence diagnostic [LagrangianOuterBound] DUALS: total=276, new_accepted=90, not_new=160, rejected_incoherent=13, rejected_cross_reader=13, accepted_mixed=0, miss rate=4.71%

Tests

• New test_flex_layout_exchange.py (mpiexec -np 6, solver-free): pins two_level_layout_exchange against fullcomm.allgather ground truth with rank-unique payloads across ten cylinder partitions of 6 ranks (including single-rank cylinders and a single-cylinder run). Wired into test_pr_and_main.yml and run_coverage.bash in this PR.
• test_flex_coherence_policy.py: the deterministic stub reader now also pins the counter buckets (strict mixed → rejected_incoherent, relaxed mixed → accepted_mixed, coherent no-advance → not_new, buckets partition the total).
• test_flexible_rank_duals.py: after both split directions, asserts the buckets partition the total, that the strict DUALS field never accepts a blended read, and that the 4+2 bound came from accepted multi-source reads (in 2+4 the spoke's slices can nest inside the hub's halves, leaving every read single-source, so counter presence is only guaranteed in 4+2). All existing flex MPI tests (cylinders, duals, xhat, xfeas, xhat_multistage, extension_fields, spwindow_multisource) pass locally at -np 6 and exercise the new exchange end-to-end.

The design doc's Option D layout-exchange note, Phase-2 bullet, and release-gate list are updated to reflect the implemented exchange, and the #742 diagnostic moves from "possible future work" into the coherence section.

Deliberately not in this PR: #749 (reduced_costs spoke at flexible ratios, Phase 7) — behavioral work on the spoke itself, better reviewed separately.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 96.72131% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.39%. Comparing base (3749661) to head (7648deb).

Files with missing lines	Patch %	Lines
mpisppy/cylinders/spcommunicator.py	96.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #816      +/-   ##
==========================================
+ Coverage   76.33%   76.39%   +0.05%     
==========================================
  Files         170      170              
  Lines       22628    22685      +57     
==========================================
+ Hits        17274    17331      +57     
  Misses       5354     5354              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.