Skip to content

[MSE] Share ordering proof and stream ordered mailbox results - #19711

Open
xiangfu0 wants to merge 7 commits into
apache:xiangfu0/codex1/sorted-exchange-corefrom
xiangfu0:xiangfu0/codex1/sorted-exchange-ordering
Open

xiangfu0 wants to merge 7 commits into
apache:xiangfu0/codex1/sorted-exchange-corefrom
xiangfu0:xiangfu0/codex1/sorted-exchange-ordering

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Streams eligible leaf ORDER BY through a logical k-way merge exchange. Matching typed or single-physical-table leaf selections retain sender ordering; the distinct merge receive applies finite global LIMIT/OFFSET after merging. Hybrid/logical expansion retains the MSE Sort. Plain sends carry ordered input without mailbox markers, structural sender checks or production row validation.

The option remains off by default. The inherited broker gate enables new merge nodes only for homogeneous matching immutable release versions; missing, unknown, mixed, unreadable or SNAPSHOT versions and multi-cluster queries keep the compatible receiver-sort plan. This cached gate is a point-in-time check. Global window receiver sorts preserve complete input beyond response caps; the agreed stalled-sender memory followup remains #19395.

Final stacked head: 755721f7fd1548f3e94f81c86f4f64cfae4c66c8, on corrected core b047dd479ea360f19cc8ea13595a874590ed66de. Its native diff contains 24 ordering feature files. All original feature commits remain in order with their author/full-message/author-date preserved; source matches the validated input tree exactly.

Validation: 526 affected JDK 25 warning/deprecation-enabled cases, plus 376 overlapping full planner/rule/window cases; zero failures/errors/skips or added-line warnings. Applicable formatting, license and checkstyle checks pass. Hosted CI for the final remote head and reviewer/base decisions remain open; local results do not represent remote green checks.

The inherited fixture followup independently passes all 593 ResourceBasedQueryPlansTest cases on this exact source tree with compiler warnings/deprecations enabled, zero failures/errors/skips and all four planner formatting/license/style goals passing. Its native feature patch is byte-identical to the preceding published stack; only core owns the 124 corrected window-input fetch expectations. These local results do not establish hosted CI status.

The existing stack base is retained. Full unit/integration PR workflows are filtered to master; no complete hosted CI result exists for this child at its current stack base. After the prerequisite PRs land, the agreed base update and exact-head CI are still required. No merge, automatic merge, CI suppression or master integration was performed.

xiangfu0 and others added 6 commits October 4, 2026 04:20
Set streamingSortedMailboxReceive=true to merge leaf ORDER BY on one physical table and stream LIMIT/OFFSET. Match producer collations and reject sender disorder across blocks.
Sample resource usage and the active deadline periodically, including single-sender blocks that pass through without a size cap.
Keep physical leaf and matching-collation coverage while reusing existing offset and limit checks.
Keep the mailbox poll assertion compiling with the simplified parent receiver. Ordering and cancellation behavior remain the same.
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex1/sorted-exchange-ordering branch from 8e86cc7 to 755721f Compare October 3, 2026 20:25
@codecov-commenter

codecov-commenter commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.56140% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.19%. Comparing base (b047dd4) to head (03319a2).
⚠️ Report is 1 commits behind head on xiangfu0/codex1/sorted-exchange-core.

Files with missing lines Patch % Lines
...calcite/rel/rules/PinotSortedLeafExchangeRule.java 74.35% 3 Missing and 7 partials ⚠️
...he/pinot/query/planner/explain/PlanNodeMerger.java 0.00% 3 Missing and 1 partial ⚠️
.../query/planner/plannode/KWayMergeExchangeNode.java 60.00% 3 Missing and 1 partial ⚠️
.../apache/pinot/query/planner/plannode/SortNode.java 50.00% 1 Missing and 3 partials ⚠️
.../query/planner/logical/RelToPlanNodeConverter.java 50.00% 0 Missing and 3 partials ⚠️
.../runtime/plan/server/ServerPlanRequestContext.java 75.00% 0 Missing and 2 partials ⚠️
...alcite/rel/logical/PinotKWayMergeSortExchange.java 90.00% 1 Missing ⚠️
...che/pinot/query/runtime/operator/LeafOperator.java 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@                            Coverage Diff                             @@
##             xiangfu0/codex1/sorted-exchange-core   #19711      +/-   ##
==========================================================================
- Coverage                                   68.25%   68.19%   -0.06%     
  Complexity                                   1450     1450              
==========================================================================
  Files                                        3525     3526       +1     
  Lines                                      229231   229252      +21     
  Branches                                    36381    36389       +8     
==========================================================================
- Hits                                       156458   156342     -116     
- Misses                                      60502    60611     +109     
- Partials                                    12271    12299      +28     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 ?
java-25 68.19% <74.56%> (-0.06%) ⬇️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 68.19% <74.56%> (-0.06%) ⬇️
unittests 68.19% <74.56%> (-0.06%) ⬇️
unittests1 58.18% <74.56%> (+0.01%) ⬆️
unittests2 39.81% <3.50%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0

xiangfu0 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

CI follow-up at 03319a25 adds six exact stacked-base pull_request branch filters (+6/-0). Feature code/tests and existing master/push, path, permission and job rules are preserved.

All 14 current-head checks passed: unit/integration, compatibility, quickstart, Java 11 client, linter and Trivy (checks). Maintainer review remains pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants