Conversation
42e3b46 to
eecbbd1
Compare
33353c5 to
ef0f689
Compare
eecbbd1 to
dbd31d9
Compare
1308a57 to
f21d3c8
Compare
dbd31d9 to
693f2f9
Compare
Hash join materializes the entire right side into a hash table before probing, so peak memory grows linearly with right-side row count, and multi-column keys fall back to ObjectLookupTable with a composite key allocation per row. When both inputs are already sorted on the join keys, a merge join avoids the hash table entirely and streams. - SortedMergeJoinOperator performs a two-pointer merge with lazy block reads (one block per side held in memory), type-dispatched key comparison, non-equi residual filter support, split equi-only/filter paths, maxRowsInJoin overflow handling (THROW/BREAK) on both the buffered right-key run and the emitted rows, periodic termination checks, and early-termination propagation so a downstream LIMIT stops the join instead of completing the full cross-product of buffered input. - Selected via /*+ joinOptions(join_strategy='sorted') */, carried through the plan as JoinStrategy.SORTED. - PinotJoinExchangeNodeInsertRule injects a LogicalSort below the sort exchange so both inputs arrive globally sorted, not merely sorted per sender. Join keys are collated NULLS LAST, matching the operator's key comparator. - Colocation: joinOptions(is_colocated_by_join_keys='true') sets the existing prePartitioned field on the sort exchange, reusing the hash-join prePartitioned machinery, so co-partitioned inputs get a direct 1:1 exchange (receive fanIn drops to 1, no cross-server shuffle) with no changes needed in MailboxAssignmentVisitor or WorkerManager. Part of apache#18667. (cherry picked from commit 7590ae4)
Reproduce with a hot join key and LIMIT 1: the old merge built every match before returning. Resume in bounded blocks and limit buffered right rows instead of total output.
Cancel a duplicate-key join after its first output block to reproduce the retained input buffers. Drop the run and cursor rows on termination while preserving emitted rows and stats.
Add an H2 case using sortedSelectionMergeMode=on and streamingSortedMailboxReceive=true with a sorted LEFT join, residuals and ORDER BY LIMIT. Confirm the join ordering method implements the shared producer contract.
Materialize accepted equi and residual matches through the same row builder. Keep bounded INNER and LEFT joins, nulls, resource limits, cancellation and cleanup, with three H2 cases for joins, filters and ordered residual LEFT joins.
f21d3c8 to
b82686e
Compare
693f2f9 to
88ef14a
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## xiangfu0/codex1/sorted-exchange-leaf #19713 +/- ##
=======================================================================
Coverage ? 68.32%
Complexity ? 1450
=======================================================================
Files ? 3529
Lines ? 230118
Branches ? 36594
=======================================================================
Hits ? 157226
Misses ? 60559
Partials ? 12333
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
CI follow-up at Current-head checks finished with 13/14 passing. Integration Set 2 failed again after one failed-job-only rerun: This PR still has an open CI gate. A separate baseline fixture startup repair and validation remain necessary; the six-line workflow change is preserved. Maintainer review remains pending. |
Adds opt-in resumable INNER and LEFT sorted merge joins. Explicit sorted strategy requires the broker capability gate, plans complete ASC NULLS LAST sender sorts and distinct k-way merge receives, and preserves explicit or inferred colocation through the Sort. Unsupported clusters fail logical planning; V2 keeps its explicit rejection.
Duplicate-key buffers and output blocks are bounded. Null/LONG/residual semantics remain intact. Early termination, exhausted inputs and BREAK drain terminal mailbox blocks so late errors and upstream statistics survive. The new leaf dependency's streaming instance-plan regression is preserved, and shared window fixtures have no incidental changes. The native diff contains 20 join feature files.
Validation: 716 unique applicable JDK 25 cases, including 35 sorted join tests, eight mailbox terminal regressions, three H2 comparisons, full following/RANGE window coverage and 57 core streaming combine cases; repeated classes are counted once and the two unchanged broker gate cases are explicitly reused. No failures/errors/skips or warnings on added lines. Mandatory affected formatting, license and checkstyle checks pass.
This PR remains stacked on #19712. Hosted CI at its final remote head and reviewer/base decisions remain open; local tests do not claim remote green CI. The cached capability gate is not an atomic downgrade guarantee, and #19395's stalled-sender memory followup remains deferred.
Final stacked head:
b82686e30f759d1610e2021967c3cef7fd3234cd; dependency head:88ef14a0fd97415f58c1aefc15c33a6620c759d6. Existing base names remain unchanged; the new dependency heads await root's guarded second publication.The inherited fixture followup independently passes all 593 ResourceBasedQueryPlansTest cases on this exact source tree with compiler warnings/deprecations enabled, zero failures/errors/skips and all four planner formatting/license/style goals passing. Its native feature patch is byte-identical to the preceding published stack; only core owns the 124 corrected window-input fetch expectations. These local results do not establish hosted CI status.
The existing stack base is retained. Full unit/integration PR workflows are filtered to master; no complete hosted CI result exists for this child at its current stack base. After the prerequisite PRs land, the agreed base update and exact-head CI are still required. No merge, automatic merge, CI suppression or master integration was performed.