Multi-Stage: Add SQL:2016 MATCH_RECOGNIZE (row pattern recognition)

[xiangfu0]

Adds SQL:2016 row pattern recognition to Pinot's multi-stage query engine: validation, logical/wire plans, exchange planning, and an intermediate-stage matcher.

Pinot's parser already accepts the MATCH_RECOGNIZE grammar through Calcite's Babel parser. This PR adds the missing supported-subset validation and execution path.

Design and delivery boundary

The parser validation, MatchNode wire representation, exchange rule, runtime operator, and expression semantics form one end-to-end query operator and must land atomically. Splitting those layers would either expose syntax that cannot execute or dispatch plans that servers cannot interpret. Existing non-MATCH_RECOGNIZE queries keep their existing plan and execution behavior; unsupported MATCH_RECOGNIZE modes fail closed with actionable validation errors. The PR remains labeled design-review for maintainer sign-off before landing.

Rollback is likewise atomic: revert the feature commits and stop issuing MATCH_RECOGNIZE queries. There is no persisted-data or table-schema migration.

Front end and planner

• Registers PREV/NEXT/FIRST/LAST/CLASSIFIER/MATCH_NUMBER/RUNNING/FINAL in the strict PinotOperatorTable allow-list.
• Adds MatchRecognizeValidator, including SQL:2016's omitted AFTER MATCH default (SKIP PAST LAST ROW) and actionable rejection of deferred constructs such as ALL ROWS PER MATCH, SUBSET, PERMUTE, exclusions, WITHIN, aggregates in DEFINE, explicit null ordering, expression partition/order keys, and multi-value partition or aggregate inputs.
• Applies SQL identifier rules consistently: unquoted pattern variables are case-insensitive, quoted identifiers retain case, and row-source aliases remain isolated from the pattern-variable namespace.
• Treats RUNNING in DEFINE as the clause's intrinsic mode and rejects unsupported FINAL semantics there.
• Derives CLASSIFIER() as nullable so Calcite preserves COUNT(classifier_measure) instead of incorrectly rewriting it to COUNT(*) for empty matches.
• Adds MatchNode and a self-contained recursive RowPattern; PatternFieldRef preserves pattern-variable identity instead of degrading to a plain input reference.
• Hash-distributes by PARTITION BY, then sorts by the exchange's actual partition-key order plus the requested order keys. Queries without PARTITION BY require the explicit allowMatchRecognizeWithoutPartitionBy query option and execute as one global partition.
• The v2 physical optimizer and lite mode are not supported yet. The planner now checks the effective broker defaults plus query overrides and rejects these combinations explicitly; SET usePhysicalOptimizer = false selects the supported path.
• Sender-side sorting for sort exchanges is tracked separately in [MSE] Support sender-side sorting for sort exchanges #19395.

Runtime and resource safety

• Compiles prioritized NFAs for source-order alternation, greedy/reluctant quantifiers, bounded quantifiers, anchors, and quantified zero-width anchors without empty-cycle hangs.
• Evaluates DEFINE predicates and MEASURES over a classifier tape, including navigation around CLASSIFIER(), bounded navigation, and exact decimal/integer aggregation behavior.
• Emits empty matches once at each input position and always advances one row for all supported skip modes. Missing skip targets remain errors for non-empty matches.
• Traverses the contiguous universal match range directly for aggregates, avoiding per-match row-index materialization; pattern-variable aggregates retain their sparse symbol indexes.
• Processes one monotonically ordered partition at a time without retaining an unbounded closed-partition set or allocating a key tuple per row.
• Emits at most 1,024 rows per transfer block and resumes matching state across calls; upstream error blocks supersede buffered output.
• Checks cancellation/deadline state every 1,024 matcher transitions.
• Stores backtracking state in primitive choice/counter logs and enforces a non-configurable 64 MiB cap on their combined array payload before growth.
• Throws instead of truncating when a row, step, or retained-state limit is reached.

Resource-limit precedence is node hint, query option, server config, then default:

Limit	Query option	Hint key	Server config
Rows buffered per partition	maxRowsInMatchPartition	max_rows_in_match_partition	pinot.query.match.max.rows.per.partition
NFA transitions per start position	maxStepsPerMatchAttempt	max_steps_per_match_attempt	pinot.query.match.max.steps.per.attempt

Scope

v1 covers PARTITION BY, mandatory ORDER BY, MEASURES, ONE ROW PER MATCH, all four supported AFTER MATCH SKIP modes, pattern alternation/concatenation/quantifiers/anchors, navigation and classifier functions, and single-variable aggregates in measures.

Trino comparison

The supported v1 behavior and tests were compared against Trino at commit 19b4cebac145, especially its row-pattern query suite, aggregation suite, analyzer suite, and feature documentation.

Coverage added from that comparison includes:

• empty-match advancement for every supported skip mode, empty-match CLASSIFIER() nullability, and COUNT(classifier_measure) versus COUNT(*) semantics;
• FIRST/LAST/PREV/NEXT navigation around CLASSIFIER(), including out-of-match nulls and composition of non-zero logical and physical offsets;
• intrinsically running DEFINE expressions and rejection of FINAL in DEFINE;
• case-insensitive unquoted pattern variables, case-sensitive quoted variables, and separation of row-source aliases from the pattern-variable namespace;
• allocation-free universal aggregates over bounded mixed-null ranges, per-variable aggregates over empty/all-null/mixed-null input and repeated sparse pattern-variable rows, plus empty input; and
• deterministic multi-key ordering with mixed DESC/ASC keys.

This comparison targets Pinot's explicitly supported v1 subset and does not claim full Trino feature parity. ALL ROWS PER MATCH, SUBSET, PERMUTE, exclusions, WITHIN, aggregates in DEFINE, optional measures/order, and row-pattern window frames remain deferred and fail validation where applicable.

Rolling-upgrade boundary

MatchNode is plan oneof field 19. Older servers preserve that protobuf field as unknown but see the node oneof as unset (NODE_NOT_SET), so they cannot execute a dispatched MATCH_RECOGNIZE stage. Upgrade all servers before issuing these queries; there is no mixed-version fallback. Existing query node kinds remain wire-compatible and are unaffected.

Testing

• 434 focused common/planner/runtime invocations executed: 422 passed and 12 expected lite/v2-optimizer variants were skipped. Coverage includes validation, rel conversion, exchange planning, plan serialization, legacy-descriptor behavior, option precedence, navigation, sparse repeated-variable aggregates, composed logical/physical offsets, numeric precision, NFA behavior and zero-width-anchor termination, cancellation, retained-state bounds, output chunking, empty matches, operator errors, and merge/equivalence protection for every MatchNode semantic field.
• Local JaCoCo instrumentation confirms all 23 executable lines in PlanNodeMerger.visitMatch and all 12 in EquivalentStagesFinder.visitMatch are exercised, with no missed or partial lines in either visitor.
• MatchRecognizeIntegrationTest: 21/21 passed against a real two-server cluster through the full 63-module dependency reactor, including empty matches under all skip modes, nullable classifier measures and classifier navigation, RUNNING/FINAL validation, unquoted identifier and alias isolation, aggregate/null semantics, multi-key ordering, empty input, partition order, cross-segment isolation, and gated global execution without PARTITION BY.
• An independent review across backward compatibility, correctness/nulls, concurrency/state, architecture, performance, testing, API/naming, and process/scope found no unresolved blockers after fixing universal-aggregate hot-path allocation and quantified-anchor coverage.
• Fresh hosted CI completed 12/12 checks successfully for final head 3034c6d21fef; final Codecov reports 81.91882% patch coverage with 343 changed lines missing and 67.86% project coverage (+0.16% versus base), with all expected uploads present.
• Spotless, checkstyle, license formatting/checking, and git diff --check passed for all affected modules on JDK 25.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.12975% with 349 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.91%. Comparing base (3b6be9c) to head (d252451).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
.../query/runtime/operator/match/MatchExpression.java	58.28%	46 Missing and 27 partials ⚠️
.../query/planner/logical/RelToPlanNodeConverter.java	80.64%	15 Missing and 15 partials ⚠️
.../query/planner/logical/PlanNodeToRelConverter.java	61.64%	12 Missing and 16 partials ⚠️
...he/pinot/query/planner/explain/PlanNodeMerger.java	0.00%	23 Missing ⚠️
...pache/pinot/query/planner/plannode/RowPattern.java	77.88%	12 Missing and 11 partials ⚠️
.../pinot/query/runtime/operator/match/MatchTerm.java	72.83%	11 Missing and 11 partials ⚠️
.../pinot/query/validate/MatchRecognizeValidator.java	86.95%	4 Missing and 17 partials ⚠️
...he/pinot/query/runtime/operator/MatchOperator.java	87.12%	10 Missing and 7 partials ⚠️
...apache/pinot/query/planner/plannode/MatchNode.java	72.41%	6 Missing and 10 partials ⚠️
.../query/planner/logical/EquivalentStagesFinder.java	0.00%	12 Missing ⚠️
... and 21 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19311      +/-   ##
============================================
+ Coverage     67.83%   67.91%   +0.07%     
  Complexity     1450     1450              
============================================
  Files          3504     3517      +13     
  Lines        226581   228103    +1522     
  Branches      35804    36124     +320     
============================================
+ Hits         153699   154912    +1213     
- Misses        60754    60912     +158     
- Partials      12128    12279     +151     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (?)
java-25	67.91% <77.12%> (+0.07%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.91% <77.12%> (+0.07%)	⬆️
unittests	67.91% <77.12%> (+0.07%)	⬆️
unittests1	58.18% <77.12%> (+0.20%)	⬆️
unittests2	39.31% <0.65%> (-0.27%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Review of current head 5f27d70c.

The correctness, rolling-upgrade, and resource-behavior issues below should be addressed before merge. I did not duplicate the two existing unresolved threads about v2/lite rejection and _closedPartitionKeys cardinality.

Process follow-ups:

• This is a 56-file, 7.5K-line change spanning protobuf, planning, execution, configuration, and integration. Please either link the reviewed design/maintainer agreement for keeping it as one vertical slice, or split it into reviewable stacked PRs.
• Please link the sender-side-sorting TODO to a tracking issue.
• Both commits contain AI Co-Authored-By trailers; repository guidance says to omit those, so please rewrite the commit messages before merge.

[xiangfu0]

Addressed the process follow-ups on head b612fab43e:

• Expanded the PR description with the end-to-end design/atomicity rationale, rollback boundary, exact supported/unsupported modes, resource-limit precedence, and the required all-servers-first rolling-upgrade sequence. The design-review label remains so maintainer sign-off is still explicit before landing.
• Linked the sender-side sort TODO to [MSE] Support sender-side sorting for sort exchanges #19395.
• Rewrote the feature commits to remove both AI Co-Authored-By trailers.
• Replied to and resolved all 13 inline threads with the pushed fix and regression evidence.

Local validation passed: the 342-test focused suite, a 32-test matcher/operator rerun after the final retained-state changes, 13/13 two-server integration cases, the full 63-module test-compile reactor, Spotless, checkstyle, license checks, and diff hygiene. Fresh CI for this rewritten head is running.

[xiangfu0]

The rebased PR head is now 3034c6d21fef on upstream master at e6787e178645. The production tree is unchanged from the previously green c05a54b7d69b; this force-push adds a test-only follow-up from the pinned Trino comparison.

Coverage added in this follow-up:

• Field-by-field positive/negative tests for PlanNodeMerger.visitMatch and EquivalentStagesFinder.visitMatch, covering pattern definitions, pattern shape, measures, partitioning, ordering, skip mode/target, rows-per-match, node type, and input/base mismatch.
• The positive merger case observes recursively merged explain attributes (rows=2+3=5), so it verifies that the merged child is actually installed.
• Repeated-pattern-variable coverage for PATTERN (A B A) verifies that aggregates and logical navigation use sparse classifier rows rather than the contiguous match span.
• Runtime expression cases for default FIRST/LAST/PREV/NEXT offsets, nested PREV(RUNNING LAST(...)), composition of non-zero logical and physical navigation offsets, out-of-range nulls, SQL true/false/unknown DEFINE truth semantics, all-null aggregate identities, and defensive stored-type conversion branches.

The comparison remains pinned to Trino 19b4cebac145. It targets Pinot's supported v1 subset and does not claim full Trino feature or planner type parity.

Validation:

• Expanded focused common/planner/runtime matrix: 434 invocations; 422 passed and 12 expected lite/v2-optimizer variants skipped
• Planner merge/equivalence suites: 47/47; final strengthened PlanNodeMergerTest rerun: 14/14
• Core runtime matcher/expression/NFA suites: 74/74
• Two-server MatchRecognizeIntegrationTest: 21/21 on the unchanged production tree through the full 63-module JDK 25 reactor
• Local JaCoCo: all 23 executable lines in PlanNodeMerger.visitMatch and all 12 in EquivalentStagesFinder.visitMatch covered, with no missed or partial lines
• Spotless, checkstyle, license format/check, and git diff --check: clean after the final test change
• Independent final review: no remaining blockers or Trino-v1 test gaps for this follow-up

Fresh hosted CI is complete: 12/12 checks passed for 3034c6d21fef. Final Codecov reports 81.91882% patch coverage with 343 changed lines missing and 67.86% project coverage (+0.16% versus base); all expected uploads are present and no coverage flag is unknown.