Prune segments from column metadata instead of materializing them

[xiangfu0]

What

ColumnValueSegmentPruner asks each segment for a column's DataSource and then reads nothing from it but the DataSourceMetadata — data type, min/max, partition function. Reaching that metadata through getDataSource forces a segment that builds its columns lazily to construct the whole ColumnIndexContainer — every index reader for the column — for a segment it is about to discard.

This adds IndexSegment#getDataSourceMetadata(String, Schema), which names what the caller actually wants and defaults to getDataSource(column, schema).getDataSourceMetadata(), so no existing implementation has to change. ImmutableSegmentImpl overrides it to answer from column metadata: ImmutableDataSourceMetadata already delegates to ColumnMetadata and holds no readers, so nothing needs building. A column the segment does not have still falls through to the data source, which is where the schema-driven default and virtual columns are created.

The pruner keeps its per-segment data-source cache for mutable segments, whose metadata is not derivable without the data source.

Why

ValueBasedSegmentPruner#prune is a serial loop over every segment the server holds, on the query thread — it implements only the two-arg prune, so SegmentPruner's three-arg default silently discards the ExecutorService (BloomFilterSegmentPruner, by contrast, does override it and prunes in parallel).

Combined with lazy column materialization (#19477) that means the pruner materializes a column per held segment, and on a tiered-storage external table each materialization parses a Parquet footer. Measured on a production server holding 44,780 segments, this dominated a query that timed out at 300 s for a filter matching 1,839 segments — the pruner was doing tens of thousands of footer parses, in series, for segments it then discarded.

The win is not limited to lazy mode: building an index container to read min/max is wasted work in any configuration.

Tests

ImmutableSegmentImplTest#testDataSourceMetadataDoesNotMaterializeTheColumn builds a lazy segment with a mocked ColumnMaterializer, reads the metadata, and asserts verifyNoInteractions(materializer) — then shows that asking for the data source does materialize, and that the two agree. It fails when the override is removed.

51 pruner tests in pinot-core and 35 segment tests in pinot-segment-local pass; spotless, checkstyle and license clean.

Stack

Based on #19486. Review this PR against its base for this layer's changes.

Current open chain; #19480 (lazy index-size storage) is already merged, and #19476 is absorbed into #19473.

1. Share FieldSpecs and repeated values across segment metadata loads #19473 shared FieldSpecs, canonical default-null values, and interned strings
2. Delegate immutable DataSourceMetadata to ColumnMetadata instead of snapshotting it #19474 delegating immutable DataSourceMetadata
3. Fold PhysicalColumnIndexContainer's IndexTypeMap into a presence mask and a dense reader array #19475 presence-mask index container
4. Materialize immutable-segment columns lazily behind an opt-in instance config (default off) #19477 opt-in lazy column materialization
5. Slim ColumnMetadataImpl to 72 bytes and derive the per-segment Schema lazily #19478 slim ColumnMetadataImpl and lazy per-segment Schema
6. Store numeric column min/max as primitives instead of boxed Comparables #19479 primitive numeric min/max
7. Hold segment column metadata in sorted arrays and derive the map on demand #19481 sorted-array column metadata storage
8. Stop the segment preprocess from building a Schema per segment #19486 segment preprocessing without building a Schema
9. Prune segments from column metadata instead of materializing them #19511 segment pruning directly from column metadata

[codecov-commenter]

Codecov Report

❌ Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.76%. Comparing base (9a59744) to head (490463e).

Files with missing lines	Patch %	Lines
...l/indexsegment/immutable/ImmutableSegmentImpl.java	33.33%	1 Missing and 1 partial ⚠️
...ot/core/query/pruner/ColumnValueSegmentPruner.java	88.88%	0 Missing and 1 partial ⚠️
...ava/org/apache/pinot/segment/spi/IndexSegment.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                              Coverage Diff                               @@
##             xiangfu0/data-3221-10-physical-column-names   #19511   +/-   ##
==============================================================================
  Coverage                                          67.76%   67.76%           
  Complexity                                          1450     1450           
==============================================================================
  Files                                               3494     3494           
  Lines                                             225406   225408    +2     
  Branches                                           35599    35595    -4     
==============================================================================
+ Hits                                              152738   152746    +8     
- Misses                                             60619    60626    +7     
+ Partials                                           12049    12036   -13     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.76% <71.42%> (+<0.01%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.76% <71.42%> (+<0.01%)	⬆️
unittests	67.76% <71.42%> (+<0.01%)	⬆️
unittests1	57.82% <71.42%> (+0.02%)	⬆️
unittests2	39.49% <14.28%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.