Delegate immutable DataSourceMetadata to ColumnMetadata instead of snapshotting it - #19474
Open
xiangfu0 wants to merge 1 commit into
Open
Conversation
This was referenced Sep 6, 2026
Open
xiangfu0
force-pushed
the
xiangfu0/data-3221-3-datasource-adapter
branch
from
September 9, 2026 01:56
9590789 to
2b73f0d
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## xiangfu0/data-3221-2-intern-parse #19474 +/- ##
=======================================================================
+ Coverage 57.80% 67.71% +9.91%
- Complexity 1 1450 +1449
=======================================================================
Files 2688 3490 +802
Lines 164376 225025 +60649
Branches 26688 35529 +8841
=======================================================================
+ Hits 95023 152384 +57361
+ Misses 61319 60614 -705
- Partials 8034 12027 +3993
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
xiangfu0/data-3221-3-datasource-adapter
branch
from
September 11, 2026 09:10
2b73f0d to
3eb0c08
Compare
xiangfu0
force-pushed
the
xiangfu0/data-3221-3-datasource-adapter
branch
2 times, most recently
from
September 13, 2026 01:13
87b1c6a to
9855655
Compare
xiangfu0
removed this pull request from stack #19484
September 13, 2026 01:16
xiangfu0
added this pull request to stack #19540
September 13, 2026 01:17
…apshotting it ImmutableDataSource and ImmutableMapDataSource each wrapped the column's ColumnMetadata in a private DataSourceMetadata that copied ten fields (field spec, sorted flag, doc/value counts, MV width, cardinality, min/max, partition function and partitions) into the adapter object. At ~56 bytes of copied fields per column, a server holding tens of thousands of wide segments (1000+ columns) retains tens of megabytes of duplicated metadata that is already reachable from the ColumnMetadata the segment keeps anyway. Replace the snapshot with a one-field adapter that delegates every accessor to the ColumnMetadata reference. The adapter drops from ~72 to 16 bytes per column while keeping the DataSourceMetadata contract exactly: - getMaxNumValuesPerMVEntry() still returns -1 for single-value columns (ColumnMetadata canonicalises it to 0), as DataFetcher, MVScanDocIdIterator and PinotSegmentColumnReader rely on; - getNumDocs()/getNumValues() map to getTotalDocs()/getTotalNumberOfEntries(); - ImmutableDataSource leaves getMaxRowLengthInBytes() at the DataSourceMetadata default (-1) rather than exposing ColumnMetadata's computed row length; - ImmutableMapDataSource keeps its unconditional isSorted() == false and its UnsupportedOperationException from getMaxRowLengthInBytes(). Delegating is observationally identical to snapshotting because every ColumnMetadata handed to these data sources (ColumnMetadataImpl, SimpleColumnMetadata, EmptyColumnMetadata) is immutable once built; the only mutator, ColumnMetadataImpl#addIndexSize, is not visible through DataSourceMetadata. Both adapters are private nested classes with no external references, DataSourceMetadata is never serialized, and no public signature, on-disk format or REST segment-metadata JSON changes, so the change is safe across mixed-version clusters. Adds ImmutableDataSourceTest and ImmutableMapDataSourceTest pinning every accessor against the backing ColumnMetadata for SV, MV, virtual, MAP and stats-unavailable columns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
xiangfu0
force-pushed
the
xiangfu0/data-3221-3-datasource-adapter
branch
from
September 13, 2026 04:14
9855655 to
caac249
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ImmutableDataSource.ImmutableDataSourceMetadatacopied ten fields out ofColumnMetadatainto a per-column snapshot, about 56 bytes each. It now holds a singleColumnMetadatareference and delegates, which keeps the semantics identical becauseColumnMetadatais immutable after construction:getNumDocs->getTotalDocs,getNumValues->getTotalNumberOfEntries, andgetMaxNumValuesPerMVEntrypreserves the-1contract for single-value columns.ImmutableMapDataSourcegets the same treatment, keeping its unconditionalisSorted() == falseand itsgetMaxRowLengthInBytesoverride.DataSourceMetadatais never serialized and no public signature changes.Tests
New
ImmutableDataSourceTestandImmutableMapDataSourceTest: everyDataSourceMetadatagetter equals the correspondingColumnMetadatagetter for a single-value INT column with min/max and partitions, a multi-value STRING column, and a virtual-column-style spec.Why
A server keeps one metadata object graph per (segment, column) for as long as the segment is loaded, so on wide tables the per-column footprint decides how many segments a server can hold. Measured end to end on a 1000-column segment, this series takes the heap retained at load from 4.08 MB to 0.175 MB per segment (4,080 to 174 bytes per column), with a fully compacting collector on both sides. No on-disk format change, the
/tables/{table}/segments/{segment}/metadataJSON stays byte-identical, and every public and SPI signature keeps working.Stack
Based on #19473. Review this PR against its base for this layer's changes.
Current open chain; #19480 (lazy index-size storage) is already merged, and #19476 is absorbed into #19473.