Repository navigation
Configure OPEN_STRUCT indexes the way every other column is configured - #19736
Merged
raghavyadav01 merged 4 commits intoOct 6, 2026
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19736 +/- ##
============================================
+ Coverage 68.22% 68.25% +0.02%
Complexity 1450 1450
============================================
Files 3520 3520
Lines 228899 228935 +36
Branches 36305 36318 +13
============================================
+ Hits 156171 156261 +90
+ Misses 60489 60395 -94
- Partials 12239 12279 +40
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…umn does A dictionary is enabled on an ordinary column by putting it in `indexes`. An OPEN_STRUCT key could not be configured that way: fromFieldConfig, which builds a key's index configs, skipped the dictionary type outright and took the answer from encodingType alone, and the validator then rejected an indexes.dictionary entry that did not already agree with it. So the two spellings disagreed depending on which kind of column you were configuring, and the key spelling failed silently -- you got a raw key and no diagnostic. An enabled inverted index hid half of it, since it forces a dictionary on its own. A key whose only index is a range one had nothing to rescue it: the dictionary entry was ignored, the key stayed raw, and the range index was asked for over a dictionary that did not exist. That case is testARangeOnlyKeyCanAskForADictionaryUnderIndexes, and it is the one that fails when the production change alone is reverted. indexes.dictionary can only turn a dictionary *on*. Turning one off stays with encodingType, so a `disabled` entry contradicting the column's own encoding is still a contradiction and still refused rather than quietly winning. The forward index follows the dictionary decision rather than the declared encoding, for the reason it already followed encodingType: a key whose dictionary was enabled this way needs a dict-encoded forward index, or a reload rebuilds one over a dictionary that is not there. 369 tests green across the OPEN_STRUCT, config and loader suites.
An OPEN_STRUCT column splits into materialized per-key child columns and
one shared blob column holding every key that was not materialized. The
per-key children already accept a FieldConfig each, so a key can ask for
a dictionary, a range index, a bloom filter and so on. The blob column
could not: index loading skipped it outright, so the only way to search
the keys living inside it was a full scan.
This adds a sparseFieldConfig to OpenStructIndexConfig. It is shaped
like the FieldConfig of any other column, and it configures the blob the
same way a per-key entry configures a child:
"openStruct": {
"sparseFieldConfig": {
"name": "props$__sparse__",
"indexes": { "json": {} }
}
}
The blob is always RAW regardless of what the config asks for -- it is a
serialized document per row, so a dictionary over it would be a
dictionary of whole documents and buys nothing. The existing
sparseJsonIndex flag still works and now means exactly indexes.json={},
so the two spellings converge on one code path instead of two.
Adds four tests covering a JSON index built on the blob through either
spelling, and keeps the existing per-key cases green.
The sparse fast path in MapFilterOperator asks the OPEN_STRUCT data source for the blob's JSON index, and that lookup only ever returned the standard one. An ordinary column does not stop there: tryJsonIndex falls back to the composite JSON index, which reads as a JsonIndexReader and answers the same predicates. So a blob given a composite JSON index built it and was then never asked. The key fell back to a full scan, silently costing exactly what the index was configured to avoid -- no error, no plan difference a reader would notice, just the slow path. Make the blob lookup do what the column lookup already does. The index ships as a plugin and is absent from most deployments, so it is resolved by id and the fallback is inert where nothing registered it.
The standard JSON index indexes every path, so asking it about any key is safe. An index that indexes a configured subset of paths is not: asked about a path it never indexed, it answers with an empty bitmap, and an empty bitmap is indistinguishable from "no document matches this". The query returns zero rows, the plan says the index served the filter, and nothing anywhere says the answer is wrong. `JsonIndexReader.isPathIndexed` already exists to answer exactly this, and defaults to true so a fully-indexed reader is unaffected. Nothing was calling it. Both JSON fast paths now do -- the one over a column and the one over an OPEN_STRUCT sparse blob -- and a key the index does not cover falls back to the scan, which is slower and right. The two existing tests that expect the index to serve now stub `isPathIndexed`: Mockito answers false for an unstubbed boolean, interface default or not.
raghavyadav01
force-pushed
the
openstruct-per-key-index-config
branch
from
October 2, 2026 04:49
50da4bf to
8f89a8d
Compare
xiangfu0
added a commit
to pinot-contrib/pinot-docs
that referenced
this pull request
Oct 6, 2026
Docs follow-up for apache/pinot#19736. Documents enabling a dictionary for a RAW materialized key through indexes.dictionary and configuring the shared sparse blob through sparseFieldConfig, including its raw encoding and compatibility with sparseJsonIndex.\n\nValidation: full docs validator passed (existing non-fatal anchor warnings only); git diff --check passed. Co-authored-by: Xiang Fu <xiangfu@Xiang-mac-mtv-2.local>
Contributor
|
Docs follow-up: pinot-contrib/pinot-docs#1086 (merged). The OPEN_STRUCT reference and ingestion guide now cover per-key indexes.dictionary and sparseFieldConfig for the shared sparse column. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR flow
Enables dictionary config via indexes for OPEN_STRUCT keys and allows index settings on the sparse blob column, making its configuration uniform with other columns.
AI-generated · Green: added · Yellow: modified · Red: removed · Gray: existing
Diff evidence
An OPEN_STRUCT column is split at segment build time into materialized
per-key child columns plus one shared blob column holding every key that was
not materialized. Both halves were configurable in their own idiosyncratic
way rather than the way the rest of Pinot is configured. Two fixes.
1. A key can ask for its dictionary under
indexesA dictionary is enabled on an ordinary column by putting it in
indexes. AnOPEN_STRUCT key could not be:
FieldIndexConfigsUtil#fromFieldConfig, whichbuilds a key's index configs, skipped
DICTIONARY_IDoutright and took theanswer from
encodingTypealone — and the validator then rejected anindexes.dictionaryentry that did not already agree with it. The same JSONmeant different things depending on which kind of column you wrote it on, and
on a key it failed silently: you got a raw key and no diagnostic.
An enabled inverted index hid half of this, because it forces a dictionary on
its own. A key whose only index is a range one had nothing to rescue it — the
dictionary entry was ignored, the key stayed raw, and the range index was then
asked for over a dictionary that did not exist.
indexes.dictionarycan only turn a dictionary on. Turning one off stayswith
encodingType, so adisabledentry that contradicts the column's ownencoding is still a contradiction and is still refused rather than quietly
winning. The forward index now follows the dictionary decision rather than the
declared encoding, for the same reason it already followed
encodingType: akey whose dictionary was enabled this way needs a dict-encoded forward index,
or a reload rebuilds one over a dictionary that is not there.
2. The sparse blob column can carry index settings
The per-key children each accept a
FieldConfig. The blob column could not —index loading skipped it before it reached the index-config builder, so the
only way to evaluate a predicate against a key living inside the blob was a
full scan over every document.
OpenStructIndexConfiggains an optionalsparseFieldConfig. It is a plainFieldConfig, shaped exactly like the one you would write for any othercolumn:
The blob is forced to
RAWwhatever the config asks for. Each value is aserialized document for one row, so a dictionary over it would be a dictionary
of whole documents — it dedupes nothing and costs a round trip per lookup.
The existing
sparseJsonIndex: trueflag keeps working and is now defined asexactly
indexes.json = {}, so the two spellings converge on one code pathinstead of being maintained separately.
Testing
OpenStructPerKeyIndexReloadTestis now 9 tests. The one that fails when thefirst production change alone is reverted is
testARangeOnlyKeyCanAskForADictionaryUnderIndexes. Two new cases build aJSON index on the blob, via
sparseJsonIndexand viasparseFieldConfignaming the blob column directly.
369 tests green across the OPEN_STRUCT, config and loader suites; existing
per-key coverage (dictionary, range, inverted, bloom) unchanged.
Compatibility
Additive. A table config with no
sparseFieldConfigbehaves exactly asbefore,
sparseJsonIndexproduces the index it always did, and the previous10-arg
OpenStructIndexConfigcreator is deprecated and delegating, soexisting configs deserialize unchanged.