Add plugin SPI for dynamic field-type inference and dynamic-template types - #22607
Add plugin SPI for dynamic field-type inference and dynamic-template types#22607naykudev wants to merge 11 commits into
Conversation
Introduce a generic extension point so mapper plugins can participate in dynamic mapping for their own field types, without core hard-coding any plugin-specific type knowledge. A single hook, tryPluginInference(), fires in innerParseObject() before the token-type switch (covering arrays, objects, and scalars). It offers an unmapped field to two plugin mechanisms: - DynamicFieldTypeInferencer: inspects the buffered value and returns a mapping config to claim the field, or null to pass. - DynamicTemplateTypeHandler: backs a plugin-registered match_mapping_type (validated against the plugin registry when it is not a builtin XContentFieldType), completing the template config before the TypeParser builds the mapper. isConfigComplete() gates eager index-creation validation. Both receive a FieldValueParserSupplier, which hands out a fresh XContentParser over the buffered field bytes on demand — lazy, no boxing, preserves JSON fidelity. If no plugin claims the field, the buffered bytes are replayed through the existing path, so all current behavior is preserved and the hook fast-exits when no plugin is registered. Plugins register via new MapperPlugin SPI methods collected by IndicesModule into MapperRegistry. All new types are @experimentalapi. Signed-off-by: ved naykude <vnaykude@amazon.com>
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 840945c. ⛔ Hard block: Issues at Medium severity or above will block this PR from merging.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
9eaca9f to
6f6a195
Compare
PR Reviewer Guide 🔍(Review updated until commit dbd28f0)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to dbd28f0 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 1e88a41
Suggestions up to commit 8301dbf
Suggestions up to commit a42ac75
Suggestions up to commit b31f7a0
Suggestions up to commit 000d534
|
Per OpenSearch triage: instead of taking the first plugin claim by registration/load order, DocumentParser now consults ALL registered plugin template types and ALL inferencers for an unmapped field and throws if more than one claims it: - Two plugin dynamic-template types matching the same field -> throw. - Two field-type inferencers claiming the same field -> throw. Zero claims still fall through to existing behavior; exactly one is used as before. The exception names the conflicting claimants so the misconfiguration is clear. Adds PluginInferenceConflictTests covering both conflict paths and the single-match (no-throw) case. Signed-off-by: ved naykude <vnaykude@amazon.com>
6f6a195 to
7863ebc
Compare
|
Persistent review updated to latest commit 7863ebc |
|
Persistent review updated to latest commit 6dae7af |
|
❕ Gradle check result for 6dae7af: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #22607 +/- ##
============================================
- Coverage 71.42% 71.38% -0.05%
+ Complexity 76853 76815 -38
============================================
Files 6148 6148
Lines 358003 358205 +202
Branches 52179 52216 +37
============================================
- Hits 255718 255696 -22
- Misses 81972 82152 +180
- Partials 20313 20357 +44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit 2ae39c1 |
|
❌ Gradle check result for 2ae39c1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Cover the previously-untested fall-through and error paths in DocumentParser.tryPluginInference and RootObjectMapper eager validation: - Throwing inferencer is caught and the field falls through to normal inference (not a parse failure). - Inferencer returning a config with no "type", or an unknown type with no registered TypeParser, falls through instead of failing. - A template handler that reports its config complete but throws during adjustMappingConfig is treated as non-fatal at index creation (deferred). Adds PluginInferencerEdgeCaseTests and a throwing-handler case in PluginDynamicTemplateTests. Improves patch coverage on the new branches. Signed-off-by: ved naykude <vnaykude@amazon.com>
2ae39c1 to
000d534
Compare
|
Persistent review updated to latest commit 000d534 |
|
❌ Gradle check result for 000d534: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
| try { | ||
| // No document is available at index creation, so the supplier's get() throws. A complete | ||
| // config never reads the field value, so this is a no-op normalization (e.g. type injection). | ||
| handler.adjustMappingConfig(fieldTypeConfig, FieldValueParserSupplier.withoutValue()); |
There was a problem hiding this comment.
Can you help me understand why do we need to adjust mapping configuration when config is complete ?
There was a problem hiding this comment.
"Complete" only means the dimension is known, not that the config is buildable. The "type" is implied by match_mapping_type: "knn_vector" so users usually omit it, but the TypeParser cannot build a mapper without it, so we inject it even for a complete config. Example: mapping { "dimension": 128 } is complete but has no type, so it becomes { "type": "knn_vector", "dimension": 128 }. The isConfigComplete check only guards whether we open the parser to infer the dimension. Added a short comment with this example.
- Restore public DynamicTemplate.parse(name, conf) to fail fast on an unknown match_mapping_type (pre-plugin-SPI behavior); the plugin path uses the registry-aware overload. No API change. - Remove the always-false handler-null guard in validatePluginDynamicTemplate (the plugin type is registry-validated at parse time). - Use proper log4j imports instead of fully-qualified names; reword the deferred-validation warning; extract the __dynamic__ template-name prefix to a constant; drop a stray inline comment. - Update tests to the registry-aware parse and the restored strict behavior. Signed-off-by: ved naykude <vnaykude@amazon.com>
|
Persistent review updated to latest commit b31f7a0 |
Remove plugin-specific examples (knn_vector, dimension) from the core dynamic-mapping SPI docs and comments so the interfaces stay generic. Also validate a plugin match_mapping_type against the registry before assigning it as the plugin match type. Signed-off-by: ved naykude <vnaykude@amazon.com>
|
Persistent review updated to latest commit a42ac75 |
|
❌ Gradle check result for a42ac75: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 8301dbf |
|
❕ Gradle check result for 8301dbf: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
|
@naykudev there has been some hard blockers called out here: #22607 (comment) can we fix them or respond to them. |
When two mapper plugins register the same dynamic template type, name both plugins in the error so the misconfiguration is easy to trace. Signed-off-by: ved naykude <vnaykude@amazon.com>
|
Persistent review updated to latest commit 1e88a41 |
Medium (DocumentParser:1186, plugin inferencer hook): this is by design and consistent with the trusted-plugin model. Plugins already run in-process with full access to document content (analyzers, ingest processors, mapper parsers), so this hook does not expand the trust boundary. It only fires for unmapped fields when no dynamic template matches, and only for plugins the cluster admin explicitly installed. There is no untrusted-plugin threat model in OpenSearch to defend against here, so no code change is warranted. Requesting a maintainer apply skip-diff-analyzer after review. Low (DynamicTemplateTests:58, deferred match_mapping_type validation): the relaxation is intentional and scoped. An unknown type is still rejected, just against the plugin registry (known builtin types plus registered plugin types) rather than only the builtin enum. Typos in a plain builtin type still fail. This is required so a plugin type like knn_vector is accepted. The public 2-arg DynamicTemplate.parse still fails fast on any unknown type, so the pre-SPI behavior is preserved where no registry is available. |
Wrap the plugin-inference body in a try/finally so path slots added by getDynamicParentMapper are released even when buffering or an ambiguous claim throws, preventing ContentPath corruption for later fields in the same document. Signed-off-by: ved naykude <vnaykude@amazon.com>
|
Persistent review updated to latest commit dbd28f0 |
|
❌ Gradle check result for dbd28f0: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Description
This PR introduces a generic core interface and plugin SPI that lets any mapper plugin participate in dynamic mapping for its own field types. Core defines the extension points (
DynamicFieldTypeInferencer,DynamicTemplateTypeHandler,FieldValueParserSupplier) and the registration SPI onMapperPlugin; core itself only detects the JSON value and delegates the "is this mine, and what type is it" decision to whichever plugin registered. No plugin-specific type knowledge lives in core.The k-NN plugin is simply the first consumer of this SPI (auto-mapping vector fields and
match_mapping_type: "knn_vector"dynamic templates, in a companion PR), but the interface is deliberately generic — e.g. a geospatial plugin could register an inferencer that claims{"lat": .., "lon": ..}objects asgeo_pointthrough the exact same SPI, with zero further core changes.Motivation. Today dynamic mapping can only produce core's built-in field types. When an unmapped numeric array arrives, core parses it element-by-element and maps it as
float— it never looks at the array as a whole, and a plugin has no way to claim it. Likewise,match_mapping_typeonly accepts the built-inXContentFieldTypevalues, so a plugin type can't be targeted by a dynamic template. This change gives plugins a single, generic seam for both.What it adds (all
@ExperimentalApi):DynamicFieldTypeInferencer—inferFieldType(FieldValueParserSupplier) → Map<String,Object> | null. Called for an unmapped field with no matching template; the plugin inspects the value and returns a mapping config to claim it, ornullto pass.DynamicTemplateTypeHandler— backs a plugin-registeredmatch_mapping_type(validated against the plugin registry when it isn't a built-inXContentFieldType).adjustMappingConfig(...)completes the template config before theTypeParserbuilds the mapper;isConfigComplete(...)gates eager index-creation validation.FieldValueParserSupplier— hands the plugin a freshXContentParserover the buffered field bytes on eachget(). Lazy (no parser allocated unless the plugin reads it), preserves JSON fidelity, no boxing. Core stays free of any deserialized-representation contract.MapperPluginSPI methodsgetDynamicFieldTypeInferencers()andgetDynamicTemplateTypes(), collected intoMapperRegistrybyIndicesModuleat startup (same pattern asgetMappers()).How it hooks in. A single hook,
tryPluginInference(), fires inDocumentParser.innerParseObject()before the token-type switch — one placement covering arrays, objects, and scalars, so the SPI is genuinely generic. Resolution order: explicit mapping → plugin template → standard template → plugin inferencer → existing per-element fallback. If no plugin claims the field, the buffered bytes are replayed through the original path, so all existing behavior is preserved. When no plugin registers an inferencer or template type, the hook fast-exits before any buffering — zero overhead for clusters without such a plugin.Ambiguity is a hard error, not a silent pick. Rather than taking the first claim by registration/load order, core consults all registered plugin template types and all inferencers for an unmapped field and throws a clear
MapperParsingException(naming the conflicting claimants) if more than one claims it. Zero claims fall through as before; exactly one is used. Duplicatematch_mapping_typeregistration across plugins is rejected at node startup.Backwards compatibility. Strictly additive. The hook only fires for unmapped fields, only when a plugin is registered; existing dynamic templates, explicit mappings, and per-element inference are unchanged. All new types are
@ExperimentalApi.Scope note. This is the core SPI only. The k-NN implementation (inferencer +
knn_vectortemplate handler) is a companion PR in the k-NN repo and depends on this change; it cannot build against core until this merges and a snapshot publishes.Testing
PluginDynamicTemplateTests(28) — parsing, index-creation validation (registered/unregistered/typo/complete/incomplete/{name}placeholder), inference claims/declines for scalars/strings/booleans, precedence (explicit beats inference, builtin fallback), and fast-path exit.PluginInferenceConflictTests(3) — two inferencers claiming the same field throw; two plugin templates matching the same field throw; a single match resolves without throwing.IndicesModuleTests(+3) — plugins register inferencers/template types correctly; duplicate template-type registration throws at startup; empty by default.DynamicTemplateTests— plugin-type parse/serialization.DocumentParserTests(130) pass — regression guard for the changed hot path.All server mapper/indices suites green;
spotlessJavaCheckpasses.Related Issues
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.