OPENNLP-1897: Add document-scoped term vector layer - #1212
Merged
Conversation
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 16, 2026
krickert
added a commit
that referenced
this pull request
Aug 16, 2026
mawiesne
changed the title
krickert
added a commit
that referenced
this pull request
Aug 26, 2026
…nto the gRPC helper base # Conflicts: # opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/termvector/TermVectorNormalizedExampleTest.java # opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/termvector/TermVectorPipelineTest.java
krickert
added a commit
that referenced
this pull request
Aug 30, 2026
krickert
added a commit
that referenced
this pull request
Sep 1, 2026
Contributor
|
@krickert Could you rebase this PR onto main? I think we can make progress with this one. |
mawiesne
added
java
Contributor
Author
I'm on it now. Thanks so much!! This is exciting. |
…quency, offsets) A new opennlp.tools.termvector package aggregates the document token layer into a document-scoped layer of TermVector records for index consumers, without touching the opennlp.tools.document container. TermVector carries the term string, the occurrence count, and the occurrence spans in original text coordinates. It comes in two shapes: full (one span per occurrence) and scoring-only (counts only, no offset storage). TermVectorAnnotator implements DocumentAnnotator: it requires Layers.TOKENS and provides its own opennlp:term-vectors key. Term identity is delegated, never analyzed: without a normalizer the token's covered text groups as-is; with an OffsetAwareNormalizer the document text is normalized once and each token span is mapped through the alignment, so tokens differing only by a normalization fold (case, eszett expansion, collapsed whitespace) group together while every emitted occurrence span still points into the original text.
…gation by mode - Delegate the two convenience constructors of TermVectorAnnotator through this(...) instead of repeating the field assignments, so the no-arg form is defined as FULL mode and the normalizer-only form as normalizer plus FULL. - Drop the requireMode and requireNormalizer helpers and do the null checks inline in the two canonical constructors, matching the argument-validation style used elsewhere in the module and keeping the thrown message next to the parameter it guards. - Split annotate into fullVectors and countVectors, one per Mode, so the scoring-only path no longer carries a null span map as a mode sentinel and the per-term branch inside the emit loop disappears. Each helper is documented and returns the annotations in first-occurrence order, which is the ordering the tests pin. - Make termOf an instance method, since it is now only reached from the two mode helpers and no longer needs to be static to be shared. - Tighten the TermVector class javadoc: the two shapes are told apart by whether spans() is empty, stated once, without the redundant aside about a flag, and the closing sentence now names the invariant instead of repeating the shape list. - Tighten the TermVectorAnnotator class javadoc the same way, and fix the termOf parameter doc that read "or null likewise" to spell out the condition. - Remove the javadoc references to AlignmentTest and DocumentPipelineExampleTest from the test fixtures. Those tests are not part of the contract under test here and the pointers go stale as soon as either file moves. - Add a DigitDeletingNormalizer fixture and a pinning test for a token the normalizer deletes entirely. It groups under the empty term rather than being dropped, which the class javadoc promises but nothing exercised. - Parameterize the frequency rejection test over 0, -1 and Integer.MIN_VALUE instead of only 0, so the guard is pinned across the whole illegal range. - Fix the span assertion in TermVectorPipelineTest. It compared Span.getCoveredText against an equivalent subSequence of the same text, which holds for any span, so it now asserts that the covered text equals the vector's own term.
…mple Adds a term vectors section to the document container chapter, citing TermVectorPipelineTest#testTokenizerAndTermVectorPipeline as the pin for the programlisting and covering the scoring-only mode.
The documented normalization workflow could not be built with any commonly wanted normalizer: TextNormalizer.Builder.buildAligned() rejects caseFold, nfc, nfkc, and accent folding, so the only OffsetAwareNormalizer chains the annotator accepted were the per-code-point folds. The restriction is unnecessary here because the occurrence spans the annotator emits are always the token's own span in the original text; the normalized text is used only as the term key. Add TermVectorAnnotator(CharSequenceNormalizer) and TermVectorAnnotator(CharSequenceNormalizer, Mode) as the general path: the normalizer is applied to each token's covered text to produce the term, the span stays the token's original span, and any normalizer works (case fold, NFC, accent fold, stemmer-backed). The OffsetAwareNormalizer constructors keep their whole-document aligned behavior unchanged; their javadoc now points at the plain-normalizer constructors as the general path. Tokens that normalize to the empty string still collapse into one empty term on both paths, pinned by matching tests. Red evidence: the new tests cannot compile against the old API (no suitable constructor found for TermVectorAnnotator(CharSequenceNormalizer)), so the tests and the fix land together in this commit. The negative pin that builder().caseFold().buildAligned() throws IllegalStateException already exists in AlignedNormalizerPipelineTest and is not duplicated.
…ted example Extend the term vector section of the manual with the plain-normalizer path: a whitespace tokenizer plus a shipped case folder built by TextNormalizer.builder().caseFold().build(), folding "Word word WORD" into one term with three occurrence spans that stay the tokens' original spans. The example lives in opennlp-runtime because the shipped folds do, and TermVectorNormalizedExampleTest#testCaseFoldedTermsKeepOriginalSpans asserts the behavior shown in the listing.
…nnotator, share the space-split test fixture, pin supplementary-plane offsets
… term vector layer An empty string is no term: it cannot be queried and its token stays accounted for in the token layer. Omission also keeps one semantic for term vectors across the library and its search consumers.
krickert
force-pushed
the
OPENNLP-1897-term-vectors
branch
from
September 1, 2026 07:54
f35d20a to
586e573
Compare
krickert
marked this pull request as ready for review
September 2, 2026 00:35
krickert
added a commit
that referenced
this pull request
Sep 4, 2026
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a document-scoped term vector layer for index consumers (OPENNLP-1897).
TermVector: one entry per distinct term, in two shapes: full (one span per occurrence, spans always in original text coordinates) or scoring-only (counts without offset storage). The shape invariant is validated.TermVectorAnnotator: rolls the token layer up into theopennlp:term-vectorslayer. Term identity is pluggable: covered text as-is, a plain per-tokenCharSequenceNormalizer(case/NFC/accent folds stemmer-backed normalizers), or anOffsetAwareNormalizerapplied to the whole document through its alignment. Tokens normalized to nothing group under the empty term rather than being dropped.TermVectorPipelineTestandTermVectorNormalizedExampleTest.Stacks on #1182 (OPENNLP-1888 document container); the shared foundation commits will drop out when that lands. We'll keep this in draft until #1182 merges.
This functionality is available in the sandbox branch for the gRPC server.
Consumer story: this is the aggregation a BM25/lexical index asks of the analysis chain; the gRPC server work (OPENNLP-1833) will expose the layer.
Thank you for contributing to Apache OpenNLP.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.