OPENNLP-1888: Document annotation container with typed offset-anchored layers - #1182
Conversation
4cd9beb to
7e65aad
Compare
|
This was proposed about a week ago. Discussions pointed to this shape - I feel like it's a great direction as a lot of research went into landing this shape:
|
|
I think that this needs to wait a bit more until a few more people ump into the discussion: https://lists.apache.org/thread/jwxxjkc2b0dqn4rwjvt1t7cdf056gqhp |
7e65aad to
b920e43
Compare
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
b920e43 to
3ab6920
Compare
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
Review of PR #1182 — OPENNLP-1888: Document annotation container with typed offset-anchored layersBlocking
Three copies of the sentence/token walk loop.
Minor
ProcessYou have eight other open PRs (#1191, #1190, #1167, #1166, #1165, #1155, #1154, #1152). The duplication, |
|
Thanks for taking the time I will address all of these within a few hours |
819070c to
5ba76d6
Compare
|
All of it is addressed. The branch is rebased on current main. BlockingZero-length mention. Interface Javadoc. The copy-on-add and sharing narrative moved to
Three copies of the walk, and
The example contradicting the contract. The MinorDone: hoisted Reversed Entity type in two places. The annotation value is now the single source. Spans are constructed untyped and the Javadoc says so, on both Extensibility. All six adapters are Since then a second pass folded the three copies of the The document package runs 65 tests, none skipped. |
…yers over the original text Adds opennlp.tools.document to opennlp-api: Document (immutable, copy-on-add layer container over the original text), Annotation (a typed value on a Span), LayerKey (open, typed layer identity), and DocumentAnnotator (pipeline step declaring the layers it requires and provides). DocumentAnalyzer assembles annotators into a pipeline validated at build time. Standard keys in Layers cover sentences, tokens, part-of-speech tags, and entities, populated through thin adapters over the existing SentenceDetector, Tokenizer, POSTagger, and TokenNameFinder interfaces, which stay the primary API for single-task use and are unchanged. All spans refer to the text as supplied. No new dependencies.
…ntainer, javadoc precision pass
…rom missing layers, validate providers at build time
… adaptive data on failure The lemmatizer adapter now slices tokens and tags per sentence like its POS and name-finder siblings, so lemmatization decisions never cross a sentence boundary, and it declares the sentence layer as required. The POS adapter rejects a tagger that returns a wrong tag count. The name-finder adapter rejects mentions whose token indices lie outside their sentence instead of silently reading the next sentence's tokens, clears adaptive data even when annotation fails, and derives UNTYPED from NameSample.DEFAULT_TYPE instead of re-declaring the literal.
… definition A blank check under the toolkit's whitespace definition, which unlike String.isBlank covers the no-break spaces, so annotators validating labels and identifiers share one predicate instead of each carrying a private copy. Reads whole code points; tests pin the no-break and figure spaces, the empty string, and a supplementary-plane letter.
…nt rule Adds the Document Annotation Container chapter to the manual, with every code example and every stated span and value mirroring the passing pipeline example test. The review pass aligns the branch with the project's conventions: layer key ids validate through StringUtil.isBlank, the annotator interface leaves thread safety implementation specific, the sentence and tokenizer adapters document annotate like their siblings, repeated rejection-message literals become per-class constants, and the name finder test's nine anonymous fixtures fold into one helper. Layers now states the key placement rule: core layer keys live there, capability layer keys on their providing annotator.
Every key the toolkit defines now carries the opennlp: id prefix (opennlp:sentences, opennlp:tokens, opennlp:pos, opennlp:entities, opennlp:lemmas, opennlp:stems). An extension defines its keys under its own prefix, and a bare id stays legal for an application-local layer, so ids from independent producers cannot collide. The rule is stated on Layers, LayerKey, and in the manual chapter.
A layer key now declares whether its layer is positional or document-scoped. A positional key, the default, guarantees a span on every annotation, so consumers never null-check one. A document-scoped key, created through LayerKey.document, carries whole-document values without spans, the home for a language id, a category distribution, or provenance. The scope is declared per key, never per annotation: the container rejects a span-less annotation under a positional key and a spanned annotation under a document-scoped key, naming the layer either way. Scope participates in key equality.
…on text The three invariants the contract tests already enforce are now stated on the Document interface and in the manual chapter: layers preserve insertion order and are never reordered, layers are immutable once added and detached from the caller's input list, and adding a layer is once-only with the rejection naming the key. Together they keep index-based references between layers valid for the lifetime of the document.
A corpus may carry a hand-annotated version of a layer beside a produced one. The convention is a gold: id prefix on the same key scheme, for example gold:opennlp:tokens beside opennlp:tokens. Because adding a layer is once-only, competing versions of a layer always live under distinct keys and never replace each other. Stated on Layers and in the manual chapter, with a contract test pinning the coexistence.
The chapter said offsets count characters; the pinned contract test shows a supplementary-plane character counts as two. Say Java chars (UTF-16 units) so the claim matches the tested behavior.
Three tests fail red: the adapters inherit the identity toString, so a pipeline validation message reads 'annotator opennlp.tools.document.SentenceDetectorAnnotator@3b96c42e requires layer ...' instead of naming the adapter. The stemmer test pins the full analyzer message exactly, since that adapter requires a single layer and the message is therefore deterministic.
All six adapters override toString with the simple class name, so a pipeline validation message names the offending annotator readably. The pinned tests pass; opennlp-api and opennlp-runtime suites green.
mawiesne
added
java
Yeah - I have jumped on this one many times over. I hope it's ready! I'll do one manual look right now but if it's green at least at some point I thought it was ready. rzo1's suggestions helped a ton. It works well with other standards so I feel it's librarian-friendly. I feel like it's really a flexible design - I have about 20 branches based off of it. Not gonna lie - to me it's the most important of tickets I have now. This in 3.0 would allow for almost all the features to confidently "drip" at whatever pace we want in the review process and not require a 4.0 release. It also unlocks most of the ability to deliver a sandbox version of the gRPC server since this shape is repeated in grpc. So please do! I spent awhile thinking about this shape so the team's feedback is super important to me. The shape is working exceptionally well with my features so far - I've tested it on about 3 sample apps - a custom BM25 search engine, a pipeline decoration step, and the opennlp grpc server. It's also similar to shapes I've seen in python counterparts. |
|
@mawiesne give me one sec - I'm doing a full check again - I am sure I addressed the comments but I didn't reply to all of them yet. I think it's good though if you want to start. |
|
@mawiesne the code concerns were addressed but I have to give a bunch of replies :) But I made the changes and think they look good. |
|
@krickert Had a first structural look on the additions. I've noticed an inconsistency, I'd like to get sorted before reading / thinking deeper: I wonder why the Same applies for the *Annotator-related tests classes, oc. These are clearly something for the checks of impl classes in the core runtime. In general, we should aim for a very lightweight API jar. Wdyt? FYI: @rzo1 @jzonthemtn Note:
|
Agree. +1 to the |
|
@mawiesne thank you for the suggestions - and agree with all of them. I'd never push back on any suggestion to make an API jar lighter. I've been working on it for a couple hours - almost done. I'll let you know when I push it. |
|
Ready for your review: I implemented both parts of your suggestion: the four concrete annotators and their tests now live in opennlp-runtime, and the annotators are placed with their respective components under sentdetect, tokenize, postag, and namefind. I also moved the API-only custom-layer contract test back to opennlp-api and removed a weaker duplicate runtime test. Ready for any other suggestions. I like how it looks a lot more now. |
# Conflicts: # opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java # opennlp-docs/src/docbkx/opennlp.xml # opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
Adapts a Chunker to the document pipeline: reads sentences, tokens, and POS tags and provides opennlp:chunks, one annotation per phrase chunk carrying its type on the span of its tokens.
Adapts a constituency Parser to the document pipeline: reads sentences and tokens and provides opennlp:phrases, one annotation per phrase node above the part-of-speech level in pre-order, carrying the label and the span of the head token the parser's head rules select.
Also makes the ParserAnnotator helpers instance methods, matching the other adapters.
…e container The manual chapter opens with why the container exists next to the task APIs, adds a table of the layers the toolkit's adapters provide with their value types and span anchors, names the tests that pin each rule, and states which established designs the shape follows.
3 participants
Adds the document annotation container discussed on OPENNLP-1888 and on the dev list in July.
Purpose. The task APIs each return one kind of result. An application that runs several over one text has to align them itself, and results that refer to other results (coreference chains, dependency arcs) have no place to live.
Documentis that place: the original text plus typed annotation layers over it. It is additive and replaces none of them.Shape. Four types in
opennlp-api, packageopennlp.tools.document:Document: the text,get(LayerKey<T>),with(LayerKey<T>, List<Annotation<T>>),layers(). Immutable; adding a layer returns a new document.Annotation<T>: aSpaninto the original text and a value.LayerKey<T>: id, value type, and scope (positional or document). Toolkit ids use theopennlp:prefix; extensions use their own;gold:marks a hand-annotated version of a layer.DocumentAnnotator:annotate,requires,provides.DocumentAnalyzerchecks the declarations when the pipeline is built.What gets attached. Adapters over the existing components provide
opennlp:sentences,opennlp:tokens,opennlp:pos,opennlp:lemmas,opennlp:stems,opennlp:entities,opennlp:chunks, andopennlp:phrases; the manual chapter has the table with value types and span anchors. The container knows none of these keys, and a new layer needs no change in the package (DocumentContractTest.testCustomLayerNeedsNoContainerChange).Rules, each pinned by a test in
DocumentContractTest: key equality is (id, type, scope); every positional annotation has a span inside the text; a layer keeps insertion order, is immutable once added, and cannot be added twice; a document-scoped layer has span-less values (language id, categories).DocumentAnalyzerTestcovers the build-time checks andDocumentPipelineExampleTestthe example in the manual.Lineage. Standoff annotation as in the UIMA CAS, typed keys as in CoreNLP's annotation map, declared requires/provides as in spaCy's pipeline analysis, at library weight.
Docs:
opennlp-docs/src/docbkx/document.xml. Has no dependency on other open PRs; the layer PRs (#1237, #1238) and the sandbox adapters build on it.