OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures - #1166
OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166krickert wants to merge 17 commits into
Conversation
e31be7c to
fdbe065
Compare
… map; cite per-branch pinning tests apache#1163 (OPENNLP-1883, the thread-safe stemmer + StemmerFactory seam) merged to apache main; move it into the merged-upstream node, repoint light-stemmers (apache#1166) and hunspell onto main since their seam dependency now ships upstream, and drop the stale open-PR row. Also lands the previously uncommitted skill-pass enrichment: each branch row now names the UsageExample/ManualExample test that pins its manual listing.
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
6ef3eba to
a7ab28a
Compare
a7ab28a to
ff681b3
Compare
|
I can't review this as a human. It is impossible in this "whole bunch" fashion. |
|
@mawiesne I have it in draft - so it's a WIP. If I flip it to green, it's my intent to signal that I have it ready. But if you see anything in here that concerns you LMK - I am checking the code and I'll work my hardest to hit all our standards once I flip it green. The draft PRs are there for the gRPC progress. SInce it requires OPENNLP-1888 + other PRs it makes sense to stack on it as part of a set of features so I can test it live. This is only to demo the grpc server and test features live before writing a ton of tests - not try to rush anything. I have zero expectation and don't want to rush anything in draft. In fact, keeping it in draft might signal there are parts not ready yet. I really like the methodical review process - so don't take my drafts as ready for review - more like "I tested it, I like what I see, I don't have major issues with it and did a few rounds of cleanup." So I kept as draft for any feedback, concerns, and if anyone wants to collaborate on the drafts. I welcome all code suggestions and changes. |
… map; cite per-branch pinning tests apache#1163 (OPENNLP-1883, the thread-safe stemmer + StemmerFactory seam) merged to apache main; move it into the merged-upstream node, repoint light-stemmers (apache#1166) and hunspell onto main since their seam dependency now ships upstream, and drop the stale open-PR row. Also lands the previously uncommitted skill-pass enrichment: each branch row now names the UsageExample/ManualExample test that pins its manual listing.
…ry parity fixtures Sixteen stemmers in the new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, Russian and the English minimal stemmer. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE. These fill the tier between no stemming and the aggressive Snowball algorithms. Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly. Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, all pairs for the small Norwegian lists) regenerated by running the Lucene classes, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.
…amples Extend docbkx/stemmer.xml with the light and minimal stemmer tiers, wire the chapter into the manual, and add StemmerFactoryUsageExampleTest and LightStemmerUsageExampleTest asserting every value the chapter prints. Also condense the StemmerFactory javadoc to the contract.
…ment with the review conventions
…ighten javadoc
- Move the varargs argument validation of the Norwegian stemmers into a new
package private NorwegianVariety.toSet(first, more) helper that null checks
first, more and every element, then returns an EnumSet of the selection.
- Have NorwegianLightStemmer and NorwegianMinimalStemmer delegate to that
helper, which drops the duplicated boolean accumulation loop from both
constructors while keeping the documented IllegalArgumentException contract.
- Replace the bare {@inheritdoc} on newStemmer() in all sixteen light and
minimal stemmers with a note that the call returns the same instance because
the stemmers are thread-safe, so the factory behaviour is not a surprise.
- Correct the FrenchMinimalStemmer class javadoc, which described it as a light
stemmer, and add the missing sentence periods on the PortugueseLightStemmer
and SpanishLightStemmer headings.
- Rewrite the SwedishMinimalStemmer class javadoc so the listed endings match
what stem() actually strips: -are in addition to -ar/arne/arna/aren, the
genitive -s, and a trailing -a, -e or -n.
- Pin the null varargs array case for NorwegianLightStemmer in
LightStemmerContractTest, which the existing null element test did not cover.
- Use assertSame instead of assertEquals for the newStemmer() identity check in
LightStemmerParityTest, since equality would pass for any two instances.
- Assert that a parity fixture line contains a tab before splitting it, so a
malformed fixture reports the offending file and line instead of failing with
a StringIndexOutOfBoundsException.
- Drop the redundant public modifier from LightStemmerUsageExampleTest to match
the package private convention of the other tests.
- Add the Jacques Savoy BSD notice to LICENSE covering the UniNE algorithms in
opennlp.tools.stemmer.light, matching the per file headers.
- Describe the light stemmer instances in stemmer.xml as immutable rather than
stateless, and rewrap the surrounding thread-safety paragraph.
Red evidence: 4 of 6 reference cases failed. crisis became crisi, lunes became lun, jerseis became jersei, and abundantes became abundant.
Red evidence: all 18 factory configurations returned the factory object itself instead of a new stemmer.
ff681b3 to
233514d
Compare
|
@mawiesne Still draft, but gettting close to "ready" - although this is fine if we put in addons. But it's based on Lucene's impl but is incredibly useful for our library. |
Import the upstream CI concurrency change. Production code, tests, and manual content are unchanged. The affected reactor compiles successfully.
# Conflicts: # LICENSE # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/EnglishMinimalStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/FinnishLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/FrenchLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/FrenchMinimalStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/GermanLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/GermanMinimalStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/HungarianLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/ItalianLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/NorwegianLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/NorwegianMinimalStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/NorwegianVariety.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/PortugueseLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/RussianLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SpanishLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SpanishMinimalStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/StemmerUtil.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SwedishLightStemmer.java # opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SwedishMinimalStemmer.java # opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/light/LightStemmerContractTest.java # opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/light/LightStemmerParityTest.java # opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/light/LightStemmerUsageExampleTest.java # opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/stemmer/light/README.txt # opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/stemmer/light/es-minimal.tsv # opennlp-distr/src/main/readme/LICENSE # rat-excludes
…-1886-light-stemmers
The runtime JAR test failed on the missing Savoy copyright before the license fragment was added. Verify the license and Apache metadata in the packaged artifact. Test NFC and lowercase preparation with decomposed German input, update the manual example, and correct Javadoc placement.
Rebased on main; #1163 (StemmerFactory) is merged, so the diff is this change alone.
Adds sixteen stemmers in a new
opennlp.tools.stemmer.lightpackage: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, and Russian, and the English minimal stemmer. They fill the tier between no stemming and the Snowball algorithms: plural and inflectional endings are removed with far less conflation. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms keep Jacques Savoy's BSD notice in each source file and in the distribution LICENSE.Each stemmer is stateless and thread-safe, implements
Stemmer, and is its ownStemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input is rejected withIllegalArgumentException.Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, complete lists for Norwegian) regenerated by running the originals, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.
Review round of 2026-09-04
Javadoc on every method and constructor,
@throwson the validating paths, theStemmerFactoryinstance contract honored and pinned, the current Spanish plural rules with their sources cited, supplementary-plane input covered, and the contract and parity tests parameterized so a failure names the word.Verification
opennlp-apiandopennlp-runtimebuild with checkstyle and forbidden APIs; the stemmer test classes pass (346 tests, of which 181 are the light and minimal contract, parity, and usage tests), and the manual section's examples are mirrored byLightStemmerUsageExampleTest.