Skip to content

OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures - #1166

Draft
krickert wants to merge 17 commits into
apache:mainfrom
ai-pipestream:light-stemmers
Draft

OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166
krickert wants to merge 17 commits into
apache:mainfrom
ai-pipestream:light-stemmers

Conversation

@krickert

@krickert krickert commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Rebased on main; #1163 (StemmerFactory) is merged, so the diff is this change alone.

Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, and Russian, and the English minimal stemmer. They fill the tier between no stemming and the Snowball algorithms: plural and inflectional endings are removed with far less conflation. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms keep Jacques Savoy's BSD notice in each source file and in the distribution LICENSE.

Each stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input is rejected with IllegalArgumentException.

Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, complete lists for Norwegian) regenerated by running the originals, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.

Review round of 2026-09-04

Javadoc on every method and constructor, @throws on the validating paths, the StemmerFactory instance contract honored and pinned, the current Spanish plural rules with their sources cited, supplementary-plane input covered, and the contract and parity tests parameterized so a failure names the word.

Verification

opennlp-api and opennlp-runtime build with checkstyle and forbidden APIs; the stemmer test classes pass (346 tests, of which 181 are the light and minimal contract, parity, and usage tests), and the manual section's examples are mirrored by LightStemmerUsageExampleTest.

@krickert krickert changed the title UniNE light and minimal stemmer tiers with vocabulary parity fixtures OPENNLP-1886 - UniNE light and minimal stemmer tiers with vocabulary parity fixtures Jul 10, 2026
@krickert krickert changed the title OPENNLP-1886 - UniNE light and minimal stemmer tiers with vocabulary parity fixtures OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures Jul 10, 2026
@krickert krickert self-assigned this Jul 10, 2026
@krickert
krickert force-pushed the light-stemmers branch 7 times, most recently from e31be7c to fdbe065 Compare July 19, 2026 11:41
krickert added a commit to ai-pipestream/opennlp that referenced this pull request Jul 20, 2026
… map; cite per-branch pinning tests

apache#1163 (OPENNLP-1883, the thread-safe stemmer + StemmerFactory seam) merged to apache
main; move it into the merged-upstream node, repoint light-stemmers (apache#1166) and hunspell
onto main since their seam dependency now ships upstream, and drop the stale open-PR row.
Also lands the previously uncommitted skill-pass enrichment: each branch row now names the
UsageExample/ManualExample test that pins its manual listing.
krickert added a commit to ai-pipestream/opennlp that referenced this pull request Jul 24, 2026
…est head

All nine open heads now sit directly on a864230 and report mergeable. Two
were reporting conflicts and both cleared: apache#1167 through a plain rebase, and
apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed
head it had drifted away from, which shrinks its diff back to the 30 commits it
owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is
upstream as a single squash, and is 3 commits now.

Also correct what the draft flag on apache#1182 means: the branch is review-ready and
waits on the upstream queue, not on unfinished work.
@krickert
krickert force-pushed the light-stemmers branch 2 times, most recently from 6ef3eba to a7ab28a Compare July 31, 2026 10:38
krickert added a commit that referenced this pull request Aug 15, 2026
# Conflicts:
#	rat-excludes
@mawiesne

Copy link
Copy Markdown
Contributor

I can't review this as a human. It is impossible in this "whole bunch" fashion.

@krickert

Copy link
Copy Markdown
Contributor Author

@mawiesne I have it in draft - so it's a WIP. If I flip it to green, it's my intent to signal that I have it ready. But if you see anything in here that concerns you LMK - I am checking the code and I'll work my hardest to hit all our standards once I flip it green.

The draft PRs are there for the gRPC progress. SInce it requires OPENNLP-1888 + other PRs it makes sense to stack on it as part of a set of features so I can test it live. This is only to demo the grpc server and test features live before writing a ton of tests - not try to rush anything.

I have zero expectation and don't want to rush anything in draft. In fact, keeping it in draft might signal there are parts not ready yet. I really like the methodical review process - so don't take my drafts as ready for review - more like "I tested it, I like what I see, I don't have major issues with it and did a few rounds of cleanup."

So I kept as draft for any feedback, concerns, and if anyone wants to collaborate on the drafts. I welcome all code suggestions and changes.

krickert added a commit that referenced this pull request Aug 30, 2026
# Conflicts:
#	rat-excludes
krickert added a commit to ai-pipestream/opennlp that referenced this pull request Sep 4, 2026
… map; cite per-branch pinning tests

apache#1163 (OPENNLP-1883, the thread-safe stemmer + StemmerFactory seam) merged to apache
main; move it into the merged-upstream node, repoint light-stemmers (apache#1166) and hunspell
onto main since their seam dependency now ships upstream, and drop the stale open-PR row.
Also lands the previously uncommitted skill-pass enrichment: each branch row now names the
UsageExample/ManualExample test that pins its manual listing.
krickert and others added 6 commits September 4, 2026 17:51
…ry parity fixtures

Sixteen stemmers in the new opennlp.tools.stemmer.light package: light and
minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk
varieties), Swedish, plus light stemmers for Finnish, Hungarian, Italian,
Portuguese, Russian and the English minimal stemmer. Adapted from Apache
Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's
BSD notice in each source file and in the distribution LICENSE. These fill
the tier between no stemming and the aggressive Snowball algorithms.

Every stemmer is stateless and thread-safe, implements Stemmer, and is its
own StemmerFactory. Input is expected lowercase, matching the algorithms'
original contract; null input fails loudly.

Parity is asserted against the original implementations: bundled fixtures
sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per
stemmer, all pairs for the small Norwegian lists) regenerated by running
the Lucene classes, so any behavioral drift fails the test. The Galician
and Portuguese minimal stemmers are excluded because they build on the RSLP
rule engine, which is its own effort.
…amples

Extend docbkx/stemmer.xml with the light and minimal stemmer tiers, wire the
chapter into the manual, and add StemmerFactoryUsageExampleTest and
LightStemmerUsageExampleTest asserting every value the chapter prints. Also
condense the StemmerFactory javadoc to the contract.
…ighten javadoc

- Move the varargs argument validation of the Norwegian stemmers into a new
  package private NorwegianVariety.toSet(first, more) helper that null checks
  first, more and every element, then returns an EnumSet of the selection.
- Have NorwegianLightStemmer and NorwegianMinimalStemmer delegate to that
  helper, which drops the duplicated boolean accumulation loop from both
  constructors while keeping the documented IllegalArgumentException contract.
- Replace the bare {@inheritdoc} on newStemmer() in all sixteen light and
  minimal stemmers with a note that the call returns the same instance because
  the stemmers are thread-safe, so the factory behaviour is not a surprise.
- Correct the FrenchMinimalStemmer class javadoc, which described it as a light
  stemmer, and add the missing sentence periods on the PortugueseLightStemmer
  and SpanishLightStemmer headings.
- Rewrite the SwedishMinimalStemmer class javadoc so the listed endings match
  what stem() actually strips: -are in addition to -ar/arne/arna/aren, the
  genitive -s, and a trailing -a, -e or -n.
- Pin the null varargs array case for NorwegianLightStemmer in
  LightStemmerContractTest, which the existing null element test did not cover.
- Use assertSame instead of assertEquals for the newStemmer() identity check in
  LightStemmerParityTest, since equality would pass for any two instances.
- Assert that a parity fixture line contains a tab before splitting it, so a
  malformed fixture reports the offending file and line instead of failing with
  a StringIndexOutOfBoundsException.
- Drop the redundant public modifier from LightStemmerUsageExampleTest to match
  the package private convention of the other tests.
- Add the Jacques Savoy BSD notice to LICENSE covering the UniNE algorithms in
  opennlp.tools.stemmer.light, matching the per file headers.
- Describe the light stemmer instances in stemmer.xml as immutable rather than
  stateless, and rewrap the surrounding thread-safety paragraph.
Red evidence: 4 of 6 reference cases failed. crisis became crisi, lunes became lun, jerseis became jersei, and abundantes became abundant.
@krickert

krickert commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@mawiesne Still draft, but gettting close to "ready" - although this is fine if we put in addons. But it's based on Lucene's impl but is incredibly useful for our library.

Import the upstream CI concurrency change. Production code, tests, and manual content are unchanged. The affected reactor compiles successfully.
krickert added a commit to ai-pipestream/opennlp that referenced this pull request Sep 5, 2026
# Conflicts:
#	LICENSE
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/EnglishMinimalStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/FinnishLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/FrenchLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/FrenchMinimalStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/GermanLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/GermanMinimalStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/HungarianLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/ItalianLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/NorwegianLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/NorwegianMinimalStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/NorwegianVariety.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/PortugueseLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/RussianLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SpanishLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SpanishMinimalStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/StemmerUtil.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SwedishLightStemmer.java
#	opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/light/SwedishMinimalStemmer.java
#	opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/light/LightStemmerContractTest.java
#	opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/light/LightStemmerParityTest.java
#	opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/light/LightStemmerUsageExampleTest.java
#	opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/stemmer/light/README.txt
#	opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/stemmer/light/es-minimal.tsv
#	opennlp-distr/src/main/readme/LICENSE
#	rat-excludes
The runtime JAR test failed on the missing Savoy copyright before the license fragment was added. Verify the license and Apache metadata in the packaged artifact.

Test NFC and lowercase preparation with decomposed German input, update the manual example, and correct Javadoc placement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants