OPENNLP-1927: Expand Hunspell dictionary support - #1270
Conversation
Add explicit partial loading with source-location diagnostics and document the remaining compatibility limits. The initial 26 rejection tests failed before the change and passed afterward. Verification passed 208 Hunspell tests with local dictionary data and 3,162 tests across the runtime and required modules.
Add conversions, compound rules, morphological analysis, and strict dictionary validation. Regression tests exposed compound boundary flags, affix combinations, duplicate components, and malformed aliases before the corresponding fixes. Validated with 3,389 runtime/API/ML unit tests, the manual build, and 17 opt-in evaluations. External dictionaries and native Hunspell are not bundled.
rzo1
left a comment
rzo1
left a comment
There was a problem hiding this comment.
Two structural changes needed before this can go in.
1. No C/C++ sources in the repo
Drop dev/hunspell-reference.cc. Maven never compiles it, CI never runs it, and no toolchain in this project covers it, so it will go stale without anyone noticing. It also exists only to link against an external GPL/LGPL/MPL project.
The method is worth documenting. Keep dev/README-hunspell-dictionaries.md and describe how you produced the reference output: the pinned Hunspell revision e184e22c..., the C API functions you called, and the g++ command. Anyone who wants to reproduce it can write those 30 lines themselves.
That means HunspellTestSupport in opennlp-runtime also has to lose the ProcessBuilder call and the opennlp.hunspell.reference property. Unit tests in opennlp-core must not fork an external native binary. The fixture dictionaries and the expected outputs you derived from the reference are the right thing to commit.
2. Eval tests should follow the existing pattern
HunspellCompatibilityEval defines its own contract: three custom system properties (opennlp.hunspell.dict.dir, opennlp.hunspell.eval.words.dir, opennlp.hunspell.reference), Assumptions-based skipping, and required input files that live nowhere we control.
The other evals in that module extend AbstractEvalTest and resolve inputs through getOpennlpDataDir() (-DOPENNLP_DATA_DIR=... -Peval-tests), checking data with verifyFileChecksum / verifyDirectoryChecksum. Do the same here:
class HunspellCompatibilityEval extends AbstractEvalTest- Read dictionaries from
new File(getOpennlpDataDir(), "hunspell/...")instead of a custom property. - Keep the SHA-256 pins, expressed the way the module already does it.
- Remove the
Assumptionsskips. Evals run under-Peval-testsagainst data that is expected to be there, and fail if it isn't.
The LibreOffice dictionaries and word lists go into opennlp-data.zip on nightlies.apache.org/opennlp/, under a hunspell/ subdirectory, not into a directory each contributor assembles by hand. Any committer can update that archive:
curl -u your_asf_username -T ./opennlp-data.zip "https://nightlies.apache.org/opennlp/"Note that the checksum file published next to the zip has to be regenerated and uploaded in the same step, otherwise the eval job fails against the new archive. See https://nightlies.apache.org/authoring.html. If you don't have an ASF account, list the exact files and revisions and one of us will upload them. Nothing is downloaded at build time either way.
Revert the addition to opennlp-docs/src/docbkx/evaltest.xml. Once the eval uses OPENNLP_DATA_DIR, that chapter already covers it. Hunspell specifics belong in stemmer.xml.
One more thing: HunspellRealDictionaryTest is removed (126 lines) without a note in the description. What covers that now?
…ementation Fixes differences found by running the reference test fixtures through the native library: - Undo affix rules that add and strip no material as single affixes, so a virtual stem completed by a zero affix is recognized and reported. - Derive ds: stems from the entry and suffixes only, then prepend sp:. - Skip the other CHECKCOMPOUNDPATTERN checks at a junction restored from a pattern replacement. - Apply CHECKCOMPOUNDDUP to the parts joined at each level only. - Bar a listed spelling with COMPOUNDFORBIDFLAG, affixed readings included, from non-final compound positions. - Keep an ONLYINCOMPOUND suffix from closing a compound. - Run the CHECKCOMPOUNDREP and word-pair checks on each remainder a compound level splits. - Let the first listed homonym decide whether a spelling is forbidden, and block compound and BREAK readings after a forbidden affixed reading. - Try a capitalized word with a further inner capital with a lowercase initial. - Map the Turkic dotted and dotless i when capitalizing. - Treat # as a comment only after the fields a directive consumes, so BREAK #, flags written as #, and affix material # load; read the SET encoding from the first field only. - Match the reference field order of analyses for prefixes without morphology and omit the stem field of an unaffixed closing compound part without data. - Match mixed-case and flag-bearing all-uppercase entries in capitalized form for all-uppercase input, as the reference does through hidden homonyms. Adds fixture tests verified with the native reference for each change, updates the evaluation's expected differences, and documents the behavior.
- Apply the Hungarian moving rule to the part of a word before a hyphen: the opening entry may qualify through the hardwired flags F, G, and H, a compound-forbidden opening entry is allowed, and size limits do not apply. - Try an all-uppercase word with an apostrophe with the part after the apostrophe capitalized, for elided articles such as L'AFRIQUE. - Remove trailing periods before lookup and restore one period when only an entry listed with it matches. - Accept numeric flags up to 65535, the range the reference parser accepts. Each rule has fixture tests verified with the native reference.
…harp s rule The manual lists where results deviate from the native library and how to work around each case, and the evaluation chapter refers to that list. The listed examples are backed by fixture tests verified with the native reference. KEEPCASE with CHECKSHARPS now accepts the all-uppercase spelling of an entry with a sharp s only when it is written with SS, and accepts the capitalized form; an all-uppercase form with a capital sharp s is rejected as in the reference.
…ase word The reference finds an entry such as İzmir for the input İZMİR outside the Turkic languages by keeping the dotted initial; the same variant is now tried. A fixture test verified with the native reference covers it.
… a native driver Removes the native driver source and the test support that forked it. The fixture tests hold the stems, analyses, and recognition outcomes captured from the pinned reference and assert OpenNLP results, naming each intended deviation so a stale entry fails. HunspellCompatibilityEval extends AbstractEvalTest, loads the dictionaries from the hunspell directory of OPENNLP_DATA_DIR, verifies MD5 digests like the other evaluations, and compares 49 inputs with captured reference results without custom properties or assumption-based skipping. The evaluation chapter of the manual is unchanged; the README describes the capture method, the pinned revision, the C API calls, and the build command.
|
There was a lot I wasn't sure about which is why I kept it in draft, thanks for chiming in. The driver source and the test support that forked it are removed. The README describes how the reference results were produced: Hunspell e184e22c, the C API calls, the input and output convention, and the build line. The fixture tests keep the captured reference outcomes and assert OpenNLP results against them; each intended deviation is named with its reason from the manual, and a named fixture must still differ, so the data cannot go stale silently. HunspellCompatibilityEval now extends AbstractEvalTest, loads from hunspell/ under OPENNLP_DATA_DIR, verifies MD5 digests with verifyFileChecksum, and has no Assumptions or custom For opennlp-data.zip, under hunspell/, from LibreOffice dictionaries commit 32b006a2c22a4ac7e8ed3f03346f7b3d85a970a4: en_US.aff, en_US.dic, README_en_US.txt, de_DE_frami.aff, de_DE_frami.dic, README_de_DE_frami.txt, hu_HU.aff, hu_HU.dic, README_hu_HU.txt. The MD5 pins in the eval match these files. HunspellRealDictionaryTest was a property-gated test in opennlp-runtime against the same three dictionaries. Its stem and compound assertions moved verbatim into HunspellCompatibilityEval.expectedInflections, which is where external data belongs; I will add that to the description. I feel like I have to do one more pass then flip to ready. |
|
@rzo1 I'll give the upload a shot :) I have the zip, a SHA, and a decent connection speed - should work for me. I want to do this because I'll probably have to start adding data to this more than once. If it doesn't work I'll let ya know. |
2 participants
Extends Hunspell dictionary loading, stemming, and morphological analysis. Unsupported directives fail by default; partial loading is an explicit choice that reports the skipped directives.
The engine now covers the directives the ticket names and the further behavior a sweep of the reference implementation's own test fixtures turned up: zero-material affixes, compound restrictions, forbidden-word ordering, hidden capitalized forms, Turkic and mixed-case input, the Hungarian hyphen rule, and the reference field order of analyses. The manual lists the remaining deviations with a workaround for each.
The fixture tests keep the stems, analyses, and recognition outcomes captured from Hunspell
e184e22cand assert OpenNLP results against them; no native code or process is part of the build.HunspellCompatibilityEvalextendsAbstractEvalTest, loads the LibreOfficeen_US,de_DE_frami, andhu_HUdictionaries fromhunspell/underOPENNLP_DATA_DIRwith MD5 checks, and compares 49 inputs with captured reference results. The dictionaries are inopennlp-data.zipon nightlies as of 2026-09-07.HunspellRealDictionaryTest, a property-gated test in opennlp-runtime, is removed; its assertions moved into that eval.No dictionaries or native libraries are bundled. The shared
Stemmerinterface is unchanged.Verification: opennlp-runtime 3,057 tests with checkstyle; the eval, 13 tests, against both the local data directory and an extraction of the published archive.
OPENNLP-1927