diff --git a/dev/README-hunspell-dictionaries.md b/dev/README-hunspell-dictionaries.md index 711435d948..33e61411fe 100644 --- a/dev/README-hunspell-dictionaries.md +++ b/dev/README-hunspell-dictionaries.md @@ -19,7 +19,8 @@ The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) reads a user-supplied `.dic` word list and its `.aff` affix file. Apache OpenNLP bundles no dictionary -data. The dictionary's readme states its license. +data. Retain the upstream copyright notices and full license text with downloaded +files. A dictionary's license is separate from OpenNLP's Apache License. ## Where dictionaries come from @@ -78,7 +79,7 @@ CharSequence stem = stemmer.stem("workers"); ``` The result depends on the loaded dictionary. The in-tree manual example uses a -small dictionary and checks that `workers` stems to `worker`. +small dictionary and checks that `workers` stems to `work`. The dictionary is immutable and safe to share between threads. The factory creates a new stemmer for each call, so each thread can use its own instance. A dictionary that @@ -87,22 +88,173 @@ accordingly; no conversion is required. ## Testing against real dictionaries -The in-tree tests use project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, also checks everyday morphology with the LibreOffice `en_US`, `de_DE_frami`, and `hu_HU` dictionaries. Point it at one directory containing all listed `.aff` and `.dic` files. A missing dictionary skips the associated test; a dictionary that cannot be loaded fails it. +The runtime tests use project-authored fixtures. `HunspellCompatibilityEval` in `opennlp-eval-tests` extends `AbstractEvalTest` and loads the LibreOffice `en_US`, `de_DE_frami`, and `hu_HU` dictionaries from the `hunspell/` directory of `OPENNLP_DATA_DIR`, the shared `opennlp-data.zip` archive every evaluation uses. It checks strict loading, expected inflections, compounds, concurrent stemming and analysis, and the results recorded from the reference implementation as described below. + +The dictionary revision is +[`32b006a2c22a4ac7e8ed3f03346f7b3d85a970a4`](https://github.com/LibreOffice/dictionaries/tree/32b006a2c22a4ac7e8ed3f03346f7b3d85a970a4). +The archive holds `.aff`, `.dic`, and `README_.txt` for each of +the three dictionaries under `hunspell/`. The evaluation verifies the MD5 digests of +the affix and word-list files before loading them, as the other evaluations do, and +fails when a file is missing or changed. These checks cover selected examples, not +all possible words or dictionaries. + +The English dictionary's `README_en_US.txt` contains the SCOWL and Ispell +copyright and license notices. The German dictionary is GPL-licensed and must +not be bundled in an Apache release. The Hungarian dictionary offers MPL-2.0 +or LGPL-3.0-or-later; select MPL-2.0 and retain that license text with the README. +They are evaluation inputs, not redistributed OpenNLP resources. +See the [ASF third-party license policy](https://www.apache.org/legal/resolved.html) +before proposing to bundle any dictionary. ``` -./mvnw test -pl opennlp-core/opennlp-runtime -am \ - -Dtest=HunspellRealDictionaryTest -Dsurefire.failIfNoSpecifiedTests=false \ - -Dopennlp.hunspell.dict.dir=/tmp/hunspell-dicts +./mvnw test -pl opennlp-eval-tests -am -Peval-tests \ + -Dtest=HunspellCompatibilityEval -Dsurefire.failIfNoSpecifiedTests=false \ + -Dopennlp.forkCount=1 -DOPENNLP_DATA_DIR=/path/to/opennlp-data +``` + +The evaluation compares 49 input forms with the stems, analyses, and recognition +recorded from the reference implementation. It reports exact result-set matches, +expected differences, unknown-input identity fallbacks, and unexpected results +separately, and fails on an unexpected result. Expected differences specify the +complete OpenNLP output for inputs whose recorded reference output differs, so a +change on either side fails. Concurrency checks compare repeated results with a +single-threaded reference. They do not measure throughput, and compatibility +counts are not accuracy scores. + +### How the reference results were recorded + +The reference outcomes recorded in `HunspellCompatibilityTest`, `HunspellCompletionTest`, and `HunspellCompatibilityEval` come from Hunspell revision [`e184e22c51fe213f4490e9b36998f0ad3e5e606b`](https://github.com/hunspell/hunspell/commit/e184e22c51fe213f4490e9b36998f0ad3e5e606b), built from source and driven through its C API. The project contains no native source, and no test forks a native process; the fixtures and the recorded outputs are what is committed. + +The driver used for recording is about thirty lines of C++ against `hunspell.h`: it calls `Hunspell_create(affixPath, dictionaryPath)`, reads one input per line from standard input in the encoding the affix file declares with `SET`, and for each line calls `Hunspell_spell`, `Hunspell_stem`, or `Hunspell_analyze` as selected by a command-line argument, printing one output line per input with multiple stems or analyses joined by a tab, then releases each result list with `Hunspell_free_list` and the handle with `Hunspell_destroy`. It builds with: + +```sh +g++ -std=c++17 -O2 -DHUNSPELL_STATIC \ + -I/path/to/hunspell/src/hunspell \ + /path/to/hunspell/src/hunspell/*.cxx driver.cc -o hunspell-reference ``` +Whitespace inside recorded analyses is normalized to single spaces. The fixture tests assert the OpenNLP results and, where recognition deliberately deviates from the recorded reference outcome, name the deviation from the manual; a fixture whose deviation disappears fails, so the recorded outcomes stay honest. To re-record after a reference upgrade, rebuild the driver from the new revision, run the fixtures and the evaluation inputs through it, and update the recorded values. + ## What the engine supports -The engine applies `PFX` and `SFX` rules with strip strings and character-class conditions. It supports a prefix and suffix cross-product, a double suffix sequence connected by continuation classes, identity rules in continuation paths, file-wide `FLAG` modes, file-wide `AF` aliases, and the `SET` encoding declaration. Numeric flags range from 1 through 65000. +The engine applies `PFX` and `SFX` rules with strip strings and character-class conditions. It supports a prefix and suffix cross-product, a double suffix sequence connected by continuation classes, rules that add and strip no material both on their own and in continuation paths, file-wide `FLAG` modes, file-wide `AF` aliases, and the `SET` encoding declaration. Numeric flags range from 1 through 65535, the full range the reference accepts. A number sign starts a comment at the beginning of a line or after the fields a directive consumes; elsewhere it is an ordinary value, so `BREAK #`, `NEEDAFFIX #`, and affix material consisting of `#` load as written. + +`COMPLEXPREFIXES` selects 2 prefix levels and 1 suffix level instead of 1 +prefix and 2 suffixes. `ICONV` and `OCONV` use longest-match conversions; +`IGNORE` removes configured characters from input, entries, and affix material. +`KEEPCASE`, `CHECKSHARPS`, `LANG`, `WARN`, and `FORBIDWARN` control case variants +and warning-marked entries. A capitalized word with a further inner capital is also +tried with a lowercase initial, and the Turkic `LANG` values map the dotted and +dotless `i` in both case directions. All-uppercase input also matches mixed-case +entries and flagged all-uppercase entries in their capitalized form, as the +reference does through hidden capitalized homonyms, so `IPODS` stems to `Ipod` +while `Ipods` stays unrecognized; these forms take no part in compounds. An +all-uppercase word with an apostrophe is also tried with the part after the +apostrophe capitalized, so `L'AFRIQUE` finds an elided article rule. Trailing +periods are removed before lookup, and one period is restored when only an entry +listed with it matches, so `texts.` stems to `text` and `etc.` stays `etc.`. +Under `LANG hu`, the part of a word before a hyphen follows the reference's +moving rule: it may be a compound whose opening entry carries one of the +hardwired flags `F`, `G`, or `H`, ignoring compound-forbid and size limits. + +Compound decomposition supports positional flags and independent `COMPOUNDRULE` +patterns, including optional and repeated flags. It applies compound permit and +forbid flags, word-count limits, `COMPOUNDROOT`, `COMPOUNDSYLLABLE`, duplicate, +case, triple-letter and pattern restrictions, simplified junctions, +`COMPOUNDMORESUFFIXES`, and `FORCEUCASE`. `CHECKCOMPOUNDREP` checks both `REP` +entries and dictionary `ph:` replacements. Compound boundaries and minimum +lengths use Unicode code points. `BREAK` splits recognized parts recursively; +the default separators are `-`, `^-`, and `-$`, and `BREAK 0` disables them. + +The compound restrictions follow the reference implementation in detail. `CHECKCOMPOUNDDUP` compares the two parts joined at each level, so only a repeated closing part rejects a compound. The `CHECKCOMPOUNDREP` and word-pair checks apply to the complete input and to every remainder a further level splits. A junction restored from a `CHECKCOMPOUNDPATTERN` replacement is exempt from the other patterns. A listed spelling whose first homonym carries `COMPOUNDFORBIDFLAG` is barred from every position but the last, including its affixed readings, and a suffix marked `ONLYINCOMPOUND` cannot close a compound. + +`NEEDAFFIX` (also named `PSEUDOROOT`), `ONLYINCOMPOUND`, `FORBIDDENWORD`, +`CIRCUMFIX`, and `FULLSTRIP` control whether an analysis is accepted. As in the +reference implementation, the first listed homonym decides whether a spelling is +forbidden, and a forbidden direct or affixed reading also blocks the compound and +`BREAK` readings of that input. Morphology +aliases use `AM`; `st:` supplies an explicit stem, `sp:` prepends surface +material, and `ds:` makes the form derived by the entry's suffixes the stem. + +`SYLLABLENUM` supports Hungarian compound syllable adjustments. The deprecated +`LEMMA_PRESENT` directive is validated but has no effect. Obsolete +`COMPOUNDFIRST`, `COMPOUNDLAST`, `ONLYROOT`, `HU_KOTOHANGZO`, and `GENERATE` +metadata have no effect on stemming or analysis in the pinned reference and +are ignored. The active compound and affix directives remain applicable. + +`HunspellStemmer.analyze(text)` returns an immutable list of distinct analyses +as space-separated Hunspell fields in the reference field order. Entries without +`st:` use the entry text. A suffix without morphological fields contributes `fl:` +and its flag after the entry fields. A prefix without morphological fields +contributes its affix text before the stem when no suffix follows and `fl:` with +its flag otherwise; an entry without fields then contributes the prefix's `fl:` +field after the stem. Compound components begin with `pa:`, and a closing +component without affixes or entry fields carries no `st:` field. Unknown input +returns an empty list. +Analysis preserves field text without `OCONV`. The shared `Stemmer` interface +is unchanged. The manual contains an executable example. + +Comments and unused metadata may contain legacy-encoded bytes even when the file uses UTF-8. Parsed rules and dictionary text are decoded strictly. Default and `long` flag modes preserve raw one-byte flag values used by published UTF-8 dictionaries. Invalid rule counts, aliases, flags, and compound limits fail during loading in both modes. Each affix or dictionary stream is rejected when it exceeds `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB). + +## Loading policy + +`HunspellDictionary.load(...)` defaults to `LoadMode.STRICT`. Unsupported affix +directives cause an `IOException` identifying the directive and source line. +Path-based loading includes the affix path. Valid Hunspell dictionaries using +unsupported features require an explicit choice to load partially. + +Unknown directive names cause rejection. Recognized metadata and settings outside stemming, +such as `NAME`, `TRY`, and `WORDCHARS`, are ignored. `REP` is parsed when +`CHECKCOMPOUNDREP` makes replacements affect compound recognition; otherwise it +is unused suggestion data. + +Use `ALLOW_PARTIAL` to skip unsupported directives and inspect the diagnostics: + +```java +HunspellDictionary partial = HunspellDictionary.load( + Path.of("dictionary.aff"), Path.of("dictionary.dic"), + HunspellDictionary.LoadMode.ALLOW_PARTIAL); +for (HunspellDictionary.UnsupportedDirective diagnostic : partial.getUnsupportedDirectives()) { + System.err.println(diagnostic.directive() + " at " + + diagnostic.source() + ":" + diagnostic.lineNumber()); +} +``` + +`getUnsupportedDirectives()` returns an immutable list containing the first +source location of each unsupported directive in file order. Recognized settings +outside stemming are excluded from the list. File paths identify file-based +loads; stream-based loads use `affix stream` as the source description. + +Partial loading does not apply skipped behavior. Strict loading does not +establish complete Hunspell compatibility. The engine does not generate +inflected forms or spelling suggestions. + +Compound search permits at most 64 parts and 2048 candidate checks per spelling +variant. Recursive word-break search has the same depth and candidate limits. +Sharp-s case expansion permits at most 64 variants. Compound-rule patterns are +limited to 4096 flag elements. Results are limited to 2048 distinct stems or +analyses. These limits can exclude valid analyses; all included candidates +must pass validation. Simplified triple letters can be restored at multiple +junctions. `CHECKCOMPOUNDPATTERN` replacement applies at one junction. -Compound decomposition supports `COMPOUNDFLAG`, `COMPOUNDBEGIN`, `COMPOUNDMIDDLE`, `COMPOUNDEND`, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBIDFLAG`, `CHECKCOMPOUNDDUP`, `CHECKCOMPOUNDCASE`, and `CHECKCOMPOUNDTRIPLE`. Compound boundaries and minimum lengths use Unicode code points. `NEEDAFFIX` (also named `PSEUDOROOT`), `ONLYINCOMPOUND`, `FORBIDDENWORD`, `CIRCUMFIX`, and `FULLSTRIP` control whether an analysis is accepted. +Native Hunspell's `stem()` and `spell()` do not have equivalent acceptance rules. +The native stemmer can return a stem for KEEPCASE or FORBIDWARN input rejected by +the spell checker, or return no stem for accepted complex-prefix and simplified +compound forms. OpenNLP applies the dictionary restrictions and returns recognized +compound part stems separately. Tests record native stemming and recognition +results independently. Compatibility requires checking recognition and output. -Other directives are skipped. Their conversion, suggestion, or advanced compound behavior is not applied by this affix stemmer. Comments and unused metadata may contain legacy-encoded bytes even when the file uses UTF-8. Parsed rules and dictionary text are decoded strictly. Default and `long` flag modes preserve raw one-byte flag values used by published UTF-8 dictionaries. Invalid rule counts, aliases, flags, and compound limits fail during loading. Each affix or dictionary stream is rejected when it exceeds `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB). +The German comparisons also distinguish standalone entries from compound-only +readings. For example, OpenNLP returns `Kind` for `Kinder`; native stemming also +returns the compound-only `kind` and an identity-affixed `kinder` reading. +For `Vorschläge`, the pinned native implementation recognizes the input but +returns no stem or morphological analysis. OpenNLP returns component stems and +fields. These are documented differences, not exact matches or a general +accuracy claim. -Skipped directives include `ICONV`, `OCONV`, `COMPLEXPREFIXES`, `COMPOUNDRULE`, -`IGNORE`, and `KEEPCASE`. Loading a dictionary does not apply these rules; -results can differ from Hunspell for words that need them. +For prefix-only forms without morphological fields, native analysis may include +untagged prefix text, such as `un st:done fl:U` for `undone`. OpenNLP uses +`fl:U st:done`. The evaluation classifies this formatting distinction as an +expected difference. It also checks incomplete native output for `well-known` +and German compounds such as `Haustür`, without treating additional OpenNLP +output as a general correctness advantage. diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellAffixTable.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellAffixTable.java new file mode 100644 index 0000000000..8d55512f57 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellAffixTable.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** Affix tables with entry counts and source locations for validation errors. */ +final class HunspellAffixTable { + + /** Prevents construction. */ + private HunspellAffixTable() { } + + /** + * An entry including the directive name. + * + * @param fields The tokenized entry. + * @param line The one-based source line. + */ + record Entry(String[] fields, int line) { } + + /** + * Checks a table header, entry count, and field counts. + * + * @param lines The tokenized affix content. + * @param tag The table directive. + * @param minimumFields The minimum fields in an entry, including the directive. + * @param maximumFields The maximum fields in an entry, including the directive. + * @return The entries, or null if the file has no declaration for this table. + * @throws IOException If the header, entries, or count is invalid. + */ + static List read(String[][] lines, String tag, int minimumFields, int maximumFields) + throws IOException { + final List entries = new ArrayList<>(); + int count = -1; + for (int i = 0; i < lines.length; i++) { + final String[] fields = lines[i]; + if (fields.length == 0 || !tag.equals(fields[0])) { + continue; + } + if (count < 0) { + try { + if (fields.length != 2) { + throw new NumberFormatException(); + } + count = Integer.parseInt(fields[1]); + if (count < 0) { + throw new NumberFormatException(); + } + } catch (NumberFormatException e) { + throw new IOException("invalid " + tag + " count at line " + (i + 1), e); + } + } else { + if (fields.length < minimumFields || fields.length > maximumFields || entries.size() == count) { + throw new IOException("invalid " + tag + " entry at line " + (i + 1)); + } + entries.add(new Entry(fields, i + 1)); + } + } + if (count < 0) { + return null; + } + if (entries.size() != count) { + throw new IOException(tag + " header specifies " + count + " entries but found " + entries.size()); + } + return entries; + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellCompoundRule.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellCompoundRule.java new file mode 100644 index 0000000000..91d75ee4c7 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellCompoundRule.java @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** A compound flag sequence with optional and repeated elements. */ +final class HunspellCompoundRule { + + /** Interprets a flag according to the affix file's FLAG setting. */ + @FunctionalInterface + interface FlagReader { + /** + * Interprets one flag. + * + * @param text The encoded flag. + * @param line The source line. + * @return The flag value. + * @throws IOException If the flag is malformed. + */ + int read(String text, int line) throws IOException; + } + + /** Maximum elements in one compound pattern. */ + private static final int MAX_ELEMENTS = 4096; + + private static final String INVALID_PATTERN = "invalid COMPOUNDRULE at line "; + + private final int[] flags; + private final char[] repetition; + + /** + * Initializes a parsed compound pattern. + * + * @param flags The required flag at each position. + * @param repetition The position's repetition operator or a space. + */ + private HunspellCompoundRule(int[] flags, char[] repetition) { + this.flags = flags; + this.repetition = repetition; + } + + /** + * Parses a compound rule without using a regular-expression engine. + * + * @param pattern The rule text. + * @param line The source line. + * @param reader The flag decoder. + * @return The parsed rule. + * @throws IOException If the pattern or a flag is malformed. + */ + static HunspellCompoundRule parse(String pattern, int line, FlagReader reader) throws IOException { + final List flags = new ArrayList<>(); + final StringBuilder repetitions = new StringBuilder(); + for (int at = 0; at < pattern.length();) { + final String flag; + if (pattern.charAt(at) == '(') { + final int end = pattern.indexOf(')', at + 1); + if (end <= at + 1) { + throw new IOException(INVALID_PATTERN + line); + } + flag = pattern.substring(at + 1, end); + at = end + 1; + } else { + final int point = pattern.codePointAt(at); + if (point == '*' || point == '?' || point == ')') { + throw new IOException(INVALID_PATTERN + line); + } + final int end = at + Character.charCount(point); + flag = pattern.substring(at, end); + at = end; + } + flags.add(reader.read(flag, line)); + if (flags.size() > MAX_ELEMENTS) { + throw new IOException("COMPOUNDRULE exceeds " + MAX_ELEMENTS + " elements at line " + line); + } + if (at < pattern.length() && (pattern.charAt(at) == '*' || pattern.charAt(at) == '?')) { + repetitions.append(pattern.charAt(at++)); + } else { + repetitions.append(' '); + } + } + if (flags.isEmpty()) { + throw new IOException("empty COMPOUNDRULE at line " + line); + } + final int[] values = new int[flags.size()]; + final char[] repetition = new char[flags.size()]; + for (int i = 0; i < values.length; i++) { + values[i] = flags.get(i); + repetition[i] = repetitions.charAt(i); + } + return new HunspellCompoundRule(values, repetition); + } + + /** + * Tests a sequence of selected homonyms. Each part consumes one flag position. + * + * @param parts The flags of one selected entry per compound part. + * @param complete Whether the sequence must complete the rule. + * @return Whether the sequence is permitted by this rule. + */ + boolean matches(List parts, boolean complete) { + boolean[] states = new boolean[flags.length + 1]; + states[0] = true; + skipOptional(states); + for (int[] part : parts) { + final boolean[] next = new boolean[states.length]; + for (int i = 0; i < flags.length; i++) { + if (states[i] && HunspellDictionary.contains(part, flags[i])) { + next[repetition[i] == '*' ? i : i + 1] = true; + } + } + skipOptional(next); + states = next; + } + if (complete) { + return states[flags.length]; + } + for (boolean state : states) { + if (state) { + return true; + } + } + return false; + } + + /** + * Advances states through optional pattern elements. + * + * @param states The active pattern positions. + */ + private void skipOptional(boolean[] states) { + for (int i = 0; i < flags.length; i++) { + if (states[i] && repetition[i] != ' ') { + states[i + 1] = true; + } + } + } + +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellConversion.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellConversion.java new file mode 100644 index 0000000000..8b3623dfed --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellConversion.java @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.function.Predicate; + +import opennlp.tools.util.StringUtil; + +/** Non-recursive longest-match input and output conversions. */ +final class HunspellConversion { + + /** An empty conversion table. */ + static final HunspellConversion NONE = new HunspellConversion(List.of()); + + /** + * A conversion with optional word-boundary requirements. + * + * @param from The input text. + * @param to The output text. + * @param initial Whether the match requires the start of the input. + * @param terminal Whether the match requires the end of the input. + */ + private record Rule(String from, String to, boolean initial, boolean terminal) { } + + private final List rules; + + /** + * Constructs an immutable table. + * + * @param rules The conversion definitions. + */ + private HunspellConversion(List rules) { + this.rules = List.copyOf(rules); + } + + /** + * Parses an affix conversion table and validates the entry count. + * + * @param lines The affix fields indexed by source line. + * @param directive The table name. + * @return The conversion table. + * @throws IOException If the table is malformed. + */ + static HunspellConversion parse(String[][] lines, String directive) throws IOException { + final List rules = new ArrayList<>(); + final boolean replacement = "REP".equals(directive); + final List entries = HunspellAffixTable.read(lines, directive, + 3, replacement ? Integer.MAX_VALUE : 3); + if (entries == null) { + return NONE; + } + for (HunspellAffixTable.Entry entry : entries) { + final String[] fields = entry.fields(); + final boolean initial = fields[1].startsWith(replacement ? "^" : "_"); + final boolean terminal = fields[1].endsWith(replacement ? "$" : "_"); + final int start = initial ? 1 : 0; + final int end = fields[1].length() - (terminal ? 1 : 0); + if (end <= start) { + throw new IOException("empty " + directive + " pattern at line " + entry.line()); + } + rules.add(new Rule(fields[1].substring(start, end), + "0".equals(fields[2]) ? "" : fields[2].replace('_', ' '), initial, terminal)); + } + return rules.isEmpty() ? NONE : new HunspellConversion(rules); + } + + /** + * Adds dictionary transliterations to the replacement table. + * + * @param entries The dictionary entries and their flags. + * @param morphology The fields of each selected entry. + * @return A table containing REP and ph: replacements. + */ + HunspellConversion withPhoneticFields(Map> entries, + Map> morphology) { + final List extended = new ArrayList<>(rules); + for (Map.Entry> word : entries.entrySet()) { + for (int[] flags : word.getValue()) { + for (String field : morphology.getOrDefault(flags, List.of())) { + if (!field.startsWith("ph:")) { + continue; + } + String from = field.substring(3); + String to = word.getKey(); + final int arrow = from.indexOf("->"); + if (arrow >= 0) { + to = from.substring(arrow + 2); + from = from.substring(0, arrow); + } else if (from.endsWith("*")) { + from = from.substring(0, from.length() - 1); + if (!from.isEmpty() && !to.isEmpty()) { + from = from.substring(0, from.offsetByCodePoints(from.length(), -1)); + to = to.substring(0, to.offsetByCodePoints(to.length(), -1)); + } + } + if (!from.isEmpty()) { + extended.add(new Rule(from, to, false, false)); + final int first = Character.charCount(from.codePointAt(0)); + final String upper = StringUtil.toUpperCase(from.substring(0, first)) + from.substring(first); + if (!from.equals(upper)) { + extended.add(new Rule(upper, to, false, false)); + } + } + } + } + } + return new HunspellConversion(extended); + } + + /** + * Replaces the longest matching pattern at each input position without rescanning + * replacement text. + * + * @param input The input text. + * @return The converted text, or the input when no conversion applies. + */ + String apply(String input) { + if (rules.isEmpty() || input.isEmpty()) { + return input; + } + final StringBuilder output = new StringBuilder(input.length()); + boolean changed = false; + for (int offset = 0; offset < input.length();) { + Rule selected = null; + for (Rule rule : rules) { + if ((!rule.initial() || offset == 0) + && (!rule.terminal() || offset + rule.from().length() == input.length()) + && input.startsWith(rule.from(), offset) + && (selected == null || rule.from().length() > selected.from().length())) { + selected = rule; + } + } + if (selected != null) { + output.append(selected.to()); + offset += selected.from().length(); + changed = true; + } else { + final int point = input.codePointAt(offset); + output.appendCodePoint(point); + offset += Character.charCount(point); + } + } + return changed ? output.toString() : input; + } + + /** + * Tests individual replacements without combining separate corrections. + * + * @param input The candidate compound. + * @param accepted Tests whether a replacement is a recognized non-compound word. + * @return Whether a replacement satisfies the test. + */ + boolean anyReplacement(String input, Predicate accepted) { + for (int offset = 0; offset < input.length();) { + for (Rule rule : rules) { + if ((!rule.initial() || offset == 0) + && (!rule.terminal() || offset + rule.from().length() == input.length()) + && input.startsWith(rule.from(), offset)) { + final String replacement = input.substring(0, offset) + rule.to() + + input.substring(offset + rule.from().length()); + if (!replacement.equals(input) && accepted.test(replacement)) { + return true; + } + } + } + offset += Character.charCount(input.codePointAt(offset)); + } + return false; + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java index 2bb06935d5..bd7c94e760 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java @@ -32,8 +32,10 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.function.Predicate; import opennlp.tools.commons.ThreadSafe; import opennlp.tools.util.StringUtil; @@ -43,21 +45,25 @@ * {@code .aff} and {@code .dic} files. OpenNLP includes no dictionary data. * *

Supported affix features are {@code PFX} and {@code SFX} rules with strip - * strings, character-class conditions, cross-product combinations, and a double suffix - * connected by continuation classes; {@code FLAG} modes {@code char}, {@code UTF-8}, - * {@code long}, and {@code num}; the {@code AF} alias table; and the {@code SET} - * encoding declaration. Compound decomposition supports {@code COMPOUNDFLAG}, - * {@code COMPOUNDBEGIN}, {@code COMPOUNDMIDDLE}, {@code COMPOUNDEND}, - * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG}, - * {@code COMPOUNDFORBIDFLAG}, {@code CHECKCOMPOUNDDUP}, - * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE}. The blocking flags + * strings, character-class conditions, cross-product combinations, and continuation + * classes. {@code COMPLEXPREFIXES} selects 2 prefixes and 1 suffix instead of + * 1 prefix and 2 suffixes. {@code FLAG}, {@code AF}, {@code AM}, and {@code SET} + * declarations apply throughout the file. Input and output conversions, ignored + * characters, case restrictions, compound flags and patterns, and word breaks are + * applied during stemming. The blocking flags * {@code NEEDAFFIX} (also named {@code PSEUDOROOT}), {@code ONLYINCOMPOUND}, and * {@code FORBIDDENWORD}, plus {@code CIRCUMFIX} and {@code FULLSTRIP}, are also * applied.

* - *

Other directives are skipped. Their spelling, conversion, suggestion, or - * advanced compound behavior is not applied by this affix stemmer. Dictionary - * morphology fields are also ignored.

+ *

Loading defaults to {@link LoadMode#STRICT}: unsupported affix directives + * cause an {@link IOException}. Recognized metadata and suggestion settings that + * do not affect stemming are ignored. {@link LoadMode#ALLOW_PARTIAL} skips other + * directives and reports them through {@link #getUnsupportedDirectives()}. + * Morphological stems use {@code st:}, {@code sp:}, and {@code ds:} fields. + * {@code SYLLABLENUM} supports Hungarian compound syllable adjustments. + * {@code LEMMA_PRESENT} is accepted but has no effect, consistent with the + * reference version. Strict loading validates declarations, not equivalence of + * all results with native Hunspell.

* *

Instances are immutable and safe to share between threads.

* @@ -81,6 +87,48 @@ public final class HunspellDictionary { */ public static final int MAX_STREAM_BYTES = 64 * 1024 * 1024; + /** Controls whether unsupported affix directives prevent dictionary loading. */ + public enum LoadMode { + /** Reject unsupported directives, including unknown directive names. */ + STRICT, + /** Skip unsupported directives and record their first source locations. */ + ALLOW_PARTIAL + } + + /** + * The first occurrence of an unsupported affix directive. + * + * @param directive The directive name. + * @param source The affix file path, or {@code affix stream} for stream loading. + * @param lineNumber The one-based source line number. + */ + public record UnsupportedDirective(String directive, String source, int lineNumber) { + + /** + * Creates a diagnostic with a directive name and source location. + * + * @param directive The nonblank directive name. + * @param source The nonblank source description. + * @param lineNumber The positive source line number. + * @throws IllegalArgumentException If a name is null or blank, or the line + * number is not positive. + */ + public UnsupportedDirective { + if (directive == null || directive.isBlank()) { + throw new IllegalArgumentException("directive must not be null or blank"); + } + if (source == null || source.isBlank()) { + throw new IllegalArgumentException("source must not be null or blank"); + } + if (lineNumber < 1) { + throw new IllegalArgumentException("lineNumber must be positive"); + } + } + } + + private static final String AFFIX_STREAM = "affix stream"; + private static final String DICTIONARY_STREAM = "dictionary stream"; + /** * One parsed affix rule of a {@code PFX} or {@code SFX} block. * @@ -90,9 +138,19 @@ public final class HunspellDictionary { * @param affix The surface material the rule adds to the stem. * @param condition The condition the stem must satisfy for the rule to apply. * @param continuation The flags of the further affixes that may stack on this one. + * @param suffix Whether the rule adds a suffix. + * @param morphology The expanded morphological fields, empty when the rule declares none. + * @param flagText The flag as written in the rule, reported as the {@code fl:} field of + * a rule without morphological fields. */ record Affix(int flag, boolean crossProduct, String strip, String affix, - AffixCondition condition, int[] continuation) { + AffixCondition condition, int[] continuation, boolean suffix, List morphology, + String flagText) { + + /** {@return the morphological fields, or the {@code fl:} flag field when none are declared} */ + List analysisFields() { + return morphology.isEmpty() ? List.of(FLAG_FIELD + flagText) : morphology; + } /** * Checks whether a further affix may stack on this one. @@ -101,12 +159,7 @@ record Affix(int flag, boolean crossProduct, String strip, String affix, * @return {@code true} if this affix's continuation classes allow it. */ boolean allowsContinuation(int otherFlag) { - for (final int candidate : continuation) { - if (candidate == otherFlag) { - return true; - } - } - return false; + return contains(continuation, otherFlag); } } @@ -135,6 +188,35 @@ enum CompoundPosition { /** The directive that defines the file-wide flag alias table. */ private static final String ALIAS_TAG = "AF"; + private static final String INPUT_CONVERSION_TAG = "ICONV"; + private static final String OUTPUT_CONVERSION_TAG = "OCONV"; + private static final String IGNORE_TAG = "IGNORE"; + private static final String KEEP_CASE_TAG = "KEEPCASE"; + private static final String COMPLEX_PREFIXES_TAG = "COMPLEXPREFIXES"; + private static final String COMPOUND_RULE_TAG = "COMPOUNDRULE"; + private static final String COMPOUND_PATTERN_TAG = "CHECKCOMPOUNDPATTERN"; + private static final String MORPHOLOGY_ALIAS_TAG = "AM"; + private static final String BREAK_TAG = "BREAK"; + private static final String LANGUAGE_TAG = "LANG"; + private static final String WARNING_TAG = "WARN"; + private static final String FORBID_WARNING_TAG = "FORBIDWARN"; + private static final String CHECK_SHARPS_TAG = "CHECKSHARPS"; + private static final String COMPOUND_ROOT_TAG = "COMPOUNDROOT"; + private static final String FORCE_UPPER_CASE_TAG = "FORCEUCASE"; + private static final String COMPOUND_MORE_SUFFIXES_TAG = "COMPOUNDMORESUFFIXES"; + private static final String SIMPLIFIED_TRIPLE_TAG = "SIMPLIFIEDTRIPLE"; + private static final String SYLLABLE_NUMBER_TAG = "SYLLABLENUM"; + private static final String LEMMA_PRESENT_TAG = "LEMMA_PRESENT"; + private static final String COMPOUND_SYLLABLE_TAG = "COMPOUNDSYLLABLE"; + private static final String REPLACEMENT_TAG = "REP"; + private static final String STEM_FIELD = "st:"; + private static final String SURFACE_PREFIX_FIELD = "sp:"; + private static final String DERIVATIONAL_SUFFIX_FIELD = "ds:"; + private static final String FLAG_FIELD = "fl:"; + + private static final List DEFAULT_WORD_BREAKS = List.of("-", "^-", "-$"); + + /** Prefix used by comment lines. */ private static final String COMMENT_PREFIX = "#"; @@ -151,16 +233,29 @@ enum CompoundPosition { private static final String NO_MATERIAL = "0"; /** - * Largest flag value permitted by {@code FLAG num}, as specified by the + * Largest flag value permitted by {@code FLAG num}. The * - * Hunspell format manual. + * Hunspell format manual names 65000, while the reference parser accepts the + * full unsigned 16-bit range and published dictionaries use values above 65000. */ - private static final int MAX_NUMERIC_FLAG = 65_000; + private static final int MAX_NUMERIC_FLAG = 65_535; + + /** + * The flags the reference implementation hardwires for Hungarian: an entry carrying + * one of them may open a compound written before a hyphen. + */ + private static final int[] HUNGARIAN_HYPHEN_FLAGS = {'F', 'G', 'H'}; /** Largest {@code COMPOUNDMIN} value that can be doubled without overflow. */ private static final int MAX_COMPOUND_MIN = Integer.MAX_VALUE / 2; private final Map> entries; + /** + * The capitalized forms of mixed-case and flagged all-uppercase entries, which the + * reference implementation adds as hidden homonyms so that all-uppercase input + * matches them. Keyed like {@link #entries}, sharing the entry flag arrays. + */ + private final Map> hiddenEntries; private final BoundaryIndex suffixesByLast; private final List suffixesWithoutMaterial; private final BoundaryIndex prefixesByFirst; @@ -181,14 +276,40 @@ enum CompoundPosition { private final boolean checkCompoundCase; private final boolean checkCompoundTriple; private final boolean fullStrip; + private final String ignoredCharacters; + private final HunspellConversion inputConversion; + private final HunspellConversion outputConversion; + private final Map> morphology; + private final boolean complexPrefixes; + private final int keepCase; + private final int warningFlag; + private final boolean forbidWarn; + private final boolean checkSharps; + private final boolean turkicCase; + private final boolean hungarian; + private final boolean syllableNumber; + private final List compoundRules; + private final List compoundPatterns; + private final int compoundRoot; + private final int forceUpperCase; + private final boolean compoundMoreSuffixes; + private final boolean simplifiedTriple; + private final boolean checkCompoundRep; + private final HunspellConversion replacements; + private final int maxCompoundSyllables; + private final String compoundVowels; + private final List wordBreaks; + private final List unsupportedDirectives; /** * Initializes the dictionary from the two parsed files. * * @param entries The words mapped to the flag sets of their entries. * @param affix The parsed affix file. + * @param unsupportedDirectives The skipped directives in encounter order. */ - private HunspellDictionary(Map> entries, AffixFile affix) { + private HunspellDictionary(Map> entries, AffixFile affix, + List unsupportedDirectives) { this.compoundFlag = affix.compoundFlag; this.compoundBegin = affix.compoundBegin; this.compoundEnd = affix.compoundEnd; @@ -205,7 +326,35 @@ private HunspellDictionary(Map> entries, AffixFile affix) { this.checkCompoundCase = affix.checkCompoundCase; this.checkCompoundTriple = affix.checkCompoundTriple; this.fullStrip = affix.fullStrip; + this.ignoredCharacters = affix.ignoredCharacters; + this.inputConversion = affix.inputConversion; + this.outputConversion = affix.outputConversion; + this.morphology = Map.copyOf(affix.entryMorphology); + this.complexPrefixes = affix.complexPrefixes; + this.keepCase = affix.keepCase; + this.warningFlag = affix.warningFlag; + this.forbidWarn = affix.forbidWarn; + this.checkSharps = affix.checkSharps; + this.compoundRules = List.copyOf(affix.compoundRules); + this.compoundPatterns = List.copyOf(affix.compoundPatterns); + this.compoundRoot = affix.compoundRoot; + this.forceUpperCase = affix.forceUpperCase; + this.compoundMoreSuffixes = affix.compoundMoreSuffixes; + this.simplifiedTriple = affix.simplifiedTriple; + this.checkCompoundRep = affix.checkCompoundRep; + this.replacements = affix.checkCompoundRep + ? affix.replacements.withPhoneticFields(entries, affix.entryMorphology) : affix.replacements; + this.maxCompoundSyllables = affix.maxCompoundSyllables; + this.compoundVowels = affix.compoundVowels; + this.wordBreaks = List.copyOf(affix.wordBreaks); + this.turkicCase = affix.language.equals("tr") || affix.language.startsWith("tr_") + || affix.language.equals("az") || affix.language.startsWith("az_") + || affix.language.equals("crh") || affix.language.startsWith("crh_"); + this.hungarian = affix.language.equals("hu") || affix.language.startsWith("hu_"); + this.syllableNumber = affix.syllableNumber; this.entries = entries; + this.hiddenEntries = hiddenCapitalizedEntries(entries); + this.unsupportedDirectives = List.copyOf(unsupportedDirectives); // A material-bearing rule can only be undone from a word whose boundary // character matches its affix material, so bucketing by that character // narrows each scan to one bucket plus the strip-only rules. @@ -287,30 +436,51 @@ private static BoundaryIndex bucketByBoundary(List rules, } /** - * Loads a dictionary from its two files. + * Loads affix and dictionary files using {@link LoadMode#STRICT}. * * @param affixFile The {@code .aff} affix file. Must not be {@code null}. * @param dictionaryFile The {@code .dic} word list. Must not be {@code null}. * @return The loaded dictionary. Never {@code null}. - * @throws IOException Thrown if reading fails or a file is malformed. + * @throws IOException Thrown if reading fails, a file is malformed, or an affix + * directive is unsupported. * @throws IllegalArgumentException Thrown if a parameter is {@code null}. */ public static HunspellDictionary load(Path affixFile, Path dictionaryFile) throws IOException { + return load(affixFile, dictionaryFile, LoadMode.STRICT); + } + + /** + * Loads affix and dictionary files with the selected directive policy. + * + * @param affixFile The non-null {@code .aff} affix file. + * @param dictionaryFile The non-null {@code .dic} word list. + * @param mode The non-null loading policy. + * @return The loaded dictionary. + * @throws IOException If reading fails, a stream exceeds {@link #MAX_STREAM_BYTES}, + * content is malformed, or strict loading encounters an unsupported directive. + * @throws IllegalArgumentException If a parameter is null. + */ + public static HunspellDictionary load(Path affixFile, Path dictionaryFile, + LoadMode mode) throws IOException { if (affixFile == null) { throw new IllegalArgumentException("affixFile must not be null"); } if (dictionaryFile == null) { throw new IllegalArgumentException("dictionaryFile must not be null"); } + if (mode == null) { + throw new IllegalArgumentException("mode must not be null"); + } try (InputStream affix = Files.newInputStream(affixFile); InputStream dictionary = Files.newInputStream(dictionaryFile)) { - return load(affix, dictionary); + return loadStreams(affix, dictionary, mode, affixFile.toString()); } } /** - * Loads a dictionary from its two streams. Each stream is buffered up to + * Loads affix and dictionary streams using {@link LoadMode#STRICT}. + * Each stream is buffered up to * {@link #MAX_STREAM_BYTES} bytes; a larger stream fails with {@link IOException}. * * @param affixStream The {@code .aff} affix content. Must not be {@code null}. Not @@ -319,33 +489,80 @@ public static HunspellDictionary load(Path affixFile, Path dictionaryFile) * {@code null}. Not closed. * @return The loaded dictionary. Never {@code null}. * @throws IOException Thrown if reading fails, a stream exceeds - * {@link #MAX_STREAM_BYTES}, or the content is malformed. + * {@link #MAX_STREAM_BYTES}, the content is malformed, or an affix directive + * is unsupported. * @throws IllegalArgumentException Thrown if a parameter is {@code null}. */ public static HunspellDictionary load(InputStream affixStream, InputStream dictionaryStream) throws IOException { + return load(affixStream, dictionaryStream, LoadMode.STRICT); + } + + /** + * Loads affix and dictionary streams with the selected directive policy. + * The streams are not closed. + * + * @param affixStream The non-null {@code .aff} affix content. + * @param dictionaryStream The non-null {@code .dic} word-list content. + * @param mode The non-null loading policy. + * @return The loaded dictionary, with diagnostics for skipped unsupported directives. + * @throws IOException If reading fails, a stream exceeds {@link #MAX_STREAM_BYTES}, + * content is malformed, or strict loading encounters an unsupported directive. + * @throws IllegalArgumentException If a parameter is null. + */ + public static HunspellDictionary load(InputStream affixStream, + InputStream dictionaryStream, LoadMode mode) throws IOException { if (affixStream == null) { throw new IllegalArgumentException("affixStream must not be null"); } if (dictionaryStream == null) { throw new IllegalArgumentException("dictionaryStream must not be null"); } - byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream"); + if (mode == null) { + throw new IllegalArgumentException("mode must not be null"); + } + return loadStreams(affixStream, dictionaryStream, mode, AFFIX_STREAM); + } + + /** + * Returns skipped unsupported directives, one entry per name in encounter order. + * Recognized settings outside stemming, such as suggestion tables, are excluded. + * + * @return An immutable list; empty for a dictionary loaded in strict mode. + */ + public List getUnsupportedDirectives() { + return unsupportedDirectives; + } + + /** + * Loads validated streams without closing them. + * + * @param affixStream The affix input. + * @param dictionaryStream The word-list input. + * @param mode The directive policy. + * @param source The affix source used in unsupported-directive diagnostics. + * @return The parsed dictionary. + * @throws IOException If reading, validation, or parsing fails. + */ + private static HunspellDictionary loadStreams(InputStream affixStream, + InputStream dictionaryStream, LoadMode mode, String source) throws IOException { + byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, AFFIX_STREAM); final Charset charset = declaredCharset(affixBytes); - maskIgnoredAffixLines(affixBytes); + final List unsupported = maskIgnoredAffixLines( + affixBytes, mode, source); final boolean rawUtf8Flags = StandardCharsets.UTF_8.equals(charset) && !usesUnicodeOrNumericFlags(affixBytes); affixBytes = normalizeUtf8ByteFlags(affixBytes, charset); - final AffixFile affix = parseAffix(decode(affixBytes, charset, "affix stream")); + final AffixFile affix = parseAffix(decode(affixBytes, charset, AFFIX_STREAM)); byte[] dictionaryBytes = readBounded(dictionaryStream, MAX_STREAM_BYTES, - "dictionary stream"); + DICTIONARY_STREAM); if (rawUtf8Flags) { dictionaryBytes = normalizeDictionaryByteFlags(dictionaryBytes); } final Map> entries = parseWordList( - decode(dictionaryBytes, charset, "dictionary stream"), - affix.flagMode, affix.flagAliases); - return new HunspellDictionary(entries, affix); + decode(dictionaryBytes, charset, DICTIONARY_STREAM), + affix); + return new HunspellDictionary(entries, affix, unsupported); } /** @@ -355,12 +572,24 @@ public static HunspellDictionary load(InputStream affixStream, * while malformed bytes in parsed directives are still reported. * * @param bytes The buffered affix file, modified in place. + * @param mode The directive policy applied before masking. + * @param source The affix source description. + * @return Unsupported directives skipped in partial mode, in encounter order. + * @throws IOException If strict loading encounters an unsupported directive. */ - private static void maskIgnoredAffixLines(byte[] bytes) { + private static List maskIgnoredAffixLines(byte[] bytes, + LoadMode mode, String source) throws IOException { + final Map unsupported = new LinkedHashMap<>(); + final boolean useReplacements = hasAffixDirective(bytes, "CHECKCOMPOUNDREP"); int lineStart = 0; + int lineNumber = 1; for (int i = 0; i <= bytes.length; i++) { if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') { int fieldStart = lineStart; + if (lineStart == 0 && i >= 3 && bytes[0] == (byte) 0xef + && bytes[1] == (byte) 0xbb && bytes[2] == (byte) 0xbf) { + fieldStart = 3; + } while (fieldStart < i && isAsciiFieldSpace(bytes[fieldStart])) { fieldStart++; } @@ -368,42 +597,170 @@ private static void maskIgnoredAffixLines(byte[] bytes) { while (fieldEnd < i && !isAsciiFieldSpace(bytes[fieldEnd])) { fieldEnd++; } + boolean parsed = false; if (fieldStart < fieldEnd && bytes[fieldStart] != '#') { final String directive = new String(bytes, fieldStart, fieldEnd - fieldStart, StandardCharsets.US_ASCII); - if (isParsedAffixDirective(directive)) { - maskInlineComment(bytes, fieldEnd, i); - lineStart = i + 1; - continue; + if (isParsedAffixDirective(directive) && (!REPLACEMENT_TAG.equals(directive) || useReplacements)) { + maskInlineComment(bytes, fieldStart, i, directive); + parsed = true; + } else if (!isIgnoredAffixDirective(directive)) { + if (mode == LoadMode.STRICT) { + throw new IOException("unsupported affix directive " + directive + + " in " + source + " at line " + lineNumber + + "; use LoadMode.ALLOW_PARTIAL to load without this behavior"); + } + if (!unsupported.containsKey(directive)) { + unsupported.put(directive, + new UnsupportedDirective(directive, source, lineNumber)); + } } } - Arrays.fill(bytes, lineStart, i, (byte) ' '); + if (!parsed) { + Arrays.fill(bytes, lineStart, i, (byte) ' '); + } + if (i < bytes.length && bytes[i] == '\r' + && i + 1 < bytes.length && bytes[i + 1] == '\n') { + i++; + } lineStart = i + 1; + lineNumber++; + } + } + return List.copyOf(unsupported.values()); + } + + /** + * Finds a directive before metadata is removed or decoded. + * + * @param bytes The affix content. + * @param directive The requested first field. + * @return Whether the directive occurs outside a comment. + */ + private static boolean hasAffixDirective(byte[] bytes, String directive) { + final int[] starts = new int[1]; + final int[] ends = new int[1]; + int from = bytes.length >= 3 && bytes[0] == (byte) 0xef + && bytes[1] == (byte) 0xbb && bytes[2] == (byte) 0xbf ? 3 : 0; + for (int to = from; to <= bytes.length; to++) { + if (to == bytes.length || bytes[to] == '\r' || bytes[to] == '\n') { + if (findAsciiFields(bytes, from, to, starts, ends) > 0 + && directive.equals(asciiField(bytes, starts[0], ends[0]))) { + return true; + } + from = to + 1; } } + return false; + } + + /** + * Identifies metadata, suggestion and command-line settings unused by stemming. + * REP is parsed separately when CHECKCOMPOUNDREP makes it affect recognition. + * + * @param directive The affix directive name. + * @return Whether the directive can be ignored by the affix stemmer. + * @see Hunspell format + */ + private static boolean isIgnoredAffixDirective(String directive) { + return switch (directive) { + case "NAME", "HOME", "VERSION", "KEY", "TRY", REPLACEMENT_TAG, "MAP", "PHONE", + "NOSUGGEST", "MAXCPDSUGS", "MAXNGRAMSUGS", "MAXDIFF", "ONLYMAXDIFF", + "NOSPLITSUGS", "SUGSWITHDOTS", "SUBSTANDARD", "WORDCHARS", + "COMPOUNDFIRST", "COMPOUNDLAST", "ONLYROOT", "HU_KOTOHANGZO", "GENERATE" -> true; + default -> false; + }; } - /** {@return whether a byte separates fields in an affix line} */ + /** + * Tests an affix field separator. + * + * @param value The input byte. + * @return Whether the byte separates fields. + */ private static boolean isAsciiFieldSpace(byte value) { return value == ' ' || value == '\t' || value == '\f'; } - /** Replaces an inline comment that starts after an affix field separator. */ - private static void maskInlineComment(byte[] bytes, int from, int to) { - boolean fieldStart = false; - for (int i = from; i < to; i++) { - if (isAsciiFieldSpace(bytes[i])) { - fieldStart = true; - } else if (fieldStart && bytes[i] == '#') { - Arrays.fill(bytes, i, to, (byte) ' '); + /** + * Replaces a trailing comment with spaces. A number sign starts a comment only in a + * field the directive does not consume, because the format allows {@code #} as a + * flag, a separator, affix material, and conversion text. The reference + * implementation ignores the fields after the ones it reads. + * + * @param bytes The mutable file content. + * @param from The first byte of the directive. + * @param to The exclusive end of the line. + * @param directive The directive name. + */ + private static void maskInlineComment(byte[] bytes, int from, int to, String directive) { + final List fields = new ArrayList<>(); + for (int i = from; i < to;) { + while (i < to && isAsciiFieldSpace(bytes[i])) { + i++; + } + final int start = i; + while (i < to && !isAsciiFieldSpace(bytes[i])) { + i++; + } + if (i > start) { + fields.add(new int[] {start, i}); + } + } + final int firstComment = firstCommentField(bytes, fields, directive); + for (int index = firstComment; index < fields.size(); index++) { + if (bytes[fields.get(index)[0]] == '#') { + Arrays.fill(bytes, fields.get(index)[0], to, (byte) ' '); return; - } else { - fieldStart = false; } } } - /** {@return whether this implementation parses a directive's fields} */ + /** + * Finds the first field index a trailing comment may occupy. + * + * @param bytes The file content. + * @param fields The field boundaries of the line. + * @param directive The directive name. + * @return The index after the fields the directive consumes. + */ + private static int firstCommentField(byte[] bytes, List fields, String directive) { + return switch (directive) { + case PREFIX_TAG, SUFFIX_TAG -> isAffixHeader(bytes, fields) ? 4 : 5; + case REPLACEMENT_TAG, INPUT_CONVERSION_TAG, OUTPUT_CONVERSION_TAG, + COMPOUND_PATTERN_TAG, MORPHOLOGY_ALIAS_TAG -> Integer.MAX_VALUE; + case COMPOUND_SYLLABLE_TAG -> 3; + default -> 2; + }; + } + + /** + * Distinguishes an affix block header, which carries a cross-product marker and a + * rule count, from a rule line. + * + * @param bytes The file content. + * @param fields The field boundaries of the line. + * @return Whether the line is a block header. + */ + private static boolean isAffixHeader(byte[] bytes, List fields) { + if (fields.size() < 4 || fields.get(2)[1] - fields.get(2)[0] != 1 + || (bytes[fields.get(2)[0]] != 'Y' && bytes[fields.get(2)[0]] != 'N')) { + return false; + } + for (int i = fields.get(3)[0]; i < fields.get(3)[1]; i++) { + if (bytes[i] < '0' || bytes[i] > '9') { + return false; + } + } + return true; + } + + /** + * Tests whether a directive is supported by the parser. + * + * @param directive The directive name. + * @return Whether the parser processes the directive's fields. + */ private static boolean isParsedAffixDirective(String directive) { return switch (directive) { case SET_TAG, FLAG_TAG, ALIAS_TAG, PREFIX_TAG, SUFFIX_TAG, @@ -411,7 +768,12 @@ private static boolean isParsedAffixDirective(String directive) { "COMPOUNDPERMITFLAG", "COMPOUNDFORBIDFLAG", "NEEDAFFIX", "PSEUDOROOT", "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX", "COMPOUNDMIN", "COMPOUNDWORDMAX", "CHECKCOMPOUNDDUP", "CHECKCOMPOUNDCASE", - "CHECKCOMPOUNDTRIPLE", "FULLSTRIP" -> true; + "CHECKCOMPOUNDTRIPLE", "FULLSTRIP", INPUT_CONVERSION_TAG, OUTPUT_CONVERSION_TAG, + IGNORE_TAG, MORPHOLOGY_ALIAS_TAG, COMPLEX_PREFIXES_TAG, KEEP_CASE_TAG, WARNING_TAG, + FORBID_WARNING_TAG, LANGUAGE_TAG, CHECK_SHARPS_TAG, + COMPOUND_RULE_TAG, COMPOUND_ROOT_TAG, FORCE_UPPER_CASE_TAG, COMPOUND_MORE_SUFFIXES_TAG, + SIMPLIFIED_TRIPLE_TAG, "CHECKCOMPOUNDREP", COMPOUND_PATTERN_TAG, + COMPOUND_SYLLABLE_TAG, SYLLABLE_NUMBER_TAG, LEMMA_PRESENT_TAG, REPLACEMENT_TAG, BREAK_TAG -> true; default -> false; }; } @@ -445,7 +807,12 @@ private static byte[] normalizeUtf8ByteFlags(byte[] bytes, Charset charset) { return normalized.toByteArray(); } - /** {@return whether {@code FLAG UTF-8} or {@code FLAG num} selects non-byte flags} */ + /** + * Tests for UTF-8 or numeric flag encoding. + * + * @param bytes The affix content. + * @return Whether {@code FLAG UTF-8} or {@code FLAG num} is configured. + */ private static boolean usesUnicodeOrNumericFlags(byte[] bytes) { final int[] starts = new int[5]; final int[] fieldEnds = new int[5]; @@ -464,7 +831,14 @@ private static boolean usesUnicodeOrNumericFlags(byte[] bytes) { return false; } - /** Writes one affix line, converting high bytes only within raw flag fields. */ + /** + * Writes one affix line, converting high bytes only within raw flag fields. + * + * @param target The converted content destination. + * @param bytes The source file content. + * @param lineStart The first byte of the line. + * @param lineEnd The exclusive end of the line. + */ private static void writeNormalizedFlagLine(ByteArrayOutputStream target, byte[] bytes, int lineStart, int lineEnd) { final int[] starts = new int[5]; @@ -476,7 +850,8 @@ private static void writeNormalizedFlagLine(ByteArrayOutputStream target, int continuationEnd = -1; if (count >= 2) { final String directive = asciiField(bytes, starts[0], fieldEnds[0]); - if (ALIAS_TAG.equals(directive) || PREFIX_TAG.equals(directive) + if (ALIAS_TAG.equals(directive) || COMPOUND_RULE_TAG.equals(directive) + || PREFIX_TAG.equals(directive) || SUFFIX_TAG.equals(directive) || isSingleFlagDirective(directive)) { firstFlagStart = starts[1]; firstFlagEnd = fieldEnds[1]; @@ -490,6 +865,22 @@ private static void writeNormalizedFlagLine(ByteArrayOutputStream target, } } } + if (count >= 3 && COMPOUND_PATTERN_TAG.equals(directive)) { + for (int i = starts[1]; i < fieldEnds[1]; i++) { + if (bytes[i] == '/') { + firstFlagStart = i + 1; + firstFlagEnd = fieldEnds[1]; + break; + } + } + for (int i = starts[2]; i < fieldEnds[2]; i++) { + if (bytes[i] == '/') { + continuationStart = i + 1; + continuationEnd = fieldEnds[2]; + break; + } + } + } } for (int i = lineStart; i < lineEnd; i++) { final boolean flagByte = i >= firstFlagStart && i < firstFlagEnd @@ -535,7 +926,13 @@ private static byte[] normalizeDictionaryByteFlags(byte[] bytes) { return normalized.toByteArray(); } - /** Writes a raw byte, converting a high flag byte to the matching UTF-8 code point. */ + /** + * Writes a raw byte, converting a high flag byte to a UTF-8 code point. + * + * @param target The converted content destination. + * @param source The source byte. + * @param flagByte Whether this byte represents a flag. + */ private static void writeNormalizedByte(ByteArrayOutputStream target, byte source, boolean flagByte) { final int value = source & 0xff; @@ -547,17 +944,32 @@ private static void writeNormalizedByte(ByteArrayOutputStream target, byte sourc } } - /** {@return whether the directive value is one Hunspell flag} */ + /** + * Tests for a directive with one flag argument. + * + * @param directive The directive name. + * @return Whether the value is one Hunspell flag. + */ private static boolean isSingleFlagDirective(String directive) { return switch (directive) { case "COMPOUNDFLAG", "COMPOUNDBEGIN", "COMPOUNDMIDDLE", "COMPOUNDEND", "COMPOUNDPERMITFLAG", "COMPOUNDFORBIDFLAG", "NEEDAFFIX", "PSEUDOROOT", - "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX" -> true; + "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX", KEEP_CASE_TAG, WARNING_TAG, + COMPOUND_ROOT_TAG, FORCE_UPPER_CASE_TAG, LEMMA_PRESENT_TAG -> true; default -> false; }; } - /** Finds the fields needed to classify one raw line. */ + /** + * Finds the fields needed to classify one raw line. + * + * @param bytes The source content. + * @param from The first byte of the line. + * @param to The exclusive end of the line. + * @param starts The destination for field start offsets. + * @param fieldEnds The destination for exclusive field end offsets. + * @return The number of fields found, limited to the destination array length. + */ private static int findAsciiFields(byte[] bytes, int from, int to, int[] starts, int[] fieldEnds) { int count = 0; @@ -579,7 +991,14 @@ private static int findAsciiFields(byte[] bytes, int from, int to, return count; } - /** Returns one raw ASCII field. */ + /** + * Extracts one ASCII field. + * + * @param bytes The source content. + * @param from The first byte of the field. + * @param to The exclusive end of the field. + * @return The field text. + */ private static String asciiField(byte[] bytes, int from, int to) { return new String(bytes, from, to - from, StandardCharsets.US_ASCII); } @@ -636,23 +1055,141 @@ static byte[] readBounded(InputStream in, int maxBytes, String label) } /** - * Looks up a word's flag sets. + * Looks up a word's flag sets among the listed entries. * * @param word The word exactly as listed. * @return The flag sets of all matching entries, or {@code null} when absent. */ List lookup(String word) { + return lookup(word, false); + } + + /** + * Looks up a word's flag sets, optionally including the hidden capitalized forms + * that all-uppercase input may match. + * + * @param word The word exactly as listed or in hidden capitalized form. + * @param includeHidden Whether hidden capitalized forms count, which the reference + * implementation allows for all-uppercase input outside compounds. + * @return The flag sets of all matching entries, or {@code null} when absent. + */ + List lookup(String word, boolean includeHidden) { final List found = entries.get(word); - if (found == null) { + final List hidden = includeHidden ? hiddenEntries.get(word) : null; + if (found == null && hidden == null) { return null; } - final List copy = new ArrayList<>(found.size()); - for (final int[] flags : found) { - copy.add(flags.clone()); - } + final List copy = new ArrayList<>(); + copyFlags(found, copy); + copyFlags(hidden, copy); return copy; } + /** + * Appends defensive copies of flag sets. + * + * @param source The flag sets to copy, or {@code null}. + * @param target The destination. + */ + private static void copyFlags(List source, List target) { + if (source != null) { + for (final int[] flags : source) { + target.add(flags.clone()); + } + } + } + + /** + * The capitalization classes of the reference implementation. + */ + enum CaseType { + /** No uppercase letter. */ + NOCAP, + /** Exactly one uppercase letter, at the start. */ + INITCAP, + /** Every letter uppercase. */ + ALLCAP, + /** An uppercase start and a further uppercase letter among lowercase ones. */ + HUHINITCAP, + /** Uppercase letters after a lowercase start or among lowercase ones. */ + HUHCAP + } + + /** + * Classifies a word's capitalization as the reference implementation does, judging + * the initial by the first character. + * + * @param word The word to classify. + * @return The capitalization class. + */ + static CaseType caseType(String word) { + int letters = 0; + int uppers = 0; + boolean firstUpper = false; + for (int i = 0; i < word.length();) { + final int point = word.codePointAt(i); + if (Character.isLowerCase(point)) { + letters++; + } else if (Character.isUpperCase(point) || Character.isTitleCase(point)) { + firstUpper |= i == 0; + uppers++; + letters++; + } + i += Character.charCount(point); + } + if (uppers == 0) { + return CaseType.NOCAP; + } + if (uppers == 1 && firstUpper) { + return CaseType.INITCAP; + } + if (uppers == letters) { + return CaseType.ALLCAP; + } + return firstUpper ? CaseType.HUHINITCAP : CaseType.HUHCAP; + } + + /** + * Derives the hidden capitalized forms the reference implementation adds for + * mixed-case entries and for all-uppercase entries carrying flags, so that + * all-uppercase input such as {@code IPOD} or {@code UNICEF'S} matches them. A form + * that is itself listed is not added, and forbidden entries contribute none. + * + * @param listed The words mapped to the flag sets of their entries. + * @return The capitalized forms mapped to the flag sets they share with their entries. + */ + private Map> hiddenCapitalizedEntries(Map> listed) { + final Map> hidden = new HashMap<>(); + for (final Map.Entry> entry : listed.entrySet()) { + final CaseType type = caseType(entry.getKey()); + if (type != CaseType.HUHCAP && type != CaseType.HUHINITCAP && type != CaseType.ALLCAP) { + continue; + } + final String capitalized = upperCaseInitial(lowerCase(entry.getKey())); + if (listed.containsKey(capitalized)) { + continue; + } + for (final int[] flags : entry.getValue()) { + if ((type == CaseType.ALLCAP && flags.length == 0) || contains(flags, forbiddenWord)) { + continue; + } + hidden.computeIfAbsent(capitalized, key -> new ArrayList<>(1)).add(flags); + } + } + return hidden; + } + + /** + * The flag sets of a root's entries, listed or hidden capitalized. + * + * @param root The entry spelling a reading selected. + * @return The flag sets in list order. Never {@code null}. + */ + private List homonyms(String root) { + final List found = entries.get(root); + return found != null ? found : hiddenEntries.getOrDefault(root, List.of()); + } + /** * The suffix rules whose affix material ends in the given code point, which are the * only material-bearing rules that can be undone from a word ending in it. @@ -688,7 +1225,270 @@ List prefixesWithoutMaterial() { /** {@return whether the affix file declares any compounding flag at all} */ boolean compoundsDeclared() { return compoundFlag != 0 || compoundBegin != 0 || compoundEnd != 0 - || compoundMiddle != 0; + || compoundMiddle != 0 || !compoundRules.isEmpty(); + } + + /** {@return the parsed independent compound flag patterns} */ + List compoundRules() { + return compoundRules; + } + + /** {@return the compound-boundary patterns} */ + List compoundPatterns() { + return compoundPatterns; + } + + /** {@return whether simplified triple-letter junctions are enabled} */ + boolean simplifiedTriple() { + return simplifiedTriple; + } + + /** {@return whether compounds permit an additional suffix level} */ + boolean compoundMoreSuffixes() { + return compoundMoreSuffixes; + } + + /** {@return the word separators, with optional start and end anchors} */ + List wordBreaks() { + return wordBreaks; + } + + /** + * Counts an entry toward COMPOUNDWORDMAX. + * + * @param flags The selected entry flags. + * @param affixes The applied rules. + * @return One unit, or an additional unit for COMPOUNDROOT. + */ + int compoundUnits(int[] flags, List affixes) { + int units = contains(flags, compoundRoot) ? 2 : 1; + if (hungarian) { + for (Affix affix : affixes) { + if (!affix.suffix() && countSyllables(affix.affix()) > 1) { + units++; + } + } + } + return units; + } + + /** + * Checks word and syllable limits on a completed compound. + * + * @param syllables The syllable count after language adjustments. + * @param units The compound-word count. + * @return Whether the limits permit the compound. + */ + boolean compoundSizeAllowed(int syllables, int units) { + if (compoundWordMax == 0 || units <= compoundWordMax) { + return true; + } + if (maxCompoundSyllables == 0) { + return false; + } + return syllables <= maxCompoundSyllables; + } + + /** + * Calculates a component's contribution to the compound syllable limit. + * + * @param part The component text. + * @param flags The selected entry flags. + * @param affixes The applied rules in order. + * @param last Whether this is the closing component. + * @return The adjusted vowel count. + */ + int compoundSyllables(String part, int[] flags, List affixes, boolean last) { + if (!hungarian) { + return last ? countSyllables(part) : 0; + } + int result = countSyllables(part); + if (!last) { + return result; + } + Affix suffix = null; + for (Affix affix : affixes) { + if (affix.suffix()) { + suffix = affix; + } + } + if (affixes.isEmpty() && contains(flags, 'I') && !contains(flags, 'J')) { + return result - 1; + } + if (suffix != null) { + if (suffix.continuation().length == 0) { + result -= countSyllables(suffix.affix()); + } else if (suffix.affix().endsWith("i") && !suffix.affix().endsWith("yi") + && !suffix.affix().endsWith("ti")) { + result--; + } + if (syllableNumber) { + if (suffix.flag() == 'c') { + result += 2; + } else if (suffix.flag() == 'J' || (suffix.flag() == 'I' && contains(flags, 'J'))) { + result++; + } + } + } + return result; + } + + /** + * Counts code points listed by COMPOUNDSYLLABLE. + * + * @param text The text to count. + * @return The vowel count. + */ + private int countSyllables(String text) { + int syllables = 0; + for (int at = 0; at < text.length();) { + final int point = text.codePointAt(at); + if (compoundVowels.indexOf(point) >= 0) { + syllables++; + } + at += Character.charCount(point); + } + return syllables; + } + + /** + * {@return whether the Hungarian moving rule applies} The reference implementation + * checks the part of a Hungarian word before a hyphen as a compound with relaxed + * rules: the opening part may qualify through the hardwired flags {@code F}, + * {@code G}, and {@code H}, a compound-forbidden opening entry is allowed, and the + * size limits do not apply. + */ + boolean hyphenMovingRule() { + return hungarian; + } + + /** + * Checks whether an entry may open a compound under the Hungarian moving rule. + * + * @param flags One entry's flag set. + * @return {@code true} if the entry carries one of the hardwired opening flags and is + * not forbidden. + */ + boolean opensHyphenatedCompound(int[] flags) { + if (contains(flags, forbiddenWord)) { + return false; + } + for (final int flag : HUNGARIAN_HYPHEN_FLAGS) { + if (contains(flags, flag)) { + return true; + } + } + return false; + } + + /** + * Checks a closing entry's capitalization requirement. + * + * @param flags The closing entry flags. + * @param word The input compound. + * @return Whether FORCEUCASE permits the input. + */ + boolean compoundCaseAllowed(int[] flags, String word) { + return !contains(flags, forceUpperCase) || Character.isUpperCase(word.codePointAt(0)) + || Character.isTitleCase(word.codePointAt(0)); + } + + /** + * Tests CHECKCOMPOUNDREP with the configured replacement table. + * + * @param word The candidate compound. + * @param recognized Tests a non-compound form. + * @return Whether a replacement invalidates the compound reading. + */ + boolean rejectsCompoundReplacement(String word, Predicate recognized) { + return checkCompoundRep && replacements.anyReplacement(word, recognized); + } + + /** + * A compound-boundary restriction and optional spelling replacement. + * + * @param end The left-side ending. + * @param endFlag The required left flag or zero. + * @param begin The right-side beginning. + * @param beginFlag The required right flag or zero. + * @param unaffixed Whether the left side must have no nonzero affix. + * @param replacement The simplified junction, or null. + */ + record CompoundPattern(String end, int endFlag, String begin, int beginFlag, + boolean unaffixed, String replacement) { + /** + * Checks the selected readings at a junction. + * + * @param left The left part spelling. + * @param leftFlags The left entry flags. + * @param right The right part spelling. + * @param rightFlags The right entry flags. + * @param leftAffixes The left part's affixes. + * @param rightAffixes The right part's affixes. + * @return Whether this pattern applies. + */ + boolean matches(String left, int[] leftFlags, String right, int[] rightFlags, + List leftAffixes, List rightAffixes) { + if (!left.endsWith(end) || !right.startsWith(begin) + || !hasBoundaryFlag(leftFlags, leftAffixes, endFlag) + || !hasBoundaryFlag(rightFlags, rightAffixes, beginFlag)) { + return false; + } + if (unaffixed) { + for (Affix affix : leftAffixes) { + if (!affix.affix().isEmpty() || !affix.strip().isEmpty()) { + return false; + } + } + } + return true; + } + + /** + * Tests entry and continuation flags for a boundary condition. + * + * @param flags The entry flags. + * @param affixes The component's rules. + * @param required The required flag, or zero for an unrestricted condition. + * @return Whether the selected reading satisfies the condition. + */ + private boolean hasBoundaryFlag(int[] flags, List affixes, int required) { + if (required == 0 || contains(flags, required)) { + return true; + } + for (Affix affix : affixes) { + if (affix.allowsContinuation(required)) { + return true; + } + } + return false; + } + } + + /** {@return whether positional or general compound flags are configured} */ + boolean positionalCompoundsDeclared() { + return compoundFlag != 0 || compoundBegin != 0 || compoundMiddle != 0 || compoundEnd != 0; + } + + /** + * Checks blocking flags on a compound-rule part. + * + * @param flags The selected entry flags. + * @param last Whether this is the closing part. + * @param affixes The applied suffixes. + * @return Whether the selected entry and affixes can form a compound part. + */ + boolean acceptsRulePart(int[] flags, boolean last, Affix... affixes) { + if (contains(flags, forbiddenWord) || (!last && contains(flags, compoundForbid)) + || (affixes.length == 0 && contains(flags, needAffix))) { + return false; + } + for (Affix affix : affixes) { + if (forbidsInCompound(affix) || circumfixOnly(affix)) { + return false; + } + } + return affixes.length != 1 || !needsFurtherAffix(affixes[0]); } /** {@return the smallest length a compound part may have} At least {@code 1}. */ @@ -721,6 +1521,241 @@ boolean fullStrip() { return fullStrip; } + /** + * Applies input conversion followed by ignored-character removal. + * + * @param word The supplied word. + * @return The dictionary lookup form. + */ + String inputForm(String word) { + return removeIgnored(inputConversion.apply(word), ignoredCharacters); + } + + /** + * Applies output conversion to a recognized stem. + * + * @param stem The dictionary stem. + * @return The converted output. + */ + String outputForm(String stem) { + return outputConversion.apply(stem); + } + + /** {@return whether prefix continuation allows an additional prefix level} */ + boolean complexPrefixes() { + return complexPrefixes; + } + + /** {@return whether uppercase SS may represent a German sharp s} */ + boolean checkSharps() { + return checkSharps; + } + + /** + * Applies the dictionary's lowercase mapping. + * + * @param text The input text. + * @return The lowercase text. + */ + String lowerCase(String text) { + if (!turkicCase) { + return StringUtil.toLowerCase(text); + } + return StringUtil.toLowerCase(text.replace('I', 'ı').replace('İ', 'i')); + } + + /** + * Checks case-sensitive and warning restrictions for a selected entry. + * + * @param flags The homonym's flags. + * @param surface The input after conversion. + * @param variant The case variant under analysis. + * @param affixes The applied affixes. + * @return Whether this reading is permitted. + */ + boolean acceptsCase(int[] flags, String surface, String variant, Affix... affixes) { + if (forbidWarn && contains(flags, warningFlag)) { + return false; + } + boolean preserveCase = contains(flags, keepCase); + for (Affix affix : affixes) { + if (affix.allowsContinuation(forbiddenWord) + || (forbidWarn && affix.allowsContinuation(warningFlag))) { + return false; + } + preserveCase |= keepCase != 0 && affix.allowsContinuation(keepCase); + } + if (!preserveCase || surface.equals(variant)) { + return true; + } + // CHECKSHARPS lets a case-preserving entry with a sharp s be written all-uppercase + // with SS, or capitalized; an all-uppercase form with a capital sharp s stays rejected + return checkSharps && variant.indexOf('ß') >= 0 + && (surface.contains("SS") || (caseType(surface) != CaseType.ALLCAP + && Character.isUpperCase(surface.codePointAt(0)))); + } + + /** + * Returns a stem after applying entry and affix morphology. + * + * @param root The dictionary entry spelling. + * @param flags The selected homonym's flags. + * @param affixes The affixes in application order. + * @return The stems of matching homonyms. + */ + List morphologicalStems(String root, int[] flags, Affix... affixes) { + final List stems = new ArrayList<>(); + for (int[] entryFlags : homonyms(root)) { + if (Arrays.equals(entryFlags, flags)) { + stems.add(morphologicalStem(root, morphology.getOrDefault(entryFlags, List.of()), affixes)); + } + } + return stems; + } + + /** + * Returns morphological fields for matching dictionary entries and applied rules, in + * the reference implementation's field order. A prefix without morphological fields + * contributes its affix text when no suffix follows and its {@code fl:} flag field + * otherwise; after the entry fields, an entry without fields contributes that prefix's + * {@code fl:} field. A bare closing compound part without entry fields contributes + * no stem field. + * + * @param root The dictionary entry text. + * @param flags The selected homonym flags. + * @param compoundEnd Whether the reading closes a compound. + * @param affixes The applied rules in order. + * @return The complete field strings, possibly empty strings. + */ + List morphologicalAnalyses(String root, int[] flags, boolean compoundEnd, + Affix... affixes) { + final List result = new ArrayList<>(); + boolean hasSuffix = false; + Affix prefixOnly = null; + for (Affix affix : affixes) { + hasSuffix |= affix.suffix(); + } + for (int[] entryFlags : homonyms(root)) { + if (!Arrays.equals(entryFlags, flags)) { + continue; + } + final List fields = new ArrayList<>(); + for (int i = affixes.length - 1; i >= 0; i--) { + final Affix prefix = affixes[i]; + if (prefix.suffix()) { + continue; + } + if (hasSuffix || !prefix.morphology().isEmpty()) { + fields.addAll(prefix.analysisFields()); + } else { + fields.add(prefix.affix()); + } + if (!hasSuffix) { + prefixOnly = prefix; + } + } + final List entry = morphology.getOrDefault(entryFlags, List.of()); + if (compoundEnd && affixes.length == 0 && entry.isEmpty()) { + result.add(""); + continue; + } + if (!hasField(entry, STEM_FIELD)) { + fields.add(STEM_FIELD + root); + } + fields.addAll(entry); + if (entry.isEmpty() && prefixOnly != null) { + fields.add(FLAG_FIELD + prefixOnly.flagText()); + } + for (Affix affix : affixes) { + if (affix.suffix()) { + fields.addAll(affix.analysisFields()); + } + } + result.add(String.join(" ", fields)); + } + return result; + } + + /** + * Applies morphological stem and affix fields to one dictionary entry. + * + * @param root The entry spelling. + * @param fields The entry's morphological fields. + * @param affixes The affixes in application order. + * @return The morphological stem. + */ + private String morphologicalStem(String root, List fields, Affix... affixes) { + String result = fieldValue(fields, STEM_FIELD, root); + if (hasField(fields, DERIVATIONAL_SUFFIX_FIELD)) { + result = root; + } + // A derivational suffix makes the derived form the stem. The reference + // implementation generates that form from the entry and its suffixes alone; prefix + // material appears in the stem only through a surface prefix field. + String derived = root; + final StringBuilder surfacePrefix = new StringBuilder(fieldValue(fields, SURFACE_PREFIX_FIELD, "")); + for (Affix affix : affixes) { + if (affix.suffix()) { + derived = derived.substring(0, derived.length() - affix.strip().length()) + affix.affix(); + if (hasField(affix.morphology(), DERIVATIONAL_SUFFIX_FIELD)) { + result = derived; + } + } + surfacePrefix.append(fieldValue(affix.morphology(), SURFACE_PREFIX_FIELD, "")); + } + return surfacePrefix.append(result).toString(); + } + + /** + * Finds a morphological field value. + * + * @param fields The expanded fields. + * @param tag The tag including the colon. + * @param fallback The value used when the tag is not present. + * @return The first field value or fallback. + */ + private String fieldValue(List fields, String tag, String fallback) { + for (String field : fields) { + if (field.startsWith(tag)) { + return field.substring(tag.length()); + } + } + return fallback; + } + + /** + * Checks for a morphological field. + * + * @param fields The expanded fields. + * @param tag The tag including the colon. + * @return Whether the tag is present. + */ + private boolean hasField(List fields, String tag) { + return fieldValue(fields, tag, null) != null; + } + + /** + * Removes characters listed in the affix file's IGNORE setting. + * + * @param text The word or affix material. + * @param ignored The characters to remove. + * @return Text without the ignored characters. + */ + private static String removeIgnored(String text, String ignored) { + if (ignored.isEmpty()) { + return text; + } + final StringBuilder result = new StringBuilder(text.length()); + for (int i = 0; i < text.length();) { + final int point = text.codePointAt(i); + if (ignored.indexOf(point) < 0) { + result.appendCodePoint(point); + } + i += Character.charCount(point); + } + return result.length() == text.length() ? text : result.toString(); + } + /** * The flag admitting a part at a compound position, next to the general * compounding flag. @@ -842,14 +1877,71 @@ private boolean forbiddenAtCompoundPosition(int[] flags, CompoundPosition positi } /** - * Checks whether any of a word's flag sets is forbidden, which a dictionary uses - * to block one specific ill-formed compound while its parts stay productive. + * Checks whether a spelling is forbidden as a whole. Only the first listed homonym + * decides, as in the reference implementation, so a dictionary can list a valid word + * first and a forbidden compound-only homonym after it. * - * @param flagSets The word's flag sets from {@link #lookup(String)}. - * @return {@code true} if some homonym carries the forbidden-word flag. + * @param flagSets The word's flag sets from {@link #lookup(String)}, in list order. + * @return {@code true} if the first homonym carries the forbidden-word flag. + */ + boolean firstForbidden(List flagSets) { + return !flagSets.isEmpty() && contains(flagSets.get(0), forbiddenWord); + } + + /** + * Checks whether an affix analysis reaches a forbidden entry, which forbids the + * affixed spelling and thereby its compound and break readings. + * + * @param flagSets The stem's flag sets from {@link #lookup(String)}. + * @param flag The removed affix's flag. + * @return {@code true} if a homonym carries both the affix flag and the forbidden flag. + */ + boolean forbidsAffixed(List flagSets, int flag) { + for (final int[] flags : flagSets) { + if (contains(flags, flag) && contains(flags, forbiddenWord)) { + return true; + } + } + return false; + } + + /** + * Checks whether a spelling may not open a compound part sequence: its first listed + * homonym carries {@code COMPOUNDFORBIDFLAG}, which the reference implementation + * applies to every part but the last, overriding affixed readings of the same + * spelling. + * + * @param flagSets The part's flag sets from {@link #lookup(String)}, in list order. + * @return {@code true} if the spelling is barred from non-final compound positions. + */ + boolean forbidsCompoundStart(List flagSets) { + return !flagSets.isEmpty() && contains(flagSets.get(0), compoundForbid); + } + + /** + * Converts the initial code point to uppercase with the dictionary's case mapping. + * + * @param word The nonempty word. + * @return The word with an uppercase initial. + */ + String upperCaseInitial(String word) { + final int first = word.codePointAt(0); + final String initial = word.substring(0, Character.charCount(first)); + final String upper = turkicCase && first == 'i' ? "İ" + : turkicCase && first == 'ı' ? "I" : StringUtil.toUpperCase(initial); + return upper + word.substring(initial.length()); + } + + /** + * Converts the initial code point to lowercase with the dictionary's case mapping. + * + * @param word The nonempty word. + * @return The word with a lowercase initial. */ - boolean anyForbidden(List flagSets) { - return hasFlag(flagSets, forbiddenWord); + String lowerCaseInitial(String word) { + final int first = word.codePointAt(0); + final String initial = word.substring(0, Character.charCount(first)); + return lowerCase(initial) + word.substring(initial.length()); } /** @@ -876,7 +1968,7 @@ static boolean hasFlag(List flagSets, int flag) { * @param flag The flag to look for. * @return {@code true} if the set contains the flag. */ - private static boolean contains(int[] flags, int flag) { + static boolean contains(int[] flags, int flag) { if (flag == 0) { return false; } @@ -940,12 +2032,7 @@ boolean supports(List flagSets, int flag) { */ boolean supportsCrossProduct(List flagSets, Affix prefix, Affix suffix) { for (final int[] flags : flagSets) { - final boolean rootHasPrefix = contains(flags, prefix.flag()); - final boolean rootHasSuffix = contains(flags, suffix.flag()); - final boolean licensesBoth = (rootHasPrefix - && (rootHasSuffix || prefix.allowsContinuation(suffix.flag()))) - || (rootHasSuffix && suffix.allowsContinuation(prefix.flag())); - if (licensesBoth && !contains(flags, onlyInCompound) + if (licensesCrossProduct(flags, prefix, suffix) && !contains(flags, onlyInCompound) && !contains(flags, forbiddenWord)) { return true; } @@ -953,6 +2040,42 @@ boolean supportsCrossProduct(List flagSets, Affix prefix, Affix suffix) { return false; } + /** + * Checks cross-product flags within a single entry. + * + * @param flags The selected entry flags. + * @param prefix The prefix rule. + * @param suffix The suffix rule. + * @return Whether the entry and continuation classes permit both rules. + */ + boolean licensesCrossProduct(int[] flags, Affix prefix, Affix suffix) { + return (contains(flags, prefix.flag()) + && (contains(flags, suffix.flag()) || prefix.allowsContinuation(suffix.flag()))) + || (contains(flags, suffix.flag()) && suffix.allowsContinuation(prefix.flag())); + } + + /** + * Checks a compound cross-product reading. + * + * @param flags The selected entry flags. + * @param position The component position. + * @param prefix The prefix rule. + * @param suffix The suffix rule. + * @param additional Additional continuation-linked rules. + * @return Whether the entry and rules permit this component. + */ + boolean supportsCompoundCrossProduct(int[] flags, CompoundPosition position, + Affix prefix, Affix suffix, Affix... additional) { + boolean positionAllowed = contains(flags, compoundFlag) || contains(flags, positionalFlag(position)) + || affixAdmits(prefix, position) || affixAdmits(suffix, position); + for (Affix affix : additional) { + positionAllowed |= affixAdmits(affix, position); + } + return licensesCrossProduct(flags, prefix, suffix) && !contains(flags, forbiddenWord) + && !forbiddenAtCompoundPosition(flags, position) + && positionAllowed; + } + /** * Checks whether a form made with this affix alone is still a virtual stem: the * affix carries the {@code NEEDAFFIX} flag among its continuation classes, so a @@ -1003,7 +2126,8 @@ private static Charset declaredCharset(byte[] affixBytes) throws IOException { for (final String line : splitLines(ascii)) { final String trimmed = trim(line); if (trimmed.startsWith(SET_PREFIX) || trimmed.startsWith(SET_TAB_PREFIX)) { - final String name = trim(trimmed.substring(SET_PREFIX.length())); + // the encoding is the first field after the directive; later fields are ignored + final String name = split(trimmed.substring(SET_PREFIX.length()))[0]; try { return Charset.forName(name); } catch (IllegalCharsetNameException | UnsupportedCharsetException e) { @@ -1050,6 +2174,29 @@ private static final class AffixFile { private boolean checkCompoundCase; private boolean checkCompoundTriple; private boolean fullStrip; + private String ignoredCharacters = ""; + private HunspellConversion inputConversion = HunspellConversion.NONE; + private HunspellConversion outputConversion = HunspellConversion.NONE; + private final List> morphologyAliases = new ArrayList<>(); + private final Map> entryMorphology = new HashMap<>(); + private boolean complexPrefixes; + private int keepCase; + private int warningFlag; + private boolean forbidWarn; + private boolean checkSharps; + private String language = ""; + private boolean syllableNumber; + private final List compoundRules = new ArrayList<>(); + private final List compoundPatterns = new ArrayList<>(); + private int compoundRoot; + private int forceUpperCase; + private boolean compoundMoreSuffixes; + private boolean simplifiedTriple; + private boolean checkCompoundRep; + private HunspellConversion replacements = HunspellConversion.NONE; + private int maxCompoundSyllables; + private String compoundVowels = ""; + private List wordBreaks = DEFAULT_WORD_BREAKS; } /** @@ -1071,6 +2218,22 @@ private static AffixFile parseAffix(String content) throws IOException { } result.flagMode = readFlagMode(fieldsByLine); result.flagAliases.addAll(readFlagAliases(fieldsByLine, result.flagMode)); + result.inputConversion = HunspellConversion.parse(fieldsByLine, INPUT_CONVERSION_TAG); + result.outputConversion = HunspellConversion.parse(fieldsByLine, OUTPUT_CONVERSION_TAG); + result.morphologyAliases.addAll(readMorphologyAliases(fieldsByLine)); + result.compoundRules.addAll(readCompoundRules(fieldsByLine, result.flagMode)); + result.compoundPatterns.addAll(readCompoundPatterns(fieldsByLine, result.flagMode)); + result.replacements = HunspellConversion.parse(fieldsByLine, REPLACEMENT_TAG); + result.wordBreaks = readWordBreaks(fieldsByLine); + for (int line = 0; line < fieldsByLine.length; line++) { + final String[] fields = fieldsByLine[line]; + if (fields.length > 0 && IGNORE_TAG.equals(fields[0])) { + if (fields.length != 2) { + throw new IOException("invalid IGNORE at line " + (line + 1)); + } + result.ignoredCharacters = fields[1]; + } + } int i = 0; while (i < lines.length) { final String[] fields = fieldsByLine[i]; @@ -1094,6 +2257,11 @@ private static AffixFile parseAffix(String content) throws IOException { case "ONLYINCOMPOUND": case "FORBIDDENWORD": case "CIRCUMFIX": + case KEEP_CASE_TAG: + case WARNING_TAG: + case COMPOUND_ROOT_TAG: + case FORCE_UPPER_CASE_TAG: + case LEMMA_PRESENT_TAG: if (fields.length < 2) { throw new IOException(fields[0] + " line without a flag at line " + (i + 1)); } @@ -1110,6 +2278,11 @@ private static AffixFile parseAffix(String content) throws IOException { case "ONLYINCOMPOUND" -> result.onlyInCompound = declared; case "CIRCUMFIX" -> result.circumfix = declared; case "FORBIDDENWORD" -> result.forbiddenWord = declared; + case KEEP_CASE_TAG -> result.keepCase = declared; + case WARNING_TAG -> result.warningFlag = declared; + case COMPOUND_ROOT_TAG -> result.compoundRoot = declared; + case FORCE_UPPER_CASE_TAG -> result.forceUpperCase = declared; + case LEMMA_PRESENT_TAG -> { } default -> throw new IOException( "unhandled flag directive " + fields[0] + " at line " + (i + 1)); } @@ -1151,6 +2324,52 @@ private static AffixFile parseAffix(String content) throws IOException { result.fullStrip = true; i++; break; + case COMPLEX_PREFIXES_TAG: + result.complexPrefixes = true; + i++; + break; + case COMPOUND_MORE_SUFFIXES_TAG: + result.compoundMoreSuffixes = true; + i++; + break; + case SIMPLIFIED_TRIPLE_TAG: + result.simplifiedTriple = true; + i++; + break; + case "CHECKCOMPOUNDREP": + result.checkCompoundRep = true; + i++; + break; + case COMPOUND_SYLLABLE_TAG: + result.maxCompoundSyllables = parseValue(fields, i + 1); + if (fields.length != 3 || result.maxCompoundSyllables < 0) { + throw new IOException("invalid COMPOUNDSYLLABLE at line " + (i + 1)); + } + result.compoundVowels = fields[2]; + i++; + break; + case FORBID_WARNING_TAG: + result.forbidWarn = true; + i++; + break; + case CHECK_SHARPS_TAG: + result.checkSharps = true; + i++; + break; + case LANGUAGE_TAG: + if (fields.length != 2) { + throw new IOException("invalid LANG at line " + (i + 1)); + } + result.language = fields[1]; + i++; + break; + case SYLLABLE_NUMBER_TAG: + if (fields.length != 2) { + throw new IOException("invalid SYLLABLENUM at line " + (i + 1)); + } + result.syllableNumber = true; + i++; + break; case ALIAS_TAG: // The file-wide table was parsed before continuation and entry flags. i++; @@ -1298,13 +2517,14 @@ private static int parseAffixBlock(String[][] fieldsByLine, int index, String[] throw new IOException("affix block truncated at line " + (line + 1)); } final String[] fields = fieldsByLine[line]; - if (fields.length < 5 || !fields[0].equals(header[0])) { + if (fields.length < 4 || !fields[0].equals(header[0])) { throw new IOException("malformed affix rule at line " + (line + 1)); } if (parseFlag(fields[1], result.flagMode, line + 1) != flag) { throw new IOException("affix rule flag does not match header at line " + (line + 1)); } - final String strip = NO_MATERIAL.equals(fields[2]) ? "" : fields[2]; + final String strip = NO_MATERIAL.equals(fields[2]) ? "" + : removeIgnored(fields[2], result.ignoredCharacters); String affixText = fields[3]; int[] continuation = new int[0]; final int slash = affixText.indexOf('/'); @@ -1316,8 +2536,14 @@ private static int parseAffixBlock(String[][] fieldsByLine, int index, String[] if (NO_MATERIAL.equals(affixText)) { affixText = ""; } + affixText = removeIgnored(affixText, result.ignoredCharacters); + final boolean hasCondition = fields.length > 4; + final List morphology = parseMorphology( + Arrays.copyOfRange(fields, hasCondition ? 5 : 4, fields.length), + result.morphologyAliases, line + 1); final Affix affix = new Affix(flag, crossProduct, strip, affixText, - AffixCondition.parse(fields[4], suffix, line + 1), continuation); + AffixCondition.parse(hasCondition ? fields[4] : ".", suffix, line + 1), continuation, suffix, + morphology, fields[1]); if (suffix) { result.suffixes.add(affix); } else { @@ -1330,22 +2556,19 @@ private static int parseAffixBlock(String[][] fieldsByLine, int index, String[] /** * Parses the word list: an optional leading entry count, then one entry per line * consisting of the word, an optional {@code /flags} run, and optional trailing - * morphological fields, which are ignored. The morphological fields are cut off + * morphological fields. The morphological fields are separated * first, because the flag separator is only meaningful in what precedes them; a word * may itself contain spaces. A slash escaped as {@code \/} belongs to the word itself * and is unescaped in the stored key. * * @param content The decoded word-list content. - * @param flagMode The flag encoding declared by the affix file. - * @param flagAliases The affix file's {@code AF} alias table, possibly empty. When - * it is not empty, a purely numeric flag field is a 1-based - * reference into it rather than a flag run of its own. + * @param affix The affix settings and destination for entry morphology. * @return The words mapped to the flag sets of their entries. Never {@code null}. * @throws IOException Thrown if a flag run is malformed or an alias reference is * out of range. */ private static Map> parseWordList(String content, - FlagMode flagMode, List flagAliases) throws IOException { + AffixFile affix) throws IOException { final String[] lines = splitLines(withoutByteOrderMark(content)); final Map> entries = new HashMap<>(); int start = 0; @@ -1357,7 +2580,13 @@ private static Map> parseWordList(String content, if (line.isEmpty()) { continue; } - final int morphology = morphologyIndex(line); + int morphology = morphologyIndex(line); + if (morphology < 0 && !affix.morphologyAliases.isEmpty()) { + final int separator = line.lastIndexOf(' '); + if (separator > 0 && isCount(line.substring(separator + 1))) { + morphology = separator; + } + } final String entry = morphology < 0 ? line : trim(line.substring(0, morphology)); String word = entry; int[] flags = new int[0]; @@ -1374,14 +2603,144 @@ private static Map> parseWordList(String content, break; } } - flags = parseAliasedFlags(flagRun, flagMode, flagAliases, i + 1); + flags = parseAliasedFlags(flagRun, affix.flagMode, affix.flagAliases, i + 1).clone(); + } + if (morphology >= 0) { + affix.entryMorphology.put(flags, parseMorphology(split(line.substring(morphology)), + affix.morphologyAliases, i + 1)); } - entries.computeIfAbsent(word.replace("\\/", "/"), key -> new ArrayList<>(1)) + entries.computeIfAbsent(removeIgnored(word.replace("\\/", "/"), affix.ignoredCharacters), + key -> new ArrayList<>(1)) .add(flags); } return entries; } + /** + * Parses the complete AM alias table before entries and affixes. + * + * @param lines The affix fields indexed by source line. + * @return The aliases in reference order. + * @throws IOException If a count or table entry is malformed. + */ + private static List> readMorphologyAliases(String[][] lines) throws IOException { + final List> aliases = new ArrayList<>(); + final List entries = + HunspellAffixTable.read(lines, MORPHOLOGY_ALIAS_TAG, 2, Integer.MAX_VALUE); + if (entries != null) { + for (HunspellAffixTable.Entry entry : entries) { + final String[] fields = entry.fields(); + aliases.add(List.of(Arrays.copyOfRange(fields, 1, fields.length))); + } + } + return aliases; + } + + /** + * Parses and validates the COMPOUNDRULE table. + * + * @param lines The affix fields indexed by line. + * @param mode The flag encoding. + * @return The parsed patterns. + * @throws IOException If a count, pattern, or flag is malformed. + */ + private static List readCompoundRules(String[][] lines, FlagMode mode) + throws IOException { + final List rules = new ArrayList<>(); + final List entries = HunspellAffixTable.read(lines, COMPOUND_RULE_TAG, 2, 2); + if (entries != null) { + for (HunspellAffixTable.Entry entry : entries) { + rules.add(HunspellCompoundRule.parse(entry.fields()[1], entry.line(), + (text, line) -> parseFlag(text, mode, line))); + } + } + return rules; + } + + /** + * Parses CHECKCOMPOUNDPATTERN restrictions and replacements. + * + * @param lines The affix fields. + * @param mode The flag encoding. + * @return The boundary patterns. + * @throws IOException If a declaration is malformed. + */ + private static List readCompoundPatterns(String[][] lines, FlagMode mode) + throws IOException { + final List patterns = new ArrayList<>(); + final List entries = + HunspellAffixTable.read(lines, COMPOUND_PATTERN_TAG, 3, 4); + if (entries == null) { + return patterns; + } + for (HunspellAffixTable.Entry entry : entries) { + final String[] fields = entry.fields(); + final int leftSlash = fields[1].indexOf('/'); + final int rightSlash = fields[2].indexOf('/'); + final String left = leftSlash < 0 ? fields[1] : fields[1].substring(0, leftSlash); + final String right = rightSlash < 0 ? fields[2] : fields[2].substring(0, rightSlash); + patterns.add(new CompoundPattern("0".equals(left) ? "" : left, + leftSlash < 0 ? 0 : parseFlag(fields[1].substring(leftSlash + 1), mode, entry.line()), + right, rightSlash < 0 ? 0 : parseFlag(fields[2].substring(rightSlash + 1), mode, entry.line()), + "0".equals(left), fields.length == 4 ? fields[3] : null)); + } + return patterns; + } + + /** + * Parses BREAK declarations, using hyphen rules when no table is configured. + * + * @param lines The affix fields. + * @return The separators and anchors. + * @throws IOException If the table or a separator is malformed. + */ + private static List readWordBreaks(String[][] lines) throws IOException { + final List entries = HunspellAffixTable.read(lines, BREAK_TAG, 2, 2); + if (entries == null) { + return DEFAULT_WORD_BREAKS; + } + final List result = new ArrayList<>(); + for (HunspellAffixTable.Entry entry : entries) { + final String separator = entry.fields()[1]; + if ("^".equals(separator) || "$".equals(separator) || "^$".equals(separator)) { + throw new IOException("invalid BREAK at line " + entry.line()); + } + result.add(separator); + } + return result; + } + + /** + * Expands an optional morphology alias. + * + * @param fields The morphological fields or alias reference. + * @param aliases The AM table. + * @param line The source line. + * @return The expanded immutable fields. + * @throws IOException If an alias reference is invalid. + */ + private static List parseMorphology(String[] fields, List> aliases, + int line) throws IOException { + if (fields.length == 0) { + return List.of(); + } + if (!aliases.isEmpty()) { + try { + if (fields.length != 1) { + throw new NumberFormatException(); + } + final int index = Integer.parseInt(fields[0]); + if (index < 1 || index > aliases.size()) { + throw new NumberFormatException(); + } + return aliases.get(index - 1); + } catch (NumberFormatException e) { + throw new IOException("invalid AM alias at line " + line, e); + } + } + return List.of(fields); + } + /** * Removes a Unicode byte-order mark decoded at the start of a file. * diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java index c5699fa149..ea8c84155d 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java @@ -18,30 +18,33 @@ package opennlp.tools.stemmer.hunspell; import java.util.ArrayList; +import java.util.HashMap; import java.util.LinkedHashSet; import java.util.List; +import java.util.Map; import java.util.Set; import opennlp.tools.commons.ThreadSafe; import opennlp.tools.stemmer.Stemmer; import opennlp.tools.stemmer.hunspell.HunspellDictionary.Affix; +import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPattern; import opennlp.tools.stemmer.hunspell.HunspellDictionary.CompoundPosition; -import opennlp.tools.util.StringUtil; /** * A dictionary-backed {@link Stemmer} over a {@link HunspellDictionary}: a surface form * is reduced to the dictionary words it can be derived from by removing one suffix, one - * prefix, a cross-product combination of both, or an additional suffix licensed by a - * continuation class. + * prefix, a cross-product combination of both, or an additional affix licensed by a + * continuation class. Compound and break-separated forms return the stems of their + * recognized parts. * *

{@link #stem(CharSequence)} returns the first analysis, preferring the word's own * dictionary entry; {@link #stemAll(CharSequence)} returns every distinct analysis. A * word with no analysis is returned unchanged, so the stemmer degrades to identity on - * unknown vocabulary. A form containing uppercase characters is also analyzed in its - * lowercase variant, so sentence-initial capitalization does not hide an entry. + * unknown input. Title-case and all-uppercase forms also use permitted case + * variants, subject to the dictionary's case restrictions. * Entries the dictionary marks as virtual stems ({@code NEEDAFFIX}), compound-only - * parts ({@code ONLYINCOMPOUND}), or forbidden words ({@code FORBIDDENWORD}) never - * count as standalone analyses, matching how hunspell reads those flags.

+ * parts ({@code ONLYINCOMPOUND}), or forbidden words ({@code FORBIDDENWORD}) do not + * count as standalone analyses.

* *

Compound part search is capped at {@value #PART_CHECK_BUDGET} part-licensing * attempts per input word; beyond that budget further compound analyses are skipped. @@ -55,14 +58,19 @@ public final class HunspellStemmer implements Stemmer { /** - * The most part-licensing attempts one decomposition search may spend. Compounding - * searches every split of every tail, which on adversarial input with a - * one-character minimum part length grows without useful bound; the budget stops - * the search there, missing analyses rather than stalling, in line with the - * engine's fail-closed posture. + * Maximum candidate checks per compound search or recursive word-break search. */ private static final int PART_CHECK_BUDGET = 2048; + /** Maximum parts in one compound analysis. */ + private static final int MAX_COMPOUND_PARTS = 64; + + /** Maximum case variants considered for CHECKSHARPS. */ + private static final int MAX_CASE_VARIANTS = 64; + + /** Maximum distinct morphological readings returned for one input. */ + private static final int MAX_ANALYSES = 2048; + private final HunspellDictionary dictionary; /** @@ -101,37 +109,426 @@ public List stemAll(CharSequence word) { throw new IllegalArgumentException("word must not be null"); } final String surface = word.toString(); - if (surface.isEmpty()) { + final List analyses = findWord(dictionary.inputForm(surface), false); + if (analyses.isEmpty()) { + return List.of(surface); + } + final Set converted = new LinkedHashSet<>(); + for (String analysis : analyses) { + converted.add(dictionary.outputForm(analysis)); + } + return List.copyOf(converted); + } + + /** + * Returns morphological analyses as space-separated Hunspell fields. + * Each result describes a complete accepted reading. Entries without an explicit + * {@code st:} field use the dictionary entry as their stem. Compound components + * begin with {@code pa:}; entry and affix fields follow in application order. + * Results preserve dictionary field text without output conversion. + * + * @param word The input to analyze. Must not be {@code null}. + * @return An immutable list of distinct analyses, or an empty list for unknown input. + * At most {@value #MAX_ANALYSES} analyses are returned. + * @throws IllegalArgumentException If {@code word} is {@code null}. + */ + public List analyze(CharSequence word) { + if (word == null) { + throw new IllegalArgumentException("word must not be null"); + } + return findWord(dictionary.inputForm(word.toString()), true); + } + + /** + * Finds the readings of a complete input. Trailing periods are removed first, as + * the reference implementation does for abbreviations; when the shortened form has + * no reading, one period is restored for entries listed with it. + * + * @param input The input after conversion. + * @param morphological Whether results contain morphological fields. + * @return Recognized stems or analyses, or an empty list. + */ + private List findWord(String input, boolean morphological) { + int end = input.length(); + while (end > 0 && input.charAt(end - 1) == '.') { + end--; + } + if (end == 0) { // a zero-length word has no morphology; without this guard a strip-only rule // could restore its strip string onto nothing and answer a non-empty stem - return List.of(surface); + return List.of(); } - final Set analyses = new LinkedHashSet<>(); - for (final String variant : variants(surface)) { - analyze(variant, analyses); + final List analyses = findStems(input.substring(0, end), 0, + new int[] {PART_CHECK_BUDGET}, new HashMap<>(), morphological); + if (analyses.isEmpty() && end < input.length()) { + return findStems(input.substring(0, end + 1), 0, + new int[] {PART_CHECK_BUDGET}, new HashMap<>(), morphological); } - if (analyses.isEmpty() && dictionary.compoundsDeclared()) { - for (final String variant : variants(surface)) { - decompose(variant, surface, analyses); + return analyses; + } + + /** Accumulates stems or complete morphology readings for one request. */ + private final class Results { + private final boolean morphological; + private final Set values = new LinkedHashSet<>(); + /** + * Whether a direct or affixed reading reached a forbidden entry, which blocks the + * compound and break readings of the same input. + */ + private boolean forbidden; + /** + * Whether compound decomposition follows the Hungarian moving rule for the part of + * a word before a hyphen. + */ + private boolean hyphenatedFirstPart; + + /** + * Selects the output representation. + * + * @param morphological Whether results contain fields instead of stems. + */ + private Results(boolean morphological) { + this.morphological = morphological; + } + + /** {@return whether a reading has not been found} */ + private boolean isEmpty() { + return values.isEmpty(); + } + + /** + * Adds distinct results up to the output limit. + * + * @param additions The accepted results. + */ + private void addAll(List additions) { + for (String value : additions) { + if (values.size() >= MAX_ANALYSES) { + return; + } + values.add(value); } } - if (analyses.isEmpty()) { - return List.of(surface); + + /** + * Adds a complete compound analysis, preserving homonym alternatives. + * + * @param parts The selected component entries and affixes. + */ + private void addCompound(List parts) { + if (!morphological) { + for (CompoundPart part : parts) { + addAll(part.stems()); + } + return; + } + List accumulated = List.of(""); + for (CompoundPart part : parts) { + final List next = new ArrayList<>(); + for (String fields : dictionary.morphologicalAnalyses(part.root(), part.flags(), + part == parts.get(parts.size() - 1), part.affixes().toArray(Affix[]::new))) { + for (String prior : accumulated) { + if (next.size() >= MAX_ANALYSES) { + break; + } + next.add(new StringBuilder(prior).append(prior.isEmpty() ? "" : " ") + .append("pa:").append(part.surface()) + .append(fields.isEmpty() ? "" : " ").append(fields).toString()); + } + } + accumulated = next; + } + addAll(accumulated); + } + + /** + * Combines accepted readings before and after a word break. + * + * @param leftText The opening text. + * @param left The opening readings. + * @param rightText The closing text. + * @param right The closing readings. + */ + private void addBroken(String leftText, List left, String rightText, List right) { + if (!morphological) { + addAll(left); + addAll(right); + return; + } + for (String first : left) { + for (String last : right) { + if (values.size() >= MAX_ANALYSES) { + return; + } + values.add(new StringBuilder() + .append(first.startsWith("pa:") ? "" : "pa:" + leftText + " ").append(first) + .append(' ').append(last.startsWith("pa:") ? "" : "pa:" + rightText + " ") + .append(last).toString()); + } + } } - return List.copyOf(analyses); + } + + /** + * Finds recognized stems without returning identity for unrecognized pieces. + * + * @param input The normalized input or a break-separated component. + * @param depth The current break depth. + * @param budget The remaining break attempts. + * @param cache Results for completed pieces of this input. + * @param morphological Whether results contain morphological fields. + * @return Recognized stems, or an empty list. + */ + private List findStems(String input, int depth, int[] budget, + Map> cache, boolean morphological) { + if (input.isEmpty() || depth >= MAX_COMPOUND_PARTS || budget[0] <= 0) { + return List.of(); + } + final List cached = cache.get(input); + if (cached != null) { + return cached; + } + final List entries = dictionary.lookup(input); + if (entries != null && dictionary.firstForbidden(entries)) { + return List.of(); + } + final Results analyses = new Results(morphological); + final boolean allCaps = HunspellDictionary.caseType(input) == HunspellDictionary.CaseType.ALLCAP; + for (final String variant : variants(input)) { + analyze(variant, new Analysis(input, variant, analyses, allCaps)); + } + // a forbidden direct or affixed reading forbids the spelling as a whole, so no + // compound or break reading is attempted, as in the reference implementation + if (analyses.isEmpty() && !analyses.forbidden && dictionary.compoundsDeclared()) { + for (final String variant : variants(input)) { + decompose(variant, input, analyses); + } + } + if (analyses.isEmpty() && !analyses.forbidden) { + for (String declaration : dictionary.wordBreaks()) { + if (budget[0] <= 0) { + break; + } + final boolean start = declaration.startsWith("^"); + final boolean end = declaration.endsWith("$"); + final String separator = declaration.substring(start ? 1 : 0, + declaration.length() - (end ? 1 : 0)); + for (int at = input.indexOf(separator); at >= 0 && budget[0] > 0; + at = input.indexOf(separator, at + separator.length())) { + final int after = at + separator.length(); + if ((start && at != 0) || (end && after != input.length()) + || (!start && at == 0) || (!end && after == input.length())) { + continue; + } + budget[0]--; + if (start && !end) { + analyses.addAll(findStems(input.substring(after), depth + 1, budget, cache, morphological)); + } else if (end && !start) { + analyses.addAll(findStems(input.substring(0, at), depth + 1, budget, cache, morphological)); + } else if (!start) { + List left = findStems(input.substring(0, at), depth + 1, + budget, cache, morphological); + if (left.isEmpty() && "-".equals(separator) && dictionary.hyphenMovingRule()) { + left = hyphenatedFirstPart(input.substring(0, at), morphological); + } + if (!left.isEmpty()) { + final List right = findStems(input.substring(after), depth + 1, + budget, cache, morphological); + if (!right.isEmpty()) { + analyses.addBroken(input.substring(0, at), left, input.substring(after), right); + } + } + } + } + } + } + final List result = List.copyOf(analyses.values); + cache.put(input, result); + return result; + } + + /** + * Finds the readings of the part of a Hungarian word before a hyphen, which the + * reference implementation accepts as a listed word ending in the hyphen or as a + * compound under the moving rule. + * + * @param text The part before the hyphen. + * @param morphological Whether results contain morphological fields. + * @return Recognized stems or analyses, or an empty list. + */ + private List hyphenatedFirstPart(String text, boolean morphological) { + final Results analyses = new Results(morphological); + final String hyphenated = text + "-"; + final boolean allCaps = HunspellDictionary.caseType(text) == HunspellDictionary.CaseType.ALLCAP; + for (final String variant : variants(hyphenated)) { + analyze(variant, new Analysis(hyphenated, variant, analyses, allCaps)); + } + if (analyses.isEmpty() && !analyses.forbidden && dictionary.compoundsDeclared()) { + analyses.hyphenatedFirstPart = true; + for (final String variant : variants(text)) { + decompose(variant, text, analyses); + } + } + return List.copyOf(analyses.values); } /** * Collects the case variants to analyze: the surface form first, then its lowercase - * form when the two differ. Ordering matters because the first analysis found wins - * in {@link #stem(CharSequence)}. + * form when the two differ. A capitalized word with a further inner capital, such as a + * mixed-case word at the start of a sentence, is also tried with a lowercase initial. + * An all-uppercase word containing an apostrophe is also tried with the part after + * the apostrophe capitalized, for the elided articles of Catalan, French, and Italian. + * Ordering matters because the first analysis found wins in {@link #stem(CharSequence)}. * * @param surface The surface form. * @return The variants in analysis order. Never {@code null} or empty. */ private List variants(String surface) { - final String lowered = StringUtil.toLowerCase(surface); - return lowered.equals(surface) ? List.of(surface) : List.of(surface, lowered); + final Set variants = new LinkedHashSet<>(); + variants.add(surface); + boolean upper = false; + boolean lowerAfterFirst = true; + boolean allUpper = true; + boolean firstUpper = false; + int uppers = 0; + int letters = 0; + for (int i = 0; i < surface.length();) { + final int point = surface.codePointAt(i); + if (Character.isLowerCase(point)) { + allUpper = false; + letters++; + } else if (Character.isUpperCase(point) || Character.isTitleCase(point)) { + if (letters > 0) { + lowerAfterFirst = false; + } + firstUpper |= letters == 0; + upper = true; + uppers++; + letters++; + } + i += Character.charCount(point); + } + if (upper && (allUpper || lowerAfterFirst)) { + final String lowered = dictionary.lowerCase(surface); + final int apostrophe = lowered.indexOf('\''); + if (allUpper && apostrophe > 0 && apostrophe < lowered.length() - 1) { + final String elided = lowered.substring(0, apostrophe + 1) + + initialUpper(lowered.substring(apostrophe + 1)); + variants.add(elided); + variants.add(initialUpper(elided)); + } + variants.add(lowered); + if (allUpper) { + variants.add(initialUpper(lowered)); + if (surface.charAt(0) == 'İ') { + // the reference keeps a dotted capital I when it capitalizes an all-uppercase + // word, so an entry such as İzmir is found outside the Turkic languages too + variants.add("İ" + lowered.substring(Character.charCount(lowered.codePointAt(0)))); + } + if (dictionary.checkSharps()) { + addSharpVariants(lowered, 0, variants); + } + } + } else if (firstUpper && uppers > 1 && !allUpper) { + variants.add(dictionary.lowerCaseInitial(surface)); + } + return List.copyOf(variants); + } + + /** + * Converts the initial code point to uppercase with the dictionary's case mapping. + * + * @param word The nonempty word. + * @return The capitalized form. + */ + private String initialUpper(String word) { + return dictionary.upperCaseInitial(word); + } + + /** + * Collects sharp-s alternatives with a fixed search limit. + * + * @param word The lowercase candidate. + * @param from The next character position to search. + * @param variants The resulting forms. + */ + private void addSharpVariants(String word, int from, Set variants) { + if (variants.size() >= MAX_CASE_VARIANTS) { + return; + } + for (int at = word.indexOf("ss", from); at >= 0; at = word.indexOf("ss", at + 1)) { + final String changed = word.substring(0, at) + "ß" + word.substring(at + 2); + variants.add(changed); + variants.add(initialUpper(changed)); + addSharpVariants(changed, at + 1, variants); + if (variants.size() >= MAX_CASE_VARIANTS) { + return; + } + } + } + + /** One case-variant analysis with request-local result storage. */ + private final class Analysis { + private final String surface; + private final String variant; + private final Results stems; + /** + * Whether the input is all uppercase, in which case the reference implementation + * also matches the hidden capitalized forms of mixed-case entries. + */ + private final boolean allCaps; + + /** + * Creates an analysis context. + * + * @param surface The input after conversion. + * @param variant The current case variant. + * @param stems The destination for recognized stems. + * @param allCaps Whether the input is all uppercase. + */ + private Analysis(String surface, String variant, Results stems, boolean allCaps) { + this.surface = surface; + this.variant = variant; + this.stems = stems; + this.allCaps = allCaps; + } + + /** + * Looks up a spelling, including hidden capitalized forms for all-uppercase input. + * + * @param word The spelling to look up. + * @return The flag sets, or {@code null} when absent. + */ + private List lookup(String word) { + return dictionary.lookup(word, allCaps); + } + + /** + * Adds morphology-aware stems after case and warning checks. + * + * @param root The entry spelling. + * @param flags The selected flags. + * @param affixes The rules in application order. + */ + private void add(String root, int[] flags, Affix... affixes) { + if (dictionary.acceptsCase(flags, surface, variant, affixes)) { + stems.addAll(stems.morphological + ? dictionary.morphologicalAnalyses(root, flags, false, affixes) + : dictionary.morphologicalStems(root, flags, affixes)); + } + } + + /** + * Records that an affix analysis reached a forbidden entry. + * + * @param flagSets The stem's flag sets. + * @param flag The removed affix's flag. + */ + private void noteForbidden(List flagSets, int flag) { + if (dictionary.forbidsAffixed(flagSets, flag)) { + stems.forbidden = true; + } + } } /** @@ -145,14 +542,17 @@ private List variants(String surface) { * @param word The case variant to analyze. * @param analyses The mutable, insertion-ordered set collecting the stems found. */ - private void analyze(String word, Set analyses) { - final List entries = dictionary.lookup(word); + private void analyze(String word, Analysis analyses) { + final List entries = analyses.lookup(word); if (entries != null) { - if (dictionary.anyForbidden(entries)) { + if (dictionary.firstForbidden(entries)) { + analyses.stems.forbidden = true; return; } - if (dictionary.validStandalone(entries)) { - analyses.add(word); + for (int[] flags : entries) { + if (dictionary.validStandalone(List.of(flags))) { + analyses.add(word, flags); + } } } for (final Affix suffix : dictionary.suffixesEndingWith( @@ -171,43 +571,265 @@ private void analyze(String word, Set analyses) { } /** - * Decomposes a word into listed compound parts when the affix analysis found - * nothing: the first part must be admitted to open a compound, every further part - * to continue or close one, each at least the declared minimum length and counted - * against the declared maximum. A part stands on its own entry or on an entry plus - * one affix, the way published dictionaries position their linking forms through - * zero or dash suffixes. The stems of the parts of every successful splitting are - * reported left to right, so the head-most material comes last. A word the - * dictionary lists as forbidden never decomposes; that is how one specific - * ill-formed compound is blocked while its parts stay productive. + * Searches compound components after standalone analysis fails. Entries and + * affixes must permit the selected component positions and junctions. Recognized + * spaced forms prevent concatenation. The candidate budget applies to compound + * decomposition; output follows component order. * * @param word The case variant to decompose. * @param surface The surface form the variant was derived from; character case at - * junctions is judged against it, so lowercasing a variant cannot - * sidestep a {@code CHECKCOMPOUNDCASE} declaration. + * junctions is checked using this input for {@code CHECKCOMPOUNDCASE}. * @param analyses The mutable, insertion-ordered set collecting the part stems. */ - private void decompose(String word, String surface, Set analyses) { + private void decompose(String word, String surface, Results analyses) { final List entries = dictionary.lookup(word); - if (entries != null && dictionary.anyForbidden(entries)) { + if (entries != null && dictionary.firstForbidden(entries)) { return; } - final int codePointCount = word.codePointCount(0, word.length()); - if (codePointCount < 2 * dictionary.compoundMin()) { + if (rejectsCompoundText(word)) { return; } - final int[] codePointOffsets = new int[codePointCount + 1]; - int offset = 0; - for (int i = 0; i < codePointCount; i++) { - codePointOffsets[i] = offset; - offset += Character.charCount(word.codePointAt(offset)); - } - codePointOffsets[codePointCount] = word.length(); // lowercasing may change the length in exceptional mappings, in which case the // offsets no longer align and the variant itself is the only usable case source final String caseSource = surface.length() == word.length() ? surface : word; - search(word, caseSource, codePointOffsets, 0, new ArrayList<>(), - new ArrayList<>(), analyses, new int[] {PART_CHECK_BUDGET}); + final int[] budget = {PART_CHECK_BUDGET}; + if (dictionary.positionalCompoundsDeclared()) { + searchSpelling(word, caseSource, analyses, budget, List.of()); + for (CompoundPattern pattern : dictionary.compoundPatterns()) { + if (pattern.replacement() == null || pattern.replacement().isEmpty()) { + continue; + } + for (int at = word.indexOf(pattern.replacement()); at >= 0 && budget[0] > 0; + at = word.indexOf(pattern.replacement(), at + 1)) { + final int end = at + pattern.replacement().length(); + final String inserted = pattern.end() + pattern.begin(); + searchSpelling(word.substring(0, at) + inserted + word.substring(end), + caseSource.substring(0, at) + inserted + caseSource.substring(end), + analyses, budget, List.of(new Junction(at + pattern.end().length(), pattern))); + } + } + if (dictionary.simplifiedTriple()) { + searchTriples(word, caseSource, 0, List.of(), analyses, budget); + } + } + for (HunspellCompoundRule rule : dictionary.compoundRules()) { + searchRule(word, caseSource, 0, rule, new ArrayList<>(), new ArrayList<>(), analyses, budget); + } + } + + /** + * Applies the text-level compound checks the reference implementation runs on the + * text every compound level splits: a {@code CHECKCOMPOUNDREP} replacement or a + * space inserted at any position must not produce a recognized non-compound form. + * + * @param text The complete input or the remainder a compound level splits. + * @return {@code true} if a check forbids splitting the text. + */ + private boolean rejectsCompoundText(String text) { + if (dictionary.rejectsCompoundReplacement(text, this::isNoncompoundForm)) { + return true; + } + for (int at = Character.charCount(text.codePointAt(0)); at < text.length(); + at += Character.charCount(text.codePointAt(at))) { + if (isNoncompoundForm(text.substring(0, at) + " " + text.substring(at))) { + return true; + } + } + return false; + } + + /** + * A required boundary after a compound spelling substitution. + * + * @param offset The boundary's UTF-16 offset. + * @param pattern The pattern permitting this boundary, or null for a repeated letter. + */ + private record Junction(int offset, CompoundPattern pattern) { } + + /** + * Restores simplified triple letters at independently checked boundaries. + * + * @param word The current text. + * @param surface The aligned case source. + * @param from The next position eligible for restoration. + * @param junctions The required boundaries accumulated so far. + * @param analyses The destination for stems. + * @param budget The remaining search attempts. + */ + private void searchTriples(String word, String surface, int from, List junctions, + Results analyses, int[] budget) { + if (junctions.size() >= MAX_COMPOUND_PARTS - 1) { + return; + } + for (int at = from; at < word.length() && budget[0] > 0;) { + final int point = word.codePointAt(at); + final int width = Character.charCount(point); + final int next = at + width; + if (next < word.length() && word.codePointAt(next) == point + && (at == 0 || word.codePointBefore(at) != point) + && (next + width == word.length() || word.codePointAt(next + width) != point)) { + final String inserted = new String(Character.toChars(point)); + final String expanded = new StringBuilder(word).insert(next, inserted).toString(); + final String expandedCase = new StringBuilder(surface).insert(next, inserted).toString(); + for (int boundary : new int[] {next, next + width}) { + if (budget[0]-- <= 0) { + return; + } + final List required = new ArrayList<>(junctions); + required.add(new Junction(boundary, null)); + searchSpelling(expanded, expandedCase, analyses, budget, required); + searchTriples(expanded, expandedCase, next + 2 * width, required, analyses, budget); + } + } + at = next; + } + } + + /** + * Searches one spelling, optionally requiring a restored compound junction. + * + * @param word The spelling with any compound substitution expanded. + * @param surface The aligned case source. + * @param analyses The destination for stems. + * @param budget The remaining candidate checks across forms. + * @param junctions Required boundaries for spelling substitutions. + */ + private void searchSpelling(String word, String surface, Results analyses, int[] budget, + List junctions) { + final int count = word.codePointCount(0, word.length()); + if (count < 2 * dictionary.compoundMin()) { + return; + } + for (Junction junction : junctions) { + if (junction.offset() <= 0 || junction.offset() >= word.length()) { + return; + } + } + final int[] offsets = new int[count + 1]; + int offset = 0; + for (int i = 0; i < count; i++) { + offsets[i] = offset; + offset += Character.charCount(word.codePointAt(offset)); + } + offsets[count] = word.length(); + search(word, surface, offsets, 0, new ArrayList<>(), analyses, budget, junctions, + new byte[count + 1]); + } + + /** + * One selected compound-part reading. + * + * @param surface The part as written. + * @param root The entry spelling. + * @param flags The selected homonym's flags. + * @param affixes The rules in application order. + * @param stems The morphology-aware stems. + */ + private record CompoundPart(String surface, String root, int[] flags, + List affixes, List stems) { } + + /** + * Searches compound-rule paths without combining flags from different homonyms. + * + * @param word The input case variant. + * @param surface The input used for case restrictions. + * @param from The current UTF-16 offset. + * @param rule The compound pattern. + * @param parts The selected readings. + * @param flags The flags corresponding to the readings. + * @param analyses The destination for stems. + * @param budget The remaining candidate checks. + */ + private void searchRule(String word, String surface, int from, HunspellCompoundRule rule, + List parts, List flags, Results analyses, int[] budget) { + if (parts.size() >= MAX_COMPOUND_PARTS + || word.codePointCount(from, word.length()) < dictionary.compoundMin()) { + return; + } + int end = word.offsetByCodePoints(from, dictionary.compoundMin()); + while (end <= word.length() && budget[0] > 0) { + final boolean last = end == word.length(); + if (!(from == 0 && last)) { + budget[0]--; + final String part = word.substring(from, end); + final String caseSource = surface.substring(from, end); + for (CompoundPart candidate : ruleParts(part, caseSource, last)) { + if (last && dictionary.checkCompoundDup() && !parts.isEmpty() + && parts.get(parts.size() - 1).root().equals(candidate.root())) { + continue; + } + parts.add(candidate); + flags.add(candidate.flags()); + if (rule.matches(flags, last)) { + if (last) { + analyses.addCompound(parts); + } else { + searchRule(word, surface, end, rule, parts, flags, analyses, budget); + } + } + parts.remove(parts.size() - 1); + flags.remove(flags.size() - 1); + } + } + if (last) { + break; + } + end += Character.charCount(word.codePointAt(end)); + } + } + + /** + * Finds selected dictionary entries for a compound-rule part. + * + * @param part The part's case variant. + * @param surface The part's supplied case. + * @param last Whether suffix removal is permitted at this position. + * @return The permitted readings. + */ + private List ruleParts(String part, String surface, boolean last) { + final List result = new ArrayList<>(); + addRulePart(part, part, surface, last, result); + if (last) { + for (Affix suffix : dictionary.suffixesEndingWith(part.codePointBefore(part.length()))) { + final String root = removeSuffixAllowingIdentity(part, suffix); + if (root != null) { + addRulePart(part, root, surface, true, result, suffix); + } + } + for (Affix suffix : dictionary.suffixesWithoutMaterial()) { + final String root = removeSuffixAllowingIdentity(part, suffix); + if (root != null) { + addRulePart(part, root, surface, true, result, suffix); + } + } + } + return result; + } + + /** + * Adds homonyms permitted by their blocking and affix flags. + * + * @param part The component text. + * @param root The restored entry spelling. + * @param surface The supplied part spelling. + * @param last Whether the part closes the compound. + * @param result The candidate destination. + * @param affixes The applied suffixes. + */ + private void addRulePart(String part, String root, String surface, boolean last, + List result, Affix... affixes) { + final List entries = dictionary.lookup(root); + if (entries == null || dictionary.firstForbidden(entries)) { + return; + } + for (int[] entry : entries) { + if (dictionary.acceptsRulePart(entry, last, affixes) + && dictionary.acceptsCase(entry, surface, part, affixes) + && (affixes.length == 0 || HunspellDictionary.hasFlag(List.of(entry), affixes[0].flag()))) { + result.add(new CompoundPart(part, root, entry, List.of(affixes), + dictionary.morphologicalStems(root, entry, affixes))); + } + } } /** @@ -223,80 +845,121 @@ private void decompose(String word, String surface, Set analyses) { * form when its offsets align with the variant. * @param codePointOffsets UTF-16 offsets for each code point boundary. * @param fromPoint The code point index where the next part starts. - * @param surfaces The surface strings of the parts taken so far. - * @param stems The licensed stems of the parts taken so far, one list per part. + * @param parts The selected part readings. * @param analyses The mutable, insertion-ordered set collecting the part stems. * @param budget The remaining part-licensing attempts, counted down in place. + * @param junctions The required boundaries for substitutions. + * @param remainderChecks The cached outcome of {@link #rejectsCompoundText(String)} + * per code point index: {@code 0} unknown, {@code 1} allowed, + * {@code 2} rejected. */ private void search(String word, String caseSource, int[] codePointOffsets, - int fromPoint, List surfaces, List> stems, - Set analyses, int[] budget) { + int fromPoint, List parts, Results analyses, int[] budget, + List junctions, byte[] remainderChecks) { final int from = codePointOffsets[fromPoint]; - if (from > 0 && violatesBoundaryChecks(word, caseSource, from)) { + Junction current = null; + for (Junction junction : junctions) { + if (junction.offset() == from) { + current = junction; + } + } + if (parts.size() >= MAX_COMPOUND_PARTS + || (from > 0 && violatesBoundaryChecks(word, caseSource, from, + current != null && current.pattern() == null))) { return; } final int min = dictionary.compoundMin(); - final int max = dictionary.compoundWordMax(); final boolean first = from == 0; final int remaining = codePointOffsets.length - 1 - fromPoint; - // every split leaving room for a further part; a first-position part must also - // leave the closing part, so the whole word is never one part - if (remaining >= 2 * min && (max == 0 || surfaces.size() + 2 <= max)) { - final int lastEndPoint = codePointOffsets.length - 1 - min; - for (int endPoint = fromPoint + min; endPoint <= lastEndPoint; endPoint++) { - if (budget[0] <= 0) { - return; - } - budget[0]--; - final int end = codePointOffsets[endPoint]; - final String part = word.substring(from, end); - if (duplicatesNeighbor(part, surfaces)) { + if (remaining < min) { + return; + } + // this call splits the remainder into further parts, which the reference + // implementation subjects to the same text checks as the complete input + if (!first) { + if (remainderChecks[fromPoint] == 0) { + remainderChecks[fromPoint] = (byte) (rejectsCompoundText(word.substring(from)) ? 2 : 1); + } + if (remainderChecks[fromPoint] == 2) { + return; + } + } + for (int endPoint = fromPoint + min; endPoint < codePointOffsets.length; endPoint++) { + if (budget[0] <= 0) { + return; + } + final int end = codePointOffsets[endPoint]; + final boolean last = end == word.length(); + boolean spansRequiredBoundary = false; + for (Junction junction : junctions) { + spansRequiredBoundary |= from < junction.offset() && end > junction.offset(); + } + if ((first && last) || spansRequiredBoundary + || (!last && codePointOffsets.length - 1 - endPoint < min)) { + continue; + } + budget[0]--; + final String part = word.substring(from, end); + final CompoundPosition position = first ? CompoundPosition.BEGIN + : last ? CompoundPosition.END : CompoundPosition.MIDDLE; + for (CompoundPart candidate : partReadings(part, caseSource.substring(from, end), + position, first, last, analyses.hyphenatedFirstPart)) { + if (!parts.isEmpty() && rejectsJunction(parts.get(parts.size() - 1), candidate, + current == null ? null : current.pattern(), last)) { continue; } - final List partStems = partStems(part, - first ? CompoundPosition.BEGIN : CompoundPosition.MIDDLE, first, false); - if (partStems.isEmpty()) { - continue; + parts.add(candidate); + if (last) { + int units = 0; + int syllables = 0; + for (CompoundPart selected : parts) { + units += dictionary.compoundUnits(selected.flags(), selected.affixes()); + syllables += dictionary.compoundSyllables(selected.surface(), selected.flags(), + selected.affixes(), selected == candidate); + } + if ((analyses.hyphenatedFirstPart || dictionary.compoundSizeAllowed(syllables, units)) + && dictionary.compoundCaseAllowed(candidate.flags(), caseSource)) { + analyses.addCompound(parts); + } + } else { + search(word, caseSource, codePointOffsets, endPoint, parts, analyses, + budget, junctions, remainderChecks); } - surfaces.add(part); - stems.add(partStems); - search(word, caseSource, codePointOffsets, endPoint, surfaces, stems, - analyses, budget); - surfaces.remove(surfaces.size() - 1); - stems.remove(stems.size() - 1); + parts.remove(parts.size() - 1); } } - // the closing part takes the whole remainder; a compound has at least two parts - if (first || remaining < min - || (max > 0 && surfaces.size() + 1 > max) || budget[0] <= 0) { - return; - } - budget[0]--; - final String part = word.substring(from); - if (duplicatesNeighbor(part, surfaces)) { - return; - } - final List partStems = partStems(part, CompoundPosition.END, false, true); - if (partStems.isEmpty()) { - return; - } - for (final List earlier : stems) { - analyses.addAll(earlier); - } - analyses.addAll(partStems); } /** - * Applies the {@code CHECKCOMPOUNDDUP} declaration: a part must not repeat the - * part directly before it. + * Applies the junction declarations. {@code CHECKCOMPOUNDDUP} forbids the closing + * part from repeating the part before it; the reference implementation compares the + * two parts it joins at each level, so an earlier repetition is not checked. A + * junction restored from a pattern replacement must satisfy that pattern's flag + * conditions and is exempt from the other patterns; any other junction is forbidden + * when some pattern matches it. * - * @param part The candidate part. - * @param surfaces The surface strings of the parts taken so far. - * @return {@code true} if the declaration forbids this part here. + * @param left The preceding part. + * @param right The candidate part. + * @param allowed The rule allowing this substituted junction, or {@code null}. + * @param last Whether the candidate closes the compound. + * @return {@code true} if a declaration forbids this part here. */ - private boolean duplicatesNeighbor(String part, List surfaces) { - return dictionary.checkCompoundDup() && !surfaces.isEmpty() - && part.equals(surfaces.get(surfaces.size() - 1)); + private boolean rejectsJunction(CompoundPart left, CompoundPart right, CompoundPattern allowed, + boolean last) { + if (last && dictionary.checkCompoundDup() && left.root().equals(right.root())) { + return true; + } + if (allowed != null) { + return !allowed.matches(left.surface(), left.flags(), right.surface(), + right.flags(), left.affixes(), right.affixes()); + } + for (CompoundPattern pattern : dictionary.compoundPatterns()) { + if (pattern.matches(left.surface(), left.flags(), right.surface(), + right.flags(), left.affixes(), right.affixes())) { + return true; + } + } + return false; } /** @@ -308,9 +971,11 @@ private boolean duplicatesNeighbor(String part, List surfaces) { * @param word The case variant under decomposition. * @param caseSource The character-case source for the uppercase judgment. * @param from The index the junction sits before; greater than zero. + * @param allowTriple Whether this junction restores a simplified repeated letter. * @return {@code true} if a declaration forbids this junction. */ - private boolean violatesBoundaryChecks(String word, String caseSource, int from) { + private boolean violatesBoundaryChecks(String word, String caseSource, int from, + boolean allowTriple) { final int before = word.codePointBefore(from); final int after = word.codePointAt(from); if (dictionary.checkCompoundCase() @@ -318,7 +983,7 @@ private boolean violatesBoundaryChecks(String word, String caseSource, int from) || Character.isUpperCase(caseSource.codePointAt(from)))) { return true; } - if (dictionary.checkCompoundTriple() && before == after) { + if (!allowTriple && dictionary.checkCompoundTriple() && before == after) { final int beforeStart = from - Character.charCount(before); final int afterEnd = from + Character.charCount(after); if ((beforeStart > 0 && word.codePointBefore(beforeStart) == after) @@ -330,64 +995,152 @@ private boolean violatesBoundaryChecks(String word, String caseSource, int from) } /** - * Collects the listed stems that admit one part at its compound position: the part - * as its own entry, or an entry plus one suffix or one prefix whose removal leaves - * a listed stem, zero-material rules included, because published dictionaries - * position their linking forms through zero and dash suffixes. An affix at a - * compound-internal boundary must carry the permit flag, a suffix facing the next - * part or a prefix facing the previous one. A part not found as written is also - * tried with its first letter uppercased, the way nouns listed capitalized appear - * lowercase inside a compound. + * Collects direct and affixed readings for a compound component. Affixes at + * internal boundaries require the permit flag. If no reading is found, a + * capitalized dictionary entry is also considered. * * @param part The part's surface text. - * @param position The part's place in the compound. + * @param surface The supplied case. + * @param position The component position. * @param first Whether the part opens the word. * @param last Whether the part closes the word. - * @return The stems admitting the part, in discovery order. Never {@code null}. + * @param movingRule Whether the Hungarian moving rule relaxes the opening parts. + * @return The permitted readings in discovery order. */ - private List partStems(String part, CompoundPosition position, - boolean first, boolean last) { - final Set stems = new LinkedHashSet<>(); - collectPartStems(part, position, first, last, stems); - if (stems.isEmpty() && !part.isEmpty()) { + private List partReadings(String part, String surface, CompoundPosition position, + boolean first, boolean last, boolean movingRule) { + final List readings = new ArrayList<>(); + final List listed = last || movingRule ? null : dictionary.lookup(part); + if (listed != null && dictionary.forbidsCompoundStart(listed)) { + // a listed spelling barred from compounding is barred in its affixed readings too + return readings; + } + collectPartReadings(part, surface, position, first, last, movingRule, readings); + if (readings.isEmpty() && !part.isEmpty()) { final int initial = part.codePointAt(0); final int upper = Character.toUpperCase(initial); if (upper != initial) { - collectPartStems(new StringBuilder().appendCodePoint(upper) + collectPartReadings(new StringBuilder().appendCodePoint(upper) .append(part, Character.charCount(initial), part.length()).toString(), - position, first, last, stems); + surface, position, first, last, movingRule, readings); } } - return List.copyOf(stems); + return readings; } /** - * Collects the stems admitting one spelling of a part, bare and through one affix. + * Collects direct entries and permitted affix combinations for a component. * * @param part The part spelling to look up. + * @param surface The supplied case. * @param position The part's place in the compound. * @param first Whether the part opens the word. * @param last Whether the part closes the word. - * @param stems The mutable, insertion-ordered set collecting the stems. + * @param movingRule Whether an opening entry may also qualify through the hardwired + * Hungarian flags. + * @param readings The destination for selected readings. */ - private void collectPartStems(String part, CompoundPosition position, - boolean first, boolean last, Set stems) { + private void collectPartReadings(String part, String surface, CompoundPosition position, + boolean first, boolean last, boolean movingRule, List readings) { final List entries = dictionary.lookup(part); - if (entries != null && dictionary.mayStand(entries, position)) { - stems.add(part); + if (entries != null) { + for (int[] flags : entries) { + if ((dictionary.mayStand(List.of(flags), position) + || (movingRule && !last && dictionary.opensHyphenatedCompound(flags))) + && dictionary.acceptsCase(flags, surface, part)) { + readings.add(new CompoundPart(part, part, flags, List.of(), + dictionary.morphologicalStems(part, flags))); + } + } } for (final Affix suffix : dictionary.suffixesEndingWith( part.codePointBefore(part.length()))) { - collectAffixedPartStem(part, suffix, true, position, last, stems); + collectAffixedPartReading(part, surface, suffix, position, first, last, readings); } for (final Affix suffix : dictionary.suffixesWithoutMaterial()) { - collectAffixedPartStem(part, suffix, true, position, last, stems); + collectAffixedPartReading(part, surface, suffix, position, first, last, readings); } for (final Affix prefix : dictionary.prefixesStartingWith(part.codePointAt(0))) { - collectAffixedPartStem(part, prefix, false, position, first, stems); + collectAffixedPartReading(part, surface, prefix, position, first, last, readings); + collectCrossPartReadings(part, surface, prefix, position, first, last, readings); } for (final Affix prefix : dictionary.prefixesWithoutMaterial()) { - collectAffixedPartStem(part, prefix, false, position, first, stems); + collectAffixedPartReading(part, surface, prefix, position, first, last, readings); + collectCrossPartReadings(part, surface, prefix, position, first, last, readings); + } + } + + /** + * Searches suffix combinations after removing a compound component's prefix. + * + * @param part The component text. + * @param surface The supplied case. + * @param prefix The prefix rule. + * @param position The component position. + * @param first Whether this is the opening component. + * @param last Whether this is the closing component. + * @param readings The destination for accepted readings. + */ + private void collectCrossPartReadings(String part, String surface, Affix prefix, + CompoundPosition position, boolean first, boolean last, List readings) { + if (!prefix.crossProduct() || dictionary.forbidsInCompound(prefix) + || (!first && !dictionary.permitsInside(prefix))) { + return; + } + final String intermediate = removePrefixAllowingIdentity(part, prefix); + if (intermediate == null) { + return; + } + for (Affix suffix : dictionary.suffixesEndingWith(intermediate.codePointBefore(intermediate.length()))) { + collectCrossPart(part, surface, intermediate, prefix, suffix, position, last, readings); + } + for (Affix suffix : dictionary.suffixesWithoutMaterial()) { + collectCrossPart(part, surface, intermediate, prefix, suffix, position, last, readings); + } + } + + /** + * Validates a compound component with a prefix and suffix. + * + * @param part The component text. + * @param surface The supplied case. + * @param intermediate The text after prefix removal. + * @param prefix The prefix rule. + * @param suffix The suffix rule. + * @param position The component position. + * @param last Whether this is the closing component. + * @param readings The destination for accepted readings. + */ + private void collectCrossPart(String part, String surface, String intermediate, + Affix prefix, Affix suffix, CompoundPosition position, boolean last, + List readings) { + if (!suffix.crossProduct() || dictionary.forbidsInCompound(suffix) + || (!last && !dictionary.permitsInside(suffix)) + || dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(suffix) + || (dictionary.needsFurtherAffix(prefix) && dictionary.needsFurtherAffix(suffix))) { + return; + } + final String root = removeSuffixAllowingIdentity(intermediate, suffix); + if (root == null) { + return; + } + final List entries = dictionary.lookup(root); + if (entries != null) { + for (int[] flags : entries) { + if (dictionary.supportsCompoundCrossProduct(flags, position, prefix, suffix) + && dictionary.acceptsCase(flags, surface, part, prefix, suffix)) { + readings.add(new CompoundPart(part, root, flags, List.of(prefix, suffix), + dictionary.morphologicalStems(root, flags, prefix, suffix))); + } + } + } + if (last && dictionary.compoundMoreSuffixes() && !dictionary.complexPrefixes()) { + for (Affix inner : dictionary.suffixesEndingWith(root.codePointBefore(root.length()))) { + collectCrossDoublePart(part, surface, root, inner, suffix, prefix, position, readings); + } + for (Affix inner : dictionary.suffixesWithoutMaterial()) { + collectCrossDoublePart(part, surface, root, inner, suffix, prefix, position, readings); + } } } @@ -396,18 +1149,21 @@ private void collectPartStems(String part, CompoundPosition position, * it at the position. * * @param part The part spelling under analysis. + * @param surface The supplied case. * @param affix The rule to undo. - * @param suffix Whether the rule is a suffix rule. * @param position The part's place in the compound. - * @param atEdge Whether the part sits at the word end the rule faces, the closing part - * for a suffix rule and the opening part for a prefix rule; an affix - * facing another part instead needs the permit flag. - * @param stems The mutable, insertion-ordered set collecting the stems. + * @param first Whether this is the opening component. + * @param last Whether this is the closing component. + * @param readings The destination for selected readings. */ - private void collectAffixedPartStem(String part, Affix affix, boolean suffix, - CompoundPosition position, boolean atEdge, Set stems) { + private void collectAffixedPartReading(String part, String surface, Affix affix, + CompoundPosition position, boolean first, boolean last, List readings) { + final boolean suffix = affix.suffix(); + final boolean atEdge = suffix ? last : first; + // a compound-only suffix joins parts and never closes a compound on its own if (dictionary.circumfixOnly(affix) || dictionary.forbidsInCompound(affix) - || (!atEdge && !dictionary.permitsInside(affix))) { + || (!atEdge && !dictionary.permitsInside(affix)) + || (suffix && last && dictionary.compoundOnly(affix))) { return; } final String stem = removeAffixInCompound(part, affix, suffix); @@ -415,12 +1171,174 @@ private void collectAffixedPartStem(String part, Affix affix, boolean suffix, return; } final List flagSets = dictionary.lookup(stem); - if (flagSets != null && dictionary.supportsPart(flagSets, affix.flag(), position, - dictionary.affixAdmits(affix, position))) { - stems.add(stem); + if (flagSets != null && !dictionary.needsFurtherAffix(affix)) { + for (int[] flags : flagSets) { + if (dictionary.supportsPart(List.of(flags), affix.flag(), position, + dictionary.affixAdmits(affix, position)) + && dictionary.acceptsCase(flags, surface, part, affix)) { + readings.add(new CompoundPart(part, stem, flags, List.of(affix), + dictionary.morphologicalStems(stem, flags, affix))); + } + } + } + if (dictionary.compoundMoreSuffixes() && suffix != dictionary.complexPrefixes() && atEdge) { + final List material = suffix + ? dictionary.suffixesEndingWith(stem.codePointBefore(stem.length())) + : dictionary.prefixesStartingWith(stem.codePointAt(0)); + final List zero = suffix + ? dictionary.suffixesWithoutMaterial() : dictionary.prefixesWithoutMaterial(); + for (Affix inner : material) { + collectDoublePart(part, surface, stem, inner, affix, position, last, readings); + } + for (Affix inner : zero) { + collectDoublePart(part, surface, stem, inner, affix, position, last, readings); + } } } + /** + * Finds a compound part with continuation-linked affixes. + * + * @param part The component text. + * @param surface The supplied case. + * @param intermediate The form after outer affix removal. + * @param inner The inner rule. + * @param outer The outer rule. + * @param position The compound position. + * @param last Whether this is the final part. + * @param readings The destination for readings. + */ + private void collectDoublePart(String part, String surface, String intermediate, + Affix inner, Affix outer, CompoundPosition position, boolean last, + List readings) { + if (!inner.allowsContinuation(outer.flag()) || dictionary.forbidsInCompound(inner)) { + return; + } + final String root = removeAffixInCompound(intermediate, inner, inner.suffix()); + if (root == null) { + return; + } + final List entries = dictionary.lookup(root); + if (entries != null && !dictionary.circumfixOnly(inner)) { + for (int[] flags : entries) { + if (dictionary.supportsPart(List.of(flags), inner.flag(), position, + dictionary.affixAdmits(inner, position) || dictionary.affixAdmits(outer, position)) + && dictionary.acceptsCase(flags, surface, part, inner, outer)) { + readings.add(new CompoundPart(part, root, flags, List.of(inner, outer), + dictionary.morphologicalStems(root, flags, inner, outer))); + } + } + } + if (!inner.suffix() && inner.crossProduct() && outer.crossProduct()) { + for (Affix suffix : dictionary.suffixesEndingWith(root.codePointBefore(root.length()))) { + collectCompoundSuffixAfterPrefixes(part, surface, root, inner, outer, suffix, + position, last, readings); + } + for (Affix suffix : dictionary.suffixesWithoutMaterial()) { + collectCompoundSuffixAfterPrefixes(part, surface, root, inner, outer, suffix, + position, last, readings); + } + } + } + + /** + * Checks a suffix after continuation-linked compound prefixes. + * + * @param part The component text. + * @param surface The supplied case. + * @param text The text after removing the prefixes. + * @param inner The inner prefix. + * @param outer The outer prefix. + * @param suffix The suffix rule. + * @param position The component position. + * @param last Whether this is the final component. + * @param readings The result destination. + */ + private void collectCompoundSuffixAfterPrefixes(String part, String surface, String text, + Affix inner, Affix outer, Affix suffix, CompoundPosition position, boolean last, + List readings) { + if (!last && !dictionary.permitsInside(suffix)) { + return; + } + final String root = removeSuffixAllowingIdentity(text, suffix); + if (root != null) { + addCompoundCrossDouble(part, surface, root, inner, outer, suffix, position, readings); + } + } + + /** + * Removes an inner compound suffix after a prefix and outer suffix. + * + * @param part The component text. + * @param surface The supplied case. + * @param text The text after removing the outer suffix and prefix. + * @param inner The inner suffix. + * @param outer The outer suffix. + * @param prefix The prefix rule. + * @param position The component position. + * @param readings The result destination. + */ + private void collectCrossDoublePart(String part, String surface, String text, Affix inner, + Affix outer, Affix prefix, CompoundPosition position, List readings) { + final String root = removeSuffixAllowingIdentity(text, inner); + if (root != null) { + addCompoundCrossDouble(part, surface, root, inner, outer, prefix, position, readings); + } + } + + /** + * Validates a continuation sequence combined with an opposite-end affix. + * + * @param part The component text. + * @param surface The supplied case. + * @param root The restored entry text. + * @param inner The inner rule in the continuation sequence. + * @param outer The outer rule in the continuation sequence. + * @param cross The opposite-end rule. + * @param position The component position. + * @param readings The result destination. + */ + private void addCompoundCrossDouble(String part, String surface, String root, Affix inner, + Affix outer, Affix cross, CompoundPosition position, List readings) { + if (!inner.allowsContinuation(outer.flag()) || !inner.crossProduct() || !cross.crossProduct() + || dictionary.forbidsInCompound(inner) || dictionary.forbidsInCompound(cross) + || dictionary.circumfixOnly(outer) + || dictionary.circumfixOnly(inner) != dictionary.circumfixOnly(cross)) { + return; + } + final Affix prefix = inner.suffix() ? cross : inner; + final Affix suffix = inner.suffix() ? inner : cross; + final Affix[] applied = inner.suffix() + ? new Affix[] {cross, inner, outer} : new Affix[] {inner, outer, cross}; + final List entries = dictionary.lookup(root); + if (entries != null) { + for (int[] flags : entries) { + if (dictionary.supportsCompoundCrossProduct(flags, position, prefix, suffix, outer) + && dictionary.acceptsCase(flags, surface, part, applied)) { + readings.add(new CompoundPart(part, root, flags, List.of(applied), + dictionary.morphologicalStems(root, flags, applied))); + } + } + } + } + + /** + * Checks a direct or affixed form without compound recursion. + * + * @param word The candidate spelling. + * @return Whether a non-compound reading exists. + */ + private boolean isNoncompoundForm(String word) { + if (word.isEmpty()) { + return false; + } + final Results stems = new Results(false); + for (String variant : variants(word)) { + analyze(variant, new Analysis(word, variant, stems, false)); + } + return !stems.isEmpty(); + } + /** * Undoes one affix rule on a compound part. Unlike the standalone removals, a rule * that neither adds nor removes material is undone here, to its own spelling with @@ -443,28 +1361,35 @@ private String removeAffixInCompound(String part, Affix affix, boolean suffix) { * the intermediate stem, adding dictionary-confirmed analyses. A rule that applies * only inside compounds or requires the matching circumfix member is not undone * because no prefix accompanies this path. A rule requiring a further affix produces - * no single-removal analysis. An identity rule also produces no single-removal - * analysis, but it can complete a two-suffix analysis through continuation classes. + * no single-removal analysis. A rule that adds and strips no material is undone like + * any other, which recognizes a virtual stem it completes and reports its fields. * * @param word The case variant under analysis. * @param suffix The suffix rule to undo. * @param analyses The mutable, insertion-ordered set collecting the stems found. */ - private void undoSuffix(String word, Affix suffix, Set analyses) { + private void undoSuffix(String word, Affix suffix, Analysis analyses) { if (dictionary.compoundOnly(suffix) || dictionary.circumfixOnly(suffix)) { return; } - final boolean identity = isIdentityRule(suffix); final String stem = removeSuffixAllowingIdentity(word, suffix); if (stem == null) { return; } - if (!identity && !dictionary.needsFurtherAffix(suffix)) { - final List flagSets = dictionary.lookup(stem); - if (flagSets != null && dictionary.supports(flagSets, suffix.flag())) { - analyses.add(stem); + if (!dictionary.needsFurtherAffix(suffix)) { + final List flagSets = analyses.lookup(stem); + if (flagSets != null) { + analyses.noteForbidden(flagSets, suffix.flag()); + for (int[] flags : flagSets) { + if (dictionary.supports(List.of(flags), suffix.flag())) { + analyses.add(stem, flags, suffix); + } + } } } + if (dictionary.complexPrefixes()) { + return; + } for (final Affix inner : dictionary.suffixesEndingWith( stem.codePointBefore(stem.length()))) { undoInnerSuffix(stem, suffix, inner, analyses); @@ -485,7 +1410,7 @@ private void undoSuffix(String word, Affix suffix, Set analyses) { * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void undoInnerSuffix(String stem, Affix outer, Affix inner, - Set analyses) { + Analysis analyses) { if (!inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner) || dictionary.circumfixOnly(inner)) { return; @@ -494,9 +1419,13 @@ private void undoInnerSuffix(String stem, Affix outer, Affix inner, if (doubleStem == null) { return; } - final List innerFlags = dictionary.lookup(doubleStem); - if (innerFlags != null && dictionary.supports(innerFlags, inner.flag())) { - analyses.add(doubleStem); + final List innerFlags = analyses.lookup(doubleStem); + if (innerFlags != null) { + for (int[] flags : innerFlags) { + if (dictionary.supports(List.of(flags), inner.flag())) { + analyses.add(doubleStem, flags, inner, outer); + } + } } } @@ -505,27 +1434,38 @@ private void undoInnerSuffix(String stem, Affix outer, Affix inner, * intermediate stem, adding dictionary-confirmed analyses. A rule that * applies only inside compounds is not undone at all. A rule marked as needing a * further affix or the matching circumfix member produces no single-removal analysis. - * An identity rule also produces no single-removal analysis. A valid cross-product - * suffix can combine with either kind of rule. + * A rule that adds and strips no material is undone like any other. A valid + * cross-product suffix can combine with either kind of rule. * * @param word The case variant under analysis. * @param prefix The prefix rule to undo. * @param analyses The mutable, insertion-ordered set collecting the stems found. */ - private void undoPrefix(String word, Affix prefix, Set analyses) { + private void undoPrefix(String word, Affix prefix, Analysis analyses) { if (dictionary.compoundOnly(prefix)) { return; } - final boolean identity = isIdentityRule(prefix); final String stem = removePrefixAllowingIdentity(word, prefix); if (stem == null) { return; } - if (!identity && !dictionary.needsFurtherAffix(prefix) - && !dictionary.circumfixOnly(prefix)) { - final List flagSets = dictionary.lookup(stem); - if (flagSets != null && dictionary.supports(flagSets, prefix.flag())) { - analyses.add(stem); + if (!dictionary.needsFurtherAffix(prefix) && !dictionary.circumfixOnly(prefix)) { + final List flagSets = analyses.lookup(stem); + if (flagSets != null) { + analyses.noteForbidden(flagSets, prefix.flag()); + for (int[] flags : flagSets) { + if (dictionary.supports(List.of(flags), prefix.flag())) { + analyses.add(stem, flags, prefix); + } + } + } + } + if (dictionary.complexPrefixes()) { + for (Affix inner : dictionary.prefixesStartingWith(stem.codePointAt(0))) { + undoInnerPrefix(stem, prefix, inner, analyses); + } + for (Affix inner : dictionary.prefixesWithoutMaterial()) { + undoInnerPrefix(stem, prefix, inner, analyses); } } if (!prefix.crossProduct()) { @@ -552,7 +1492,7 @@ private void undoPrefix(String word, Affix prefix, Set analyses) { * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix, - Set analyses) { + Analysis analyses) { if (!suffix.crossProduct() || dictionary.compoundOnly(suffix) || dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(suffix)) { return; @@ -563,11 +1503,20 @@ private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix, } // One member can satisfy the other member's needs-further-affix marker. Both rule // flags must occur in one homonym's flag set. - final List both = dictionary.lookup(doubleStem); - if (both != null && dictionary.supportsCrossProduct(both, prefix, suffix) - && !(dictionary.needsFurtherAffix(prefix) - && dictionary.needsFurtherAffix(suffix))) { - analyses.add(doubleStem); + final List both = analyses.lookup(doubleStem); + if (both != null && !(dictionary.needsFurtherAffix(prefix) + && dictionary.needsFurtherAffix(suffix))) { + for (int[] flags : both) { + if (dictionary.licensesCrossProduct(flags, prefix, suffix)) { + analyses.noteForbidden(List.of(flags), suffix.flag()); + } + if (dictionary.supportsCrossProduct(List.of(flags), prefix, suffix)) { + analyses.add(doubleStem, flags, prefix, suffix); + } + } + } + if (dictionary.complexPrefixes()) { + return; } for (final Affix inner : dictionary.suffixesEndingWith( doubleStem.codePointBefore(doubleStem.length()))) { @@ -590,7 +1539,7 @@ private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix, * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void undoCrossProductInnerSuffix(String stem, Affix prefix, Affix outer, - Affix inner, Set analyses) { + Affix inner, Analysis analyses) { if (!inner.crossProduct() || !inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner) || dictionary.circumfixOnly(outer) @@ -601,9 +1550,76 @@ private void undoCrossProductInnerSuffix(String stem, Affix prefix, Affix outer, if (root == null) { return; } - final List flagSets = dictionary.lookup(root); - if (flagSets != null && dictionary.supportsCrossProduct(flagSets, prefix, inner)) { - analyses.add(root); + final List flagSets = analyses.lookup(root); + if (flagSets != null) { + for (int[] flags : flagSets) { + if (dictionary.supportsCrossProduct(List.of(flags), prefix, inner)) { + analyses.add(root, flags, prefix, inner, outer); + } + } + } + } + + /** + * Removes a continuation-linked inner prefix in COMPLEXPREFIXES mode. + * + * @param stem The form after the outer prefix was removed. + * @param outer The outer prefix. + * @param inner The inner prefix. + * @param analyses The result context. + */ + private void undoInnerPrefix(String stem, Affix outer, Affix inner, Analysis analyses) { + if (!inner.allowsContinuation(outer.flag()) || dictionary.compoundOnly(inner) + || dictionary.circumfixOnly(outer)) { + return; + } + final String root = removePrefixAllowingIdentity(stem, inner); + if (root == null) { + return; + } + final List entries = analyses.lookup(root); + if (entries != null && !dictionary.circumfixOnly(inner)) { + for (int[] flags : entries) { + if (dictionary.supports(List.of(flags), inner.flag())) { + analyses.add(root, flags, inner, outer); + } + } + } + if (inner.crossProduct() && outer.crossProduct()) { + for (Affix suffix : dictionary.suffixesEndingWith(root.codePointBefore(root.length()))) { + undoDoublePrefixSuffix(root, inner, outer, suffix, analyses); + } + for (Affix suffix : dictionary.suffixesWithoutMaterial()) { + undoDoublePrefixSuffix(root, inner, outer, suffix, analyses); + } + } + } + + /** + * Removes the suffix following a continuation-linked prefix combination. + * + * @param word The form after prefix removal. + * @param inner The inner prefix. + * @param outer The outer prefix. + * @param suffix The candidate suffix. + * @param analyses The result context. + */ + private void undoDoublePrefixSuffix(String word, Affix inner, Affix outer, + Affix suffix, Analysis analyses) { + if (!suffix.crossProduct() || dictionary.compoundOnly(suffix) + || dictionary.circumfixOnly(inner) != dictionary.circumfixOnly(suffix)) { + return; + } + final String root = removeSuffixAllowingIdentity(word, suffix); + if (root != null) { + final List entries = analyses.lookup(root); + if (entries != null) { + for (int[] flags : entries) { + if (dictionary.supportsCrossProduct(List.of(flags), inner, suffix)) { + analyses.add(root, flags, inner, outer, suffix); + } + } + } } } @@ -612,10 +1628,11 @@ private void undoCrossProductInnerSuffix(String stem, Affix prefix, Affix outer, * the strip string the rule removed on application, and checks the rule's condition * against the restored stem. A strip-only rule, whose affix material is empty, is * undone by restoring its strip string alone. Rules that neither add nor remove - * material and candidates that would leave an empty stem are rejected. A word the - * affix material covers entirely reverses a full-strip application, which hunspell - * only performs when the affix file declares {@code FULLSTRIP}; without that - * declaration the rule does not apply. + * material are handled by {@link #removeSuffixAllowingIdentity(String, Affix)}, and + * candidates that would leave an empty stem are rejected. A word the affix material + * covers entirely reverses a full-strip application, which hunspell only performs + * when the affix file declares {@code FULLSTRIP}; without that declaration the rule + * does not apply. * * @param word The surface form. * @param suffix The rule to undo. @@ -678,7 +1695,8 @@ private boolean isIdentityRule(Affix affix) { * restores the strip string the rule removed on application, and checks the rule's * condition against the restored stem. A strip-only rule, whose affix material is * empty, is undone by restoring its strip string alone. Rules that neither add nor - * remove material and candidates that would leave an empty stem are rejected. A + * remove material are handled by {@link #removePrefixAllowingIdentity(String, Affix)}, + * and candidates that would leave an empty stem are rejected. A * word the affix material covers entirely reverses a full-strip application, which * hunspell only performs when the affix file declares {@code FULLSTRIP}; without * that declaration the rule does not apply. diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompatibilityTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompatibilityTest.java new file mode 100644 index 0000000000..23c3243bed --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompatibilityTest.java @@ -0,0 +1,429 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; + +/** Original dictionary fixtures for Hunspell conversion, casing and morphology. */ +class HunspellCompatibilityTest { + + private static final String PLURAL = "SFX A Y 1\nSFX A 0 s .\n"; + private static final String COMPOUND = "COMPOUNDFLAG C\nCOMPOUNDMIN 1\n"; + + /** + * One original dictionary and an expected stemming result. + * + * @param name The test identifier. + * @param affix The affix content. + * @param words The dictionary content. + * @param input The input form. + * @param expected The expected stems, with identity for unknown input. + */ + private record Example(String name, String affix, String words, + String input, List expected) { + /** {@inheritDoc} */ + @Override + public String toString() { + return name; + } + } + + /** + * Supplies independently written fixtures, not excerpts from published dictionaries. + * + * @return The examples. + */ + private static Stream examples() { + return Stream.of( + new Example("input-ligature", "ICONV 1\nICONV fi fi\n" + PLURAL, + "1\nfield/A\n", "fields", List.of("field")), + new Example("longest-input-match", "ICONV 2\nICONV æ ae\nICONV æx ax\n" + PLURAL, + "1\nax/A\n", "æxs", List.of("ax")), + new Example("input-end-anchor", "ICONV 1\nICONV z_ s\n" + PLURAL, + "1\nquartz/A\n", "quartzz", List.of("quartz")), + new Example("output-conversion", "OCONV 1\nOCONV ae ä\n" + PLURAL, + "1\nbaer/A\n", "baers", List.of("bär")), + new Example("ignored-input-and-dictionary", "IGNORE ’\n" + PLURAL, + "1\npe’arl/A\n", "pear’ls", List.of("pearl")), + new Example("ignored-affix", "IGNORE ’\nSFX A Y 1\nSFX A 0 s’ .\n", + "1\npearl/A\n", "pearls", List.of("pearl")), + new Example("keepcase-exact", "KEEPCASE K\n" + PLURAL, + "1\ncard/AK\n", "cards", List.of("card")), + new Example("keepcase-title", "KEEPCASE K\n" + PLURAL, + "1\ncard/AK\n", "Cards", List.of("Cards")), + new Example("keepcase-uppercase", "KEEPCASE K\n" + PLURAL, + "1\ncard/AK\n", "CARDS", List.of("CARDS")), + new Example("mixed-case-is-not-lowercase", PLURAL, + "1\ncard/A\n", "cArds", List.of("cArds")), + new Example("uppercase-proper-name", PLURAL, + "1\nMaren/A\n", "MARENS", List.of("Maren")), + new Example("turkish-case", "LANG tr\n" + PLURAL, + "1\nılık/A\n", "ILIKS", List.of("ılık")), + new Example("complex-prefixes", "COMPLEXPREFIXES\nPFX B Y 1\n" + + "PFX B 0 re/C .\nPFX C Y 1\nPFX C 0 un .\n" + PLURAL, + "1\ndo/AB\n", "unredos", List.of("do")), + new Example("complex-prefix-requires-continuation", "COMPLEXPREFIXES\n" + + "PFX B Y 1\nPFX B 0 re .\nPFX C Y 1\nPFX C 0 un .\n", + "1\ndo/BC\n", "unredo", List.of("unredo")), + new Example("complex-prefix-single-suffix", "COMPLEXPREFIXES\n" + + "SFX B Y 1\nSFX B 0 er/C .\nSFX C Y 1\nSFX C 0 s .\n", + "1\nwalk/B\n", "walkers", List.of("walkers")), + new Example("explicit-stem", PLURAL, "1\nfeet/A st:foot is:plural\n", + "feet", List.of("foot")), + new Example("affixed-explicit-stem", PLURAL, + "1\nfeet/A st:foot is:plural\n", "feets", List.of("foot")), + new Example("morphology-alias", "AM 1\nAM st:goose is:plural\n" + PLURAL, + "1\ngeese/A\t1\n", "geese", List.of("goose")), + new Example("derivational-suffix", "SFX A Y 1\nSFX A 0 ness/B . ds:noun\n" + + "SFX B Y 1\nSFX B 0 es . is:plural\n", + "1\nkind/A po:adj\n", "kindnesses", List.of("kindness")), + new Example("surface-prefix", PLURAL, "1\nroot/A sp:pre st:base\n", + "roots", List.of("prebase")), + new Example("sharp-s-uppercase", "CHECKSHARPS\n" + PLURAL, + "1\nstraße/A\n", "STRASSES", List.of("straße")), + new Example("sharp-s-keepcase", "CHECKSHARPS\nKEEPCASE K\n" + PLURAL, + "1\nstraße/AK\n", "STRASSES", List.of("straße")), + new Example("forbidden-warning", "WARN W\nFORBIDWARN\n" + PLURAL, + "1\ncard/AW\n", "cards", List.of("cards")), + new Example("compound-rule", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE RS\n" + + PLURAL, "2\nriver/R\nboat/AS\n", "riverboats", List.of("river", "boat")), + new Example("compound-rule-order", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE RS\n", + "2\nriver/R\nboat/S\n", "boatriver", List.of("boatriver")), + new Example("compound-rule-star", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE R*S\n", + "3\nriver/R\nstone/R\nboat/S\n", "riverstoneboat", List.of("river", "stone", "boat")), + new Example("compound-rule-optional", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\nCOMPOUNDRULE R?TS\n", + "3\nriver/R\nstone/T\nboat/S\n", "stoneboat", List.of("stone", "boat")), + new Example("compound-rule-long", "FLAG long\nCOMPOUNDMIN 1\nCOMPOUNDRULE 1\n" + + "COMPOUNDRULE (Ra)(Sa)\n", "2\nriver/Ra\nboat/Sa\n", + "riverboat", List.of("river", "boat")), + new Example("compound-rule-numeric", "FLAG num\nCOMPOUNDMIN 1\nCOMPOUNDRULE 1\n" + + "COMPOUNDRULE (12)(34)\n", "2\nriver/12\nboat/34\n", + "riverboat", List.of("river", "boat")), + new Example("compound-rule-homonyms", "COMPOUNDMIN 1\nCOMPOUNDRULE 1\n" + + "COMPOUNDRULE RTS\n", "3\nriver/R\nriver/T\nboat/S\n", + "riverboat", List.of("riverboat")), + new Example("compound-force-uppercase", COMPOUND + "FORCEUCASE U\n", + "2\nriver/C\nboat/CU\n", "Riverboat", List.of("river", "boat")), + new Example("compound-force-uppercase-reject", COMPOUND + "FORCEUCASE U\n", + "2\nriver/C\nboat/CU\n", "riverboat", List.of("riverboat")), + new Example("compound-root-count", COMPOUND + "COMPOUNDROOT R\nCOMPOUNDWORDMAX 3\n", + "3\nrain/CR\ncoat/C\nrack/C\n", "raincoatrack", List.of("raincoatrack")), + new Example("compound-root-count-accept", COMPOUND + "COMPOUNDROOT R\nCOMPOUNDWORDMAX 4\n", + "3\nrain/CR\ncoat/C\nrack/C\n", "raincoatrack", List.of("rain", "coat", "rack")), + new Example("compound-replacement-check", COMPOUND + "CHECKCOMPOUNDREP\nREP 1\nREP coat boat\n", + "3\nrain/C\ncoat/C\nrainboat\n", "raincoat", List.of("raincoat")), + new Example("compound-pattern", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n" + + "CHECKCOMPOUNDPATTERN er b\n", "2\nriver/C\nboat/C\n", + "riverboat", List.of("riverboat")), + new Example("compound-pattern-flags", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n" + + "CHECKCOMPOUNDPATTERN er/X b/Y\n", "2\nriver/C\nboat/CY\n", + "riverboat", List.of("river", "boat")), + new Example("compound-pattern-replacement", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n" + + "CHECKCOMPOUNDPATTERN er b X\n", "2\nriver/C\nboat/C\n", + "rivXoat", List.of("river", "boat")), + new Example("compound-simplified-triple", COMPOUND + "CHECKCOMPOUNDTRIPLE\nSIMPLIFIEDTRIPLE\n", + "2\nmill/C\nloom/C\n", "milloom", List.of("mill", "loom")), + new Example("compound-more-suffixes", COMPOUND + "COMPOUNDMORESUFFIXES\n" + + "SFX A Y 1\nSFX A 0 er/B .\nSFX B Y 1\nSFX B 0 s .\n", + "2\nriver/C\nboat/CA\n", "riverboaters", List.of("river", "boat")), + new Example("compound-syllable-limit", COMPOUND + "LANG hu\nCOMPOUNDWORDMAX 2\n" + + "COMPOUNDSYLLABLE 4 aeiouy\n", "3\nray/C\nme/C\nfa/C\n", + "raymefa", List.of("ray", "me", "fa")), + new Example("compound-syllable-limit-reject", COMPOUND + "LANG hu\nCOMPOUNDWORDMAX 2\n" + + "COMPOUNDSYLLABLE 2 aeiouy\n", "3\nray/C\nme/C\nfa/C\n", + "raymefa", List.of("raymefa")), + new Example("break-default", PLURAL, "2\nriver/A\nboat/A\n", + "rivers-boats", List.of("river", "boat")), + new Example("break-recursive", PLURAL, "2\nriver/A\nboat/A\n", + "rivers-boats-rivers", List.of("river", "boat")), + new Example("break-start", PLURAL, "1\nriver/A\n", "-rivers", List.of("river")), + new Example("break-end", PLURAL, "1\nriver/A\n", "rivers-", List.of("river")), + new Example("break-custom", "BREAK 1\nBREAK ::\n" + PLURAL, + "2\nriver/A\nboat/A\n", "rivers::boats", List.of("river", "boat")), + new Example("break-disabled", "BREAK 0\n" + PLURAL, + "2\nriver/A\nboat/A\n", "rivers-boats", List.of("rivers-boats")), + new Example("break-unknown-part", PLURAL, "1\nriver/A\n", + "rivers-absent", List.of("rivers-absent")), + new Example("break-internal-only", "BREAK 1\nBREAK -\n" + PLURAL, + "1\nriver/A\n", "-rivers", List.of("-rivers")), + new Example("replacement-trailing-fields", COMPOUND + "CHECKCOMPOUNDREP\n" + + "REP 1\nREP coat boat trailing_metadata\n", + "3\nrain/C\ncoat/C\nrainboat\n", "raincoat", List.of("raincoat")), + new Example("replacement-morphology", COMPOUND + "CHECKCOMPOUNDREP\n", + "3\nrain/C\ncoat/C\nrainboat ph:raincoat\n", "raincoat", List.of("raincoat")), + new Example("replacement-morphology-arrow", COMPOUND + "CHECKCOMPOUNDREP\n", + "3\nrain/C\ncoat/C\nrainboat ph:coat->boat\n", "raincoat", List.of("raincoat")), + new Example("replacement-morphology-star-unlisted", COMPOUND + "CHECKCOMPOUNDREP\n" + PLURAL, + "3\nrain/C\ncoat/C\nrainboats/A ph:raincoats*\n", "raincoat", List.of("rain", "coat")), + new Example("replacement-morphology-star", COMPOUND + "CHECKCOMPOUNDREP\n" + PLURAL, + "4\nrain/C\ncoat/C\nrainboats/A ph:raincoats*\nrainboat\n", + "raincoat", List.of("raincoat")), + new Example("derivation-surface-prefix", "PFX U Y 1\nPFX U 0 un . dp:pfx_un sp:un\n" + + "SFX A Y 1\nSFX A 0 able/U . ds:der_able\n", "1\ndrink/A po:verb\n", + "undrinkable", List.of("undrinkable")), + new Example("derivation-inflectional-prefix", "PFX P Y 1\nPFX P 0 un . ip:un\n" + + "SFX R Y 1\nSFX R 0 able/P . ds:DER\n", "1\ndrink/R po:verb\n", + "undrinkable", List.of("drinkable")), + new Example("compound-pattern-substitution-only", COMPOUND + "CHECKCOMPOUNDPATTERN 2\n" + + "CHECKCOMPOUNDPATTERN o b z\nCHECKCOMPOUNDPATTERN oo ba u\n", + "2\nfoo/C\nbar/C\n", "fozar", List.of("foo", "bar")), + new Example("compound-pattern-substitution-second", COMPOUND + "CHECKCOMPOUNDPATTERN 2\n" + + "CHECKCOMPOUNDPATTERN o b z\nCHECKCOMPOUNDPATTERN oo ba u\n", + "2\nfoo/C\nbar/C\n", "fur", List.of("foo", "bar")), + new Example("compound-duplicate-last-parts", COMPOUND + "CHECKCOMPOUNDDUP\n", + "2\nfoo/C\nbar/C\n", "foofoobar", List.of("foo", "bar")), + new Example("compound-duplicate-reject", COMPOUND + "CHECKCOMPOUNDDUP\n", + "2\nfoo/C\nbar/C\n", "foobarbar", List.of("foobarbar")), + new Example("compound-forbid-entry", "COMPOUNDFLAG X\nCOMPOUNDPERMITFLAG Y\n" + + "COMPOUNDFORBIDFLAG Z\nSFX S Y 2\nSFX S 0 bar/YX .\nSFX S 0 baz/YX .\n", + "3\nfoo/S\nexample/X\nfoobaz/Z\n", "foobazexample", List.of("foobazexample")), + new Example("compound-forbid-entry-other-suffix", "COMPOUNDFLAG X\nCOMPOUNDPERMITFLAG Y\n" + + "COMPOUNDFORBIDFLAG Z\nSFX S Y 2\nSFX S 0 bar/YX .\nSFX S 0 baz/YX .\n", + "3\nfoo/S\nexample/X\nfoobaz/Z\n", "foobarexample", List.of("foo", "example")), + new Example("compound-only-suffix-at-end", COMPOUND + "ONLYINCOMPOUND O\n" + + "COMPOUNDPERMITFLAG P\nSFX B Y 1\nSFX B 0 s/OP .\n", + "2\nfoo/C\npseudo/CB\n", "foopseudos", List.of("foopseudos")), + new Example("compound-only-suffix-inside", COMPOUND + "ONLYINCOMPOUND O\n" + + "COMPOUNDPERMITFLAG P\nSFX B Y 1\nSFX B 0 s/OP .\n", + "2\nfoo/C\npseudo/CB\n", "pseudosfoo", List.of("pseudo", "foo")), + new Example("compound-replacement-inner", COMPOUND + "CHECKCOMPOUNDREP\n" + + "REP 1\nREP forbiddenroot forbidden_root\n", + "3\nroot/C\nforbidden/C\nforbidden root\n", "rootforbiddenroot", + List.of("rootforbiddenroot")), + new Example("compound-replacement-inner-unaffected", COMPOUND + "CHECKCOMPOUNDREP\n" + + "REP 1\nREP forbiddenroot forbidden_root\n", + "3\nroot/C\nforbidden/C\nforbidden root\n", "rootforbidden", + List.of("root", "forbidden")), + new Example("mixed-case-initial-capital", "PFX a Y 1\nPFX a u no u\n", + "1\nuLinda/a\n", "NoLinda", List.of("uLinda")), + new Example("mixed-case-initial-capital-entry", "PFX a Y 1\nPFX a u no u\n", + "1\nuLinda/a\n", "ULinda", List.of("uLinda")), + new Example("forbidden-affixed-blocks-compound", "FORBIDDENWORD F\nCOMPOUNDFLAG C\n" + + "COMPOUNDMIN 1\nSFX S Y 1\nSFX S 0 s .\n", + "4\nfoo/CS\nword/C\nbar/CS\nfoowordbar/FS\n", "foowordbars", List.of("foowordbars")), + new Example("forbidden-affixed-other-order", "FORBIDDENWORD F\nCOMPOUNDFLAG C\n" + + "COMPOUNDMIN 1\nSFX S Y 1\nSFX S 0 s .\n", + "4\nfoo/CS\nword/C\nbar/CS\nfoowordbar/FS\n", "barwordfoos", + List.of("bar", "word", "foo")), + new Example("turkic-capitalized-entry", "LANG tr\n", "1\nİzmir\n", + "İZMİR", List.of("İzmir")), + new Example("break-number-sign", "BREAK 1\nBREAK #\n" + PLURAL, + "2\nriver/A\nboat/A\n", "rivers#boats", List.of("river", "boat")), + new Example("flag-number-sign", "NEEDAFFIX #\n" + PLURAL, + "2\nfoo/#A\nbar/A\n", "foos", List.of("foo")), + new Example("flag-number-sign-virtual-stem", "NEEDAFFIX #\n" + PLURAL, + "2\nfoo/#A\nbar/A\n", "foo", List.of("foo")), + new Example("hidden-capital-mixed-case", PLURAL, "1\niPod/A\n", "IPODS", List.of("Ipod")), + new Example("hidden-capital-initial-capital", PLURAL, "1\niPod/A\n", "Ipods", List.of("Ipods")), + new Example("hidden-capital-all-caps-entry", "SFX S N 1\nSFX S 0 's .\n", + "1\nUNICEF/S\n", "UNICEF'S", List.of("Unicef")), + new Example("hidden-capital-listed-form-wins", PLURAL, "2\niPod/A\nIpod\n", + "IPODS", List.of("IPODS")), + new Example("hidden-capital-unflagged-all-caps", PLURAL, "1\nNASA\n", "Nasa", List.of("Nasa")), + new Example("hidden-capital-not-in-compound", COMPOUND + PLURAL, "2\niPod/AC\ncase/C\n", + "IPODCASE", List.of("IPODCASE")), + new Example("hungarian-hyphen-moving-rule", HUNGARIAN_HYPHEN, HUNGARIAN_WORDS, + "forróvíz-tartály", List.of("forró", "víz", "tartály")), + new Example("hungarian-hyphen-rule-needs-hyphen", HUNGARIAN_HYPHEN, HUNGARIAN_WORDS, + "forróvíz", List.of("forróvíz")), + new Example("hungarian-hyphen-rule-needs-language", HUNGARIAN_HYPHEN.replace("LANG hu", "LANG de"), + HUNGARIAN_WORDS, "forróvíz-tartály", List.of("forróvíz-tartály")), + new Example("hungarian-hyphen-rule-needs-flag", HUNGARIAN_HYPHEN, + "3\nforr/S\nvíz/Y\ntartály/Y\n", "forrvíz-tartály", List.of("forrvíz-tartály")), + new Example("hungarian-hyphen-rule-first-part-only", HUNGARIAN_HYPHEN, HUNGARIAN_WORDS, + "tartály-forróvíz", List.of("tartály-forróvíz")), + new Example("apostrophe-all-caps", "PFX P Y 1\nPFX P 0 l' .\n", "1\nAfrique/P\n", + "L'AFRIQUE", List.of("Afrique")), + new Example("apostrophe-capitalized", "PFX P Y 1\nPFX P 0 l' .\n", "1\nAfrique/P\n", + "L'Afrique", List.of("Afrique")), + new Example("trailing-period", PLURAL, "2\ntext/A\netc.\n", "texts.", List.of("text")), + new Example("trailing-periods", PLURAL, "2\ntext/A\netc.\n", "texts...", List.of("text")), + new Example("trailing-period-entry", PLURAL, "2\ntext/A\netc.\n", "etc.", List.of("etc.")), + new Example("trailing-period-not-added", PLURAL, "2\ntext/A\netc.\n", "etc", List.of("etc")), + new Example("numeric-flag-maximum", "FLAG num\nSFX 65535 Y 1\nSFX 65535 0 s .\n", + "1\ndog/65535\n", "dogs", List.of("dog")), + // the manual's example of part stems here against the concatenated native stem + new Example("compound-part-stems", COMPOUND + PLURAL, "2\nriver/C\nboat/CA\n", + "riverboats", List.of("river", "boat"))); + } + + private static final String HUNGARIAN_HYPHEN = "LANG hu\nCOMPOUNDFLAG Y\nCOMPOUNDMIN 2\n" + + "COMPOUNDFORBIDFLAG !\nBREAK 1\nBREAK -\nSFX S Y 1\nSFX S 0 ó .\n"; + private static final String HUNGARIAN_WORDS = "4\nforr/S\nvíz/Y\nforró/F!\ntartály/Y\n"; + + /** + * Checks the Java implementation using the original fixture. + * + * @param example The dictionary and assertion. + * @throws IOException If loading fails. + */ + @ParameterizedTest(name = "{0}") + @MethodSource("examples") + void testStemming(Example example) throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(("SET UTF-8\n" + example.affix()) + .getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8))); + final List actual = new HunspellStemmer(dictionary).stemAll(example.input()) + .stream().map(CharSequence::toString).toList(); + Assertions.assertEquals(example.expected(), actual); + } + + /** + * The stems the reference implementation returned for a fixture when the fixtures were + * recorded, as described in {@code dev/README-hunspell-dictionaries.md}. An empty + * reference result is recorded as the input itself. + * + * @param example The fixture. + * @return The recorded reference stems. + */ + private static List referenceStems(Example example) { + return switch (example.name()) { + case "keepcase-title", "keepcase-uppercase", "forbidden-warning" -> List.of("card"); + case "complex-prefixes", "sharp-s-uppercase", "sharp-s-keepcase" -> List.of(example.input()); + case "compound-rule", "compound-force-uppercase", "compound-force-uppercase-reject", + "compound-pattern-flags" -> List.of("river"); + case "compound-root-count-accept" -> List.of("raincoat"); + case "compound-replacement-check", "replacement-trailing-fields", + "replacement-morphology", "replacement-morphology-arrow", "replacement-morphology-star", + "replacement-morphology-star-unlisted" -> + List.of("rain"); + case "compound-pattern-replacement", "compound-simplified-triple", + "compound-pattern-substitution-only", "compound-pattern-substitution-second", + "compound-only-suffix-inside" -> List.of(example.input()); + case "compound-syllable-limit" -> List.of("rayme"); + case "compound-duplicate-last-parts" -> List.of("foofoo"); + case "compound-forbid-entry-other-suffix" -> List.of("foobar"); + case "compound-replacement-inner" -> List.of("rootforbidden"); + case "compound-replacement-inner-unaffected" -> List.of("root"); + case "forbidden-affixed-other-order" -> List.of("barwordfoo"); + case "forbidden-affixed-blocks-compound" -> List.of("foowordbar"); + case "hidden-capital-initial-capital" -> List.of("Ipod"); + case "mixed-case-initial-capital", "mixed-case-initial-capital-entry" -> + List.of(example.input()); + case "break-default", "break-recursive", "break-start", "break-end", "break-custom", + "break-number-sign", "hungarian-hyphen-moving-rule", "apostrophe-all-caps", + "apostrophe-capitalized" -> List.of(example.input()); + default -> example.name().startsWith("compound-") && example.expected().size() > 1 + ? List.of(String.join("", example.expected())) : example.expected(); + }; + } + + /** + * Whether the reference spell checker accepted a fixture input when the fixtures + * were recorded. + * + * @param example The fixture. + * @return The recorded recognition outcome. + */ + private static boolean referenceAccepts(Example example) { + return switch (example.name()) { + case "keepcase-title", "keepcase-uppercase", "forbidden-warning", + "mixed-case-is-not-lowercase", "complex-prefix-requires-continuation", + "complex-prefix-single-suffix", "compound-rule-order", "compound-rule-homonyms", + "compound-force-uppercase-reject", "compound-root-count", "compound-replacement-check", + "compound-pattern", "compound-syllable-limit-reject", "break-disabled", + "break-unknown-part", "break-internal-only", "replacement-trailing-fields", + "replacement-morphology", "replacement-morphology-arrow", "replacement-morphology-star", + "compound-duplicate-reject", "compound-forbid-entry", "compound-only-suffix-at-end", + "compound-replacement-inner", "forbidden-affixed-blocks-compound", + "flag-number-sign-virtual-stem", "hidden-capital-initial-capital", + "hidden-capital-listed-form-wins", "hidden-capital-unflagged-all-caps", + "hidden-capital-not-in-compound", "hungarian-hyphen-rule-needs-hyphen", + "hungarian-hyphen-rule-needs-language", "hungarian-hyphen-rule-needs-flag", + "hungarian-hyphen-rule-first-part-only", "trailing-period-not-added", + "turkic-capitalized-entry" -> false; + default -> true; + }; + } + + /** + * The fixtures whose recognition deliberately differs from the recorded reference + * spell-checker outcome, each with the manual's reason. + */ + private static final Map RECOGNITION_DEVIATIONS = Map.of( + "turkic-capitalized-entry", "the reference checker rejects what its analyzer stems"); + + /** + * The fixtures whose single stem differs from the recorded reference stem, each with + * the manual's reason. Fixtures with several part stems, and fixtures the reference + * checker rejects while its analyzer still stems them, are covered structurally. + */ + private static final Map STEM_DEVIATIONS = Map.ofEntries( + Map.entry("complex-prefixes", "the reference analyzer reverses field text under COMPLEXPREFIXES"), + Map.entry("sharp-s-uppercase", "the reference analyzer does not expand SS to a sharp s"), + Map.entry("sharp-s-keepcase", "the reference analyzer does not expand SS to a sharp s"), + Map.entry("compound-pattern-replacement", "the reference analyzer does not restore replaced junctions"), + Map.entry("compound-simplified-triple", "the reference analyzer does not restore simplified triples"), + Map.entry("mixed-case-initial-capital", "the reference analyzer keeps the initial capital"), + Map.entry("mixed-case-initial-capital-entry", "the reference analyzer keeps the initial capital"), + Map.entry("apostrophe-all-caps", "the reference analyzer does not undo an elided-article prefix"), + Map.entry("apostrophe-capitalized", "the reference analyzer does not undo an elided-article prefix"), + Map.entry("hidden-capital-initial-capital", "the reference analyzer ignores the capitalized input"), + Map.entry("break-start", "the reference analyzer does not split at BREAK separators"), + Map.entry("break-end", "the reference analyzer does not split at BREAK separators")); + + /** + * Tests recognition against the recorded reference outcome. A fixture listed in + * {@link #RECOGNITION_DEVIATIONS} must differ, so a stale entry fails too. + * + * @param example The fixture. + * @throws IOException If loading fails. + */ + @ParameterizedTest(name = "recognition {0}") + @MethodSource("examples") + void testRecognitionAgainstReference(Example example) throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(("SET UTF-8\n" + example.affix()).getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8))); + final boolean recognized = !new HunspellStemmer(dictionary).analyze(example.input()).isEmpty(); + final String deviation = RECOGNITION_DEVIATIONS.get(example.name()); + Assertions.assertEquals(deviation == null, recognized == referenceAccepts(example), + deviation == null ? "recognition differs from the reference" : deviation); + } + + /** + * Tests stems against the recorded reference stems. Part stems of compounds and + * break forms differ by design from the concatenated reference stem, the reference + * analyzer stems some forms its checker rejects, and the remaining differences are + * listed in {@link #STEM_DEVIATIONS}; such a fixture must differ, so a stale entry + * fails too. + * + * @param example The fixture. + */ + @ParameterizedTest(name = "stems {0}") + @MethodSource("examples") + void testStemsAgainstReference(Example example) { + final boolean same = example.expected().equals(referenceStems(example)); + if (example.expected().size() > 1 || !referenceAccepts(example)) { + return; + } + final String deviation = STEM_DEVIATIONS.get(example.name()); + Assertions.assertEquals(deviation == null, same, + deviation == null ? "stems differ from the reference: " + referenceStems(example) : deviation); + } +} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompletionTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompletionTest.java new file mode 100644 index 0000000000..13628ce743 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellCompletionTest.java @@ -0,0 +1,408 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.TreeSet; +import java.util.concurrent.Callable; +import java.util.concurrent.Executors; +import java.util.stream.Stream; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +import opennlp.tools.util.StringUtil; + +/** Original compound and morphology fixtures. */ +class HunspellCompletionTest { + + private static final String COMPOUND = "COMPOUNDFLAG C\nCOMPOUNDMIN 1\n"; + private static final String HUNGARIAN = COMPOUND + "LANG hu\nCOMPOUNDWORDMAX 2\n"; + private static final String THREE_WORDS = "3\nray/C\nme/C\nfa/C"; + + /** + * An original dictionary with expected recognition and stems. + * + * @param name The case identifier. + * @param affix The affix definitions. + * @param words The dictionary entries. + * @param input The text to analyze. + * @param stems The expected stems. + * @param accepted The native recognition expectation. + */ + private record Example(String name, String affix, String words, String input, + List stems, boolean accepted) { + /** {@inheritDoc} */ + @Override + public String toString() { + return name; + } + } + + /** {@return compound and morphology cases with independently written data} */ + private static Stream examples() { + return Stream.of( + new Example("legacy-lemma", "LEMMA_PRESENT L\nSFX A Y 1\nSFX A 0 s .\n", + "1\nfeet/AL st:foot\n", "feets", List.of("foot"), true), + new Example("syllable-c-reject", HUNGARIAN + "COMPOUNDSYLLABLE 5 aeiouy\n" + + "SYLLABLENUM klmc\nSFX c Y 1\nSFX c 0 s .\n", + THREE_WORDS + "c\n", "raymefas", List.of("raymefas"), false), + new Example("syllable-c-accept", HUNGARIAN + "COMPOUNDSYLLABLE 6 aeiouy\n" + + "SYLLABLENUM klmc\nSFX c Y 1\nSFX c 0 s .\n", + THREE_WORDS + "c\n", "raymefas", List.of("ray", "me", "fa"), true), + new Example("syllable-j-reject", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n" + + "SYLLABLENUM klmc\nSFX J Y 1\nSFX J 0 s .\n", + THREE_WORDS + "J\n", "raymefas", List.of("raymefas"), false), + new Example("syllable-i-with-j", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n" + + "SYLLABLENUM klmc\nSFX I Y 1\nSFX I 0 s .\n", + THREE_WORDS + "IJ\n", "raymefas", List.of("raymefas"), false), + new Example("syllable-i-without-j", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n" + + "SYLLABLENUM klmc\nSFX I Y 1\nSFX I 0 s .\n", + THREE_WORDS + "I\n", "raymefas", List.of("ray", "me", "fa"), true), + new Example("syllable-terminal-inflection", HUNGARIAN + "COMPOUNDSYLLABLE 4 aeiouy\n" + + "SFX A Y 1\nSFX A 0 a .\n", THREE_WORDS + "A\n", + "raymefaa", List.of("ray", "me", "fa"), true), + new Example("syllable-unaffixed-i", HUNGARIAN + "COMPOUNDSYLLABLE 3 aeiouy\n", + THREE_WORDS + "I\n", "raymefa", List.of("ray", "me", "fa"), true), + new Example("syllable-prefix-word-count", HUNGARIAN + "COMPOUNDSYLLABLE 2 aeiou\n" + + "PFX A Y 1\nPFX A 0 reco .\n", "2\nme/CA\nfa/C\n", + "recomefa", List.of("recomefa"), false), + new Example("multiple-triple-junctions", COMPOUND + "CHECKCOMPOUNDTRIPLE\nSIMPLIFIEDTRIPLE\n", + "2\nmill/C\nloom/C\n", "milloommilloom", List.of("mill", "loom"), true), + new Example("multiple-pattern-junctions", COMPOUND + "CHECKCOMPOUNDPATTERN 1\n" + + "CHECKCOMPOUNDPATTERN er b X\n", "2\nriver/C\nboat/C\n", + "rivXoatrivXoat", List.of("rivXoatrivXoat"), false), + new Example("compound-cross-product", COMPOUND + "COMPOUNDPERMITFLAG P\n" + + "PFX A Y 1\nPFX A 0 re .\nSFX B Y 1\nSFX B 0 s/P .\n", + "2\nriver/CAB\nboat/C\n", "reriversboat", List.of("river", "boat"), true), + new Example("compound-cross-product-no-permit", COMPOUND + "COMPOUNDPERMITFLAG P\n" + + "PFX A Y 1\nPFX A 0 re .\nSFX B Y 1\nSFX B 0 s .\n", + "2\nriver/CAB\nboat/C\n", "reriversboat", List.of("reriversboat"), false), + new Example("optional-affix-condition", "SFX A Y 1\nSFX A 0 s\n", + "1\ncard/A\n", "cards", List.of("card"), true), + new Example("optional-condition-with-morphology", "SFX A Y 1\nSFX A 0 s is:plural\n", + "1\ncard/A\n", "cards", List.of("cards"), false), + new Example("obsolete-compound-first", "COMPOUNDFIRST V\n", + "2\nriver/V\nboat/V\n", "riverboat", List.of("riverboat"), false), + new Example("obsolete-compound-last", "COMPOUNDLAST V\n", + "2\nriver/V\nboat/V\n", "riverboat", List.of("riverboat"), false), + new Example("obsolete-only-root", "ONLYROOT V\nSFX A Y 1\nSFX A 0 s .\n", + "1\ncard/VA\n", "cards", List.of("card"), true), + new Example("obsolete-hungarian-linking-vowel", "LANG hu\nHU_KOTOHANGZO V\n" + + "SFX A Y 1\nSFX A 0 s .\n", "1\ncard/VA\n", "cards", List.of("card"), true), + new Example("generation-option", "GENERATE 1\nSFX A Y 1\nSFX A 0 s .\n", + "1\ncard/A\n", "cards", List.of("card"), true), + new Example("compound-word-with-space", COMPOUND, + "3\nriver/C\nboat/C\nriver boat\n", "riverboat", List.of("riverboat"), false), + new Example("compound-affixed-duplicate", COMPOUND + "CHECKCOMPOUNDDUP\n" + + "COMPOUNDPERMITFLAG P\nSFX A Y 1\nSFX A 0 s/P .\n", + "1\nriver/CA\n", "riversriver", List.of("riversriver"), false), + // the reference spell checker rejects these forms while its analyzer stems them + new Example("turkish-capitalized-name", "LANG tr_TR\n", "1\nİpek\n", + "İPEK", List.of("İpek"), false), + new Example("azerbaijani-capitalized-name", "LANG az_AZ\n", "1\nİpek\n", + "İPEK", List.of("İpek"), false), + new Example("compound-pattern-suffix-flag", COMPOUND + "COMPOUNDPERMITFLAG P\n" + + "CHECKCOMPOUNDPATTERN 1\nCHECKCOMPOUNDPATTERN s/X b\n" + + "SFX A Y 1\nSFX A 0 s/PX .\n", "2\nriver/CA\nboat/C\n", + "riversboat", List.of("riversboat"), false), + new Example("compound-pattern-prefix-flag", COMPOUND + "COMPOUNDPERMITFLAG P\n" + + "CHECKCOMPOUNDPATTERN 1\nCHECKCOMPOUNDPATTERN r r/X\n" + + "PFX A Y 1\nPFX A 0 re/PX .\n", "2\nriver/C\nboat/CA\n", + "riverreboat", List.of("riverreboat"), false), + new Example("compound-cross-double-suffix", COMPOUND + "COMPOUNDPERMITFLAG P\n" + + "COMPOUNDMORESUFFIXES\nPFX R Y 1\nPFX R 0 re .\n" + + "SFX A Y 1\nSFX A 0 er/BP .\nSFX B Y 1\nSFX B 0 s/P .\n", + "2\nboat/CAR\nriver/C\n", "reboatersriver", List.of("reboatersriver"), false), + new Example("compound-cross-double-suffix-final", COMPOUND + "COMPOUNDPERMITFLAG P\n" + + "COMPOUNDMORESUFFIXES\nPFX R Y 1\nPFX R 0 re/P .\n" + + "SFX A Y 1\nSFX A 0 er/B .\nSFX B Y 1\nSFX B 0 s .\n", + "2\nboat/CAR\nriver/C\n", "riverreboaters", List.of("river", "boat"), true), + new Example("compound-complex-prefix", COMPOUND + "COMPLEXPREFIXES\n" + + "COMPOUNDMORESUFFIXES\nPFX A Y 1\nPFX A 0 re/B .\n" + + "PFX B Y 1\nPFX B 0 un .\n", "2\nriver/CA\nboat/C\n", + "unreriverboat", List.of("river", "boat"), true), + new Example("compound-complex-prefix-suffix", COMPOUND + "COMPLEXPREFIXES\n" + + "COMPOUNDMORESUFFIXES\nCOMPOUNDPERMITFLAG P\n" + + "PFX A Y 1\nPFX A 0 re/B .\nPFX B Y 1\nPFX B 0 un .\n" + + "SFX S Y 1\nSFX S 0 s/P .\n", "2\nriver/CAS\nboat/C\n", + "unreriversboat", List.of("river", "boat"), true), + // deviations listed in the manual: Unicode case mapping, the suggester-based + // rejection of multi-part compounds, and numeric tokens + new Example("dotted-capital-i", "", "1\nimply\n", "İmply", List.of("imply"), false), + new Example("dotted-capital-i-all-caps", "", "1\nİzmir\n", "İZMİR", List.of("İzmir"), true), + // KEEPCASE with CHECKSHARPS admits the SS spelling of an all-uppercase form only + new Example("keepcase-sharp-s-double-s", "CHECKSHARPS\nKEEPCASE k\n", "1\nmüßig/k\n", + "MÜSSIG", List.of("müßig"), true), + new Example("keepcase-capital-sharp-s", "CHECKSHARPS\nKEEPCASE k\n", "1\nmüßig/k\n", + "MÜẞIG", List.of("MÜẞIG"), false), + new Example("keepcase-sharp-s-capitalized", "CHECKSHARPS\nKEEPCASE k\n", "1\nmüßig/k\n", + "Müßig", List.of("müßig"), true), + new Example("multi-part-compound-near-listed-word", "TRY esianrtolcdugmphbyfvkwz\n" + + "COMPOUNDFLAG x\n", "5\nfoo/x\nbar/x\nbaz/x\ngoobar\ngoobarbaz\n", + "foobarbaz", List.of("foo", "bar", "baz"), false), + new Example("numeric-token", "", "1\nfoo\n", "1.5", List.of("1.5"), true)); + } + + /** + * Tests Java stems under strict loading. + * + * @param example The fixture. + * @throws IOException If loading fails. + */ + @ParameterizedTest(name = "{0}") + @MethodSource("examples") + void testStems(Example example) throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(("SET UTF-8\n" + example.affix()).getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8))); + Assertions.assertEquals(example.stems(), new HunspellStemmer(dictionary).stemAll(example.input())); + } + + /** + * The fixtures whose recognition deliberately differs from the recorded reference + * spell-checker outcome, each with the manual's reason. + */ + private static final Map RECOGNITION_DEVIATIONS = Map.of( + "turkish-capitalized-name", "the reference checker rejects what its analyzer stems", + "azerbaijani-capitalized-name", "the reference checker rejects what its analyzer stems", + "dotted-capital-i", "a dotted capital I is lowercased outside the Turkic languages", + "multi-part-compound-near-listed-word", "the reference rejects it through its suggester", + "numeric-token", "numbers are accepted natively before any lookup"); + + /** + * Tests recognition against the outcome recorded from the reference spell checker. + * A fixture listed in {@link #RECOGNITION_DEVIATIONS} must differ, so a stale entry + * fails too. + * + * @param example The fixture. + * @throws IOException If loading fails. + */ + @ParameterizedTest(name = "recognition {0}") + @MethodSource("examples") + void testRecognitionAgainstReference(Example example) throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(("SET UTF-8\n" + example.affix()).getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(example.words().getBytes(StandardCharsets.UTF_8))); + final boolean recognized = !new HunspellStemmer(dictionary).analyze(example.input()).isEmpty(); + final String deviation = RECOGNITION_DEVIATIONS.get(example.name()); + Assertions.assertEquals(deviation == null, recognized == example.accepted(), + deviation == null ? "recognition differs from the reference" : deviation); + } + + /** + * Requires a public analysis operation preserving entry and affix fields. + * + * @throws Exception If reflection, loading, or analysis fails. + */ + @Test + void testMorphologicalAnalysis() throws Exception { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(("SFX A Y 1\nSFX A 0 s . is:plural\n") + .getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream("2\ncard/A po:noun\ncard/A po:verb\n".getBytes(StandardCharsets.UTF_8))); + final HunspellStemmer stemmer = new HunspellStemmer(dictionary); + Assertions.assertEquals(List.of("st:card po:noun is:plural", "st:card po:verb is:plural"), + stemmer.analyze("cards")); + Assertions.assertEquals(List.of(), stemmer.analyze("unlisted")); + Assertions.assertThrows(IllegalArgumentException.class, () -> stemmer.analyze(null)); + Assertions.assertEquals(List.of(), stemmer.analyze("")); + Assertions.assertThrows(UnsupportedOperationException.class, () -> stemmer.analyze("cards").clear()); + } + + /** + * Checks the documented maximum number of CHECKSHARPS case variants. + * + * @throws IOException If fixture loading fails. + */ + @Test + void testSharpVariantLimit() throws IOException { + final StringBuilder words = new StringBuilder("129\nSSSSSSSSSSSS\n"); + for (int mask = 0; mask < 64; mask++) { + final StringBuilder entry = new StringBuilder(); + for (int bit = 0; bit < 6; bit++) { + entry.append((mask & (1 << bit)) == 0 ? "ss" : "ß"); + } + words.append(entry).append('\n'); + words.append(StringUtil.toUpperCase(entry.substring(0, 1))).append(entry.substring(1)).append('\n'); + } + final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream("CHECKSHARPS\n".getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(words.toString().getBytes(StandardCharsets.UTF_8)))); + Assertions.assertTrue(stemmer.stemAll("SSSSSSSSSSSS").size() <= 64); + } + + /** + * Checks shared use and protection of dictionary flags returned for inspection. + * + * @throws Exception If fixture loading or a worker fails. + */ + @Test + void testSharedMorphologyAndImmutableFlags() throws Exception { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream("SFX A Y 1\nSFX A 0 s . is:plural\n".getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream("2\ncard/A po:noun\ncard/A po:verb\n".getBytes(StandardCharsets.UTF_8))); + dictionary.lookup("card").getFirst()[0] = 'Z'; + dictionary.lookup("card").clear(); + final HunspellStemmer stemmer = new HunspellStemmer(dictionary); + final List expected = List.of("st:card po:noun is:plural", "st:card po:verb is:plural"); + try (var executor = Executors.newFixedThreadPool(4)) { + final List>> tasks = new ArrayList<>(); + for (int i = 0; i < 64; i++) { + tasks.add(() -> { + Assertions.assertEquals(List.of("card"), stemmer.stemAll("cards")); + Assertions.assertEquals(List.of(), stemmer.analyze("unlisted")); + return stemmer.analyze("cards"); + }); + } + for (var result : executor.invokeAll(tasks)) { + Assertions.assertEquals(expected, result.get()); + } + } + } + + /** + * Supplies morphology expected from the native reference. + * + * @return Affix content, dictionary content, input, and expected analysis. + */ + private static Stream morphology() { + return Stream.of( + new String[] {"SFX A Y 1\nSFX A 0 s .\n", "1\ncard/A\n", "cards", "st:card fl:A"}, + new String[] {"SFX A Y 1\nSFX A 0 s . is:plural\n", "1\ncard/A po:noun\n", + "cards", "st:card po:noun is:plural"}, + new String[] {"AM 2\nAM st:foot ts:present\nAM is:plural\nSFX A Y 1\nSFX A 0 s . 2\n", + "1\nfeet/A\t1\n", "feets", "st:foot ts:present is:plural"}, + new String[] {"PFX B Y 1\nPFX B 0 re . dp:again\nSFX A Y 1\nSFX A 0 s . is:plural\n", + "1\ngo/AB po:verb\n", "regos", "dp:again st:go po:verb is:plural"}, + new String[] {COMPOUND + "SFX A Y 1\nSFX A 0 s . is:plural\n", + "2\nriver/C po:noun\nboat/CA po:noun\n", "riverboats", + "pa:river st:river po:noun pa:boats st:boat po:noun is:plural"}, + new String[] {"SFX A Y 1\nSFX A 0 s . is:plural\n", "1\ncard/A custom\n", + "cards", "st:card is:plural"}, + new String[] {"NEEDAFFIX X\nSFX A Y 1\nSFX A 0 0 .\n", "1\nfoo/XA\n", "foo", "st:foo fl:A"}, + new String[] {"PFX C Y 1\nPFX C 0 pre .\n", "1\nfoo/C\n", "prefoo", "pre st:foo fl:C"}, + new String[] {"PFX C Y 1\nPFX C 0 pre .\n", "1\nfoo/C po:noun\n", "prefoo", "pre st:foo po:noun"}, + new String[] {"PFX B Y 1\nPFX B 0 re .\nSFX A Y 1\nSFX A 0 s . is:plural\n", + "1\ngo/AB po:verb\n", "regos", "fl:B st:go po:verb is:plural"}, + new String[] {"PFX B Y 1\nPFX B 0 re . dp:again\nSFX A Y 1\nSFX A 0 s .\n", + "1\ngo/AB\n", "regos", "dp:again st:go fl:A"}, + new String[] {"SFX A Y 1\nSFX A 0 er/B .\nSFX B Y 1\nSFX B 0 s .\n", + "1\nwalk/A po:verb\n", "walkers", "st:walk po:verb fl:A fl:B"}, + new String[] {COMPOUND, "2\nfoo/C id:1\nbar/C\n", "foobar", "pa:foo st:foo id:1 pa:bar"}, + new String[] {COMPOUND, "3\nfoo/C\nbar/C id:2\nbaz/C\n", "foobarbaz", + "pa:foo st:foo pa:bar st:bar id:2 pa:baz"}, + new String[] {COMPOUND + "SFX A Y 1\nSFX A 0 s .\n", "2\nfoo/C id:1\nbar/CA\n", "foobars", + "pa:foo st:foo id:1 pa:bars st:bar fl:A"}, + new String[] {"SFX A Y 1\nSFX A 0 s .\n", "1\niPod/A po:noun\n", "IPODS", "st:Ipod po:noun fl:A"}); + } + + /** + * Compares morphological fields with native output on original fixtures. + * + * @param affix The affix content. + * @param words The entry content. + * @param input The input word. + * @param expected The expected analysis. + * @throws Exception If parsing or reference execution fails. + */ + /** + * Analyses that include a rule adding and removing no material, in reference order. + * + * @return Affix content, word list, input, and every expected analysis. + */ + private static Stream zeroAffixAnalyses() { + return Stream.of( + Arguments.of("SFX A Y 1\nSFX A 0 0 . is:zero\n", "1\nbar/A\n", "bar", + List.of("st:bar", "st:bar is:zero")), + Arguments.of("PFX A Y 1\nPFX A 0 0 . dp:zero\n", "1\nbar/A\n", "bar", + List.of("st:bar", "dp:zero st:bar fl:A")), + Arguments.of("NEEDAFFIX X\nSFX A Y 1\nSFX A 0 0 . >\nSFX B Y 1\nSFX B 0 0 . >\n" + + "SFX C Y 2\nSFX C 0 0/XAB . \nSFX C 0 baz/XAB . \n", + "1\nbar/XABC\t", "st:bar >", "st:bar >", + "st:bar >"))); + } + + /** + * Tests that a rule without material is undone on its own and inside a continuation. + * + * @param affix The affix content. + * @param words The word list. + * @param input The analyzed word. + * @param expected The distinct analyses, as recorded from the reference analyzer. + * @throws IOException If loading fails. + */ + @ParameterizedTest + @MethodSource("zeroAffixAnalyses") + void testZeroAffixAnalyses(String affix, String words, String input, List expected) + throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(words.getBytes(StandardCharsets.UTF_8)))); + Assertions.assertEquals(new TreeSet<>(expected), new TreeSet<>(stemmer.analyze(input))); + } + + /** + * Tests that only the first listed homonym decides whether a spelling is forbidden. + * The reference spell checker accepts {@code foo} with the valid homonym listed first + * and rejects it with the forbidden homonym listed first. + * + * @throws IOException If loading fails. + */ + @Test + void testForbiddenFirstHomonym() throws IOException { + final String affix = "FORBIDDENWORD X\nCOMPOUNDFLAG Y\nCOMPOUNDMIN 1\n"; + final HunspellStemmer allowed = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream("2\nfoo/S\nfoo/YX\n".getBytes(StandardCharsets.UTF_8)))); + Assertions.assertEquals(List.of("st:foo"), allowed.analyze("foo")); + Assertions.assertEquals(List.of(), allowed.analyze("foofoo")); + final HunspellStemmer forbidden = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream("2\nfoo/YX\nfoo/S\n".getBytes(StandardCharsets.UTF_8)))); + Assertions.assertEquals(List.of(), forbidden.analyze("foo")); + } + + /** + * Tests analyses against the field text recorded from the reference analyzer, with + * separator whitespace normalized to single spaces. + * + * @param affix The affix content. + * @param words The word list. + * @param input The analyzed word. + * @param expected The recorded analysis. + * @throws IOException If loading fails. + */ + @ParameterizedTest + @MethodSource("morphology") + void testMorphologyFields(String affix, String words, String input, String expected) throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream(affix.getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(words.getBytes(StandardCharsets.UTF_8)))); + Assertions.assertEquals(List.of(expected), stemmer.analyze(input)); + } +} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryLoadTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryLoadTest.java new file mode 100644 index 0000000000..b5221f7ee1 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryLoadTest.java @@ -0,0 +1,442 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer.hunspell; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Stream; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.EnumSource; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.ValueSource; + +import opennlp.tools.stemmer.hunspell.HunspellDictionary.LoadMode; +import opennlp.tools.stemmer.hunspell.HunspellDictionary.UnsupportedDirective; + +/** Tests the loading policy with project-authored affix and dictionary content. */ +class HunspellDictionaryLoadTest { + + private static final String WORDS = "1\ndog/A\n"; + private static final String RULES = "SFX A Y 1\nSFX A 0 s .\n"; + + @TempDir + private Path directory; + + /** + * Rejects directives with behavior not implemented by the stemmer. + * + * @param line An unsupported directive with arguments. + */ + @ParameterizedTest + @ValueSource(strings = { + "UNSUPPORTED_CONVERSION 1", "UNSUPPORTED_CASE k", "UNSUPPORTED_SYLLABLES ABC", + "UNSUPPORTED_LEMMA L", "UNRECOGNIZED value" + }) + void testUnsupportedDirectiveFailsByDefault(String line) { + final IOException error = Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(stream("SET UTF-8\n" + line + "\n" + RULES), + stream(WORDS))); + final int separator = line.indexOf(' '); + final String directive = separator < 0 ? line : line.substring(0, separator); + Assertions.assertTrue(error.getMessage().contains(directive)); + Assertions.assertTrue(error.getMessage().contains("affix stream")); + Assertions.assertTrue(error.getMessage().contains("line 2")); + } + + /** + * Counts logical lines with each supported line separator. + * + * @param separator A supported line separator. + */ + @ParameterizedTest + @ValueSource(strings = {"\n", "\r\n", "\r"}) + void testUnsupportedDirectiveLineNumber(String separator) { + final String affix = "# comment" + separator + separator + " UNSUPPORTED_CASE K"; + final IOException error = Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(stream(affix), stream(WORDS))); + Assertions.assertTrue(error.getMessage().contains("UNSUPPORTED_CASE")); + Assertions.assertTrue(error.getMessage().contains("line 3")); + } + + /** Rejects an unsupported directive immediately after a UTF-8 byte-order mark. */ + @Test + void testByteOrderMarkDoesNotHideUnsupportedDirective() { + final IOException error = Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(stream("\uFEFFUNSUPPORTED_CONVERSION 1\n"), stream(WORDS))); + Assertions.assertTrue(error.getMessage().contains("UNSUPPORTED_CONVERSION")); + Assertions.assertTrue(error.getMessage().contains("line 1")); + } + + /** + * Identifies the source file when path-based loading rejects a directive. + * + * @throws IOException If writing a fixture fails. + */ + @Test + void testPathErrorIdentifiesAffixFile() throws IOException { + final Path affix = directory.resolve("sample.aff"); + final Path words = directory.resolve("sample.dic"); + Files.writeString(affix, "SET UTF-8\nUNSUPPORTED_CASE K\n"); + Files.writeString(words, WORDS); + final IOException error = Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(affix, words)); + Assertions.assertTrue(error.getMessage().contains(affix.toString())); + Assertions.assertTrue(error.getMessage().contains("line 2")); + } + + /** + * Loads settings outside the stemmer's operations without a diagnostic. + * + * @param setting A metadata or suggestion setting. + * @throws IOException If loading fails. + */ + @ParameterizedTest + @ValueSource(strings = { + "NAME Example", "HOME https://example.org", "VERSION 1", "KEY abc|def", + "TRY abc", "REP 1\nREP ph f", "MAP 1\nMAP aá", "PHONE 1\nPHONE ph f", + "NOSUGGEST N", "MAXCPDSUGS 0", "MAXNGRAMSUGS 0", "MAXDIFF 5", + "ONLYMAXDIFF", "NOSPLITSUGS", "SUGSWITHDOTS", "WARN W", + "SUBSTANDARD S", "WORDCHARS -" + }) + void testSettingsOutsideStemmingDoNotPreventStrictLoading(String setting) + throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + stream(setting + "\n" + RULES), stream(WORDS)); + Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString()); + Assertions.assertTrue(dictionary.getUnsupportedDirectives().isEmpty()); + } + + /** + * Keeps a number sign that is a directive value, which the reference format allows + * for flags, separators, and affix material. + * + * @param affix Affix content in which {@code #} is a value. + * @param input The stemmed word. + * @param expected The stem. + * @throws IOException If loading fails. + */ + @ParameterizedTest + @MethodSource("numberSignValues") + void testNumberSignValuesAreKept(String affix, String words, String input, String expected) + throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load(stream(affix), stream(words)); + Assertions.assertEquals(expected, new HunspellStemmer(dictionary).stem(input).toString()); + } + + /** + * Directive values consisting of a number sign. + * + * @return Affix content, word list, input, and expected stem. + */ + private static Stream numberSignValues() { + return Stream.of( + Arguments.of("BREAK 1\nBREAK #\n" + RULES, WORDS, "dogs#dogs", "dog"), + Arguments.of("NEEDAFFIX #\n" + RULES, "1\ndog/#A\n", "dogs", "dog"), + Arguments.of("FLAG long\nAF 1\nAF #A\nSFX #A Y 1\nSFX #A 0 s .\n", "1\ndog/1\n", "dogs", "dog"), + Arguments.of("SFX A Y 1\nSFX A 0 # .\n", WORDS, "dog#", "dog"), + Arguments.of("SFX A Y 1\nSFX A # s [#]\n", "1\ndog#/A\n", "dogs", "dog#")); + } + + /** + * Ignores trailing comments after the fields a directive consumes, as the reference + * implementation ignores those fields. + * + * @param affix Affix content with a trailing comment. + * @throws IOException If loading fails. + */ + @ParameterizedTest + @ValueSource(strings = { + "COMPOUNDMIN 3 # comment\nSFX A Y 1 # comment\nSFX A 0 s . # comment", + "NEEDAFFIX X # comment\nSFX A Y 1\nSFX A 0 s .", + "SET UTF-8 # comment\nFLAG UTF-8 # comment\nSFX A Y 1\nSFX A 0 s .", + "AF 1\nAF A # comment\nSFX A Y 1\nSFX A 0 s ." + }) + void testTrailingCommentsAreIgnored(String affix) throws IOException { + final String words = affix.startsWith("AF") ? "1\ndog/1\n" : WORDS; + final HunspellDictionary dictionary = HunspellDictionary.load(stream(affix), stream(words)); + Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString()); + } + + /** + * Reports the first location for each skipped directive in source order. + * + * @param separator A supported line separator. + * @throws IOException If partial loading fails. + */ + @ParameterizedTest + @ValueSource(strings = {"\n", "\r\n", "\r"}) + void testPartialLoadingReportsFirstOccurrences(String separator) throws IOException { + final String affix = String.join(separator, "UNSUPPORTED_CONVERSION 1", "UNSUPPORTED_CONVERSION a b", + "UNSUPPORTED_CASE K", "UNRECOGNIZED 1", "UNRECOGNIZED x", RULES); + final HunspellDictionary dictionary = HunspellDictionary.load( + stream(affix), stream(WORDS), LoadMode.ALLOW_PARTIAL); + final List diagnostics = dictionary.getUnsupportedDirectives(); + Assertions.assertEquals(List.of( + new UnsupportedDirective("UNSUPPORTED_CONVERSION", "affix stream", 1), + new UnsupportedDirective("UNSUPPORTED_CASE", "affix stream", 3), + new UnsupportedDirective("UNRECOGNIZED", "affix stream", 4)), diagnostics); + Assertions.assertThrows(UnsupportedOperationException.class, diagnostics::clear); + Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString()); + } + + /** + * Includes the affix path in partial-loading diagnostics. + * + * @throws IOException If writing or loading fixtures fails. + */ + @Test + void testPartialLoadingReportsFilePath() throws IOException { + final Path affix = directory.resolve("partial.aff"); + final Path words = directory.resolve("partial.dic"); + Files.writeString(affix, "SET UTF-8\nUNSUPPORTED_CASE K\n" + RULES); + Files.writeString(words, WORDS); + final HunspellDictionary dictionary = HunspellDictionary.load( + affix, words, LoadMode.ALLOW_PARTIAL); + Assertions.assertEquals(List.of(new UnsupportedDirective( + "UNSUPPORTED_CASE", affix.toString(), 2)), dictionary.getUnsupportedDirectives()); + Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString()); + } + + /** + * Rejects supported malformed content under either loading policy. + * + * @param malformed Malformed affix content. + */ + @ParameterizedTest + @ValueSource(strings = {"AF -1\n", "FLAG num\nSFX 65536 Y 0\n", + "COMPOUNDMIN -1\n", "SFX A Y 2\nSFX A 0 s .\n", "FLAG short\n", + "ICONV 1\n", "ICONV -1\n", "ICONV 1\nICONV a b\nICONV c d\n", + "OCONV 1\nOCONV _ x\n", "AM 1\n", "AM -1\n", + "COMPOUNDRULE 1\n", "COMPOUNDRULE 1\nCOMPOUNDRULE *A\n", + "COMPOUNDRULE 1\nCOMPOUNDRULE (\n", "CHECKCOMPOUNDPATTERN 1\n", + "CHECKCOMPOUNDPATTERN -1\n", "BREAK 1\n", "BREAK -1\n", "BREAK 1\nBREAK ^\n", + "IGNORE\n", "LANG\n", "COMPOUNDSYLLABLE -1 ae\n", "KEEPCASE\n", + "SYLLABLENUM\n", "LEMMA_PRESENT\n", "LEMMA_PRESENT AB\n", + "AM 1 extra\nAM po:noun\n", "CHECKCOMPOUNDPATTERN 0 extra\n"}) + void testPartialLoadingDoesNotIgnoreMalformedRules(String malformed) { + Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load( + stream(malformed), stream("1\ndog\n"), LoadMode.STRICT)); + Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load( + stream("UNSUPPORTED_CASE K\n" + malformed), stream("1\ndog\n"), LoadMode.ALLOW_PARTIAL)); + } + + /** + * Rejects invalid AM references in entries and affix fields under either policy. + * + * @param reference The invalid reference. + */ + @ParameterizedTest + @ValueSource(strings = {"-1", "0", "2", "invalid", "1 1"}) + void testInvalidMorphologyAliases(String reference) { + final String aliases = "AM 1\nAM po:noun\n"; + for (LoadMode mode : LoadMode.values()) { + Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load( + stream(aliases), stream("1\ndog\t" + reference + "\n"), mode)); + Assertions.assertThrows(IOException.class, () -> HunspellDictionary.load( + stream(aliases + "SFX A Y 1\nSFX A 0 s . " + reference + "\n"), stream(WORDS), mode)); + } + } + + /** + * Rejects malformed text in partial mode. + * + * @param file The file containing malformed UTF-8. + */ + @ParameterizedTest + @ValueSource(strings = {"affix", "dictionary"}) + void testPartialLoadingRejectsMalformedText(String file) { + final byte[] malformed = {(byte) 0xc3}; + final ByteArrayInputStream affix = "affix".equals(file) + ? new ByteArrayInputStream(concat("SET UTF-8\nUNSUPPORTED_CASE K\nSFX A Y 1\nSFX A 0 ", + malformed)) : stream("UNSUPPORTED_CASE K\n"); + final ByteArrayInputStream words = "dictionary".equals(file) + ? new ByteArrayInputStream(concat("1\n", malformed)) : stream(WORDS); + final IOException error = Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(affix, words, LoadMode.ALLOW_PARTIAL)); + Assertions.assertEquals(file + " stream is not valid UTF-8", error.getMessage()); + } + + /** + * Accepts legacy bytes in recognized metadata without decoding them as rules. + * + * @throws IOException If loading fails. + */ + @Test + void testLegacyMetadataBytesAreIgnored() throws IOException { + final byte[] affix = concat("SET UTF-8\nNAME ", new byte[] {(byte) 0xc3}); + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(affix), stream(WORDS)); + Assertions.assertNotNull(dictionary.lookup("dog")); + Assertions.assertTrue(dictionary.getUnsupportedDirectives().isEmpty()); + } + + /** + * Preserves raw flag bytes in compound-boundary conditions without changing word text. + * + * @param matchingFlag Whether the left entry has the boundary flag. + * @throws IOException If loading fails. + */ + @ParameterizedTest + @ValueSource(booleans = {true, false}) + void testCompoundPatternByteFlag(boolean matchingFlag) throws IOException { + final byte[] prefix = concat("SET UTF-8\nCOMPOUNDFLAG C\nCOMPOUNDMIN 1\n" + + "CHECKCOMPOUNDPATTERN 1\nCHECKCOMPOUNDPATTERN er/", new byte[] {(byte) 0xc3}); + final byte[] affix = Arrays.copyOf(prefix, prefix.length + 3); + System.arraycopy(" b\n".getBytes(StandardCharsets.UTF_8), 0, affix, prefix.length, 3); + final byte[] wordPrefix = concat("2\nriver/C", matchingFlag + ? new byte[] {(byte) 0xc3} : new byte[0]); + final byte[] ending = "\nboat/C\n".getBytes(StandardCharsets.UTF_8); + final byte[] words = Arrays.copyOf(wordPrefix, wordPrefix.length + ending.length); + System.arraycopy(ending, 0, words, wordPrefix.length, ending.length); + final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream(affix), new ByteArrayInputStream(words))); + Assertions.assertEquals(matchingFlag ? List.of("riverboat") : List.of("river", "boat"), + stemmer.stemAll("riverboat")); + } + + /** + * Loads supported affix rules after a UTF-8 byte-order mark. + * + * @throws IOException If loading fails. + */ + @Test + void testByteOrderMarkDoesNotHideSupportedDirective() throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + stream("\uFEFF" + RULES), stream(WORDS)); + Assertions.assertEquals("dog", new HunspellStemmer(dictionary).stem("dogs").toString()); + } + + /** + * Rejects null arguments before reading either stream or opening a file. + * + * @param mode The loading policy. + */ + @ParameterizedTest + @EnumSource(LoadMode.class) + void testNullArgumentsAreRejected(LoadMode mode) { + Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load((Path) null, directory, mode)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(directory, null, mode)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(null, stream(WORDS), mode)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(stream(RULES), null, mode)); + } + + /** Rejects a null loading policy through both public entry points. */ + @Test + void testNullModeIsRejected() { + Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(directory, directory, null)); + final ByteArrayInputStream affix = stream(RULES); + Assertions.assertThrows(IllegalArgumentException.class, + () -> HunspellDictionary.load(affix, stream(WORDS), null)); + Assertions.assertEquals(RULES.getBytes(StandardCharsets.UTF_8).length, affix.available()); + } + + /** + * Preserves ownership of input streams on success and failure. + * + * @param mode The loading policy. + * @throws IOException If valid content fails to load. + */ + @ParameterizedTest + @EnumSource(LoadMode.class) + void testStreamsAreNotClosed(LoadMode mode) throws IOException { + final TrackedStream affix = new TrackedStream(RULES); + final TrackedStream words = new TrackedStream(WORDS); + HunspellDictionary.load(affix, words, mode); + Assertions.assertFalse(affix.closed); + Assertions.assertFalse(words.closed); + final TrackedStream invalid = new TrackedStream("AF -1\n"); + Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(invalid, words, mode)); + Assertions.assertFalse(invalid.closed); + Assertions.assertFalse(words.closed); + } + + /** Verifies validation of a diagnostic's public fields. */ + @Test + void testDiagnosticArgumentsAreValidated() { + Assertions.assertThrows(IllegalArgumentException.class, + () -> new UnsupportedDirective(null, "source", 1)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> new UnsupportedDirective(" ", "source", 1)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> new UnsupportedDirective("UNSUPPORTED_CONVERSION", null, 1)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> new UnsupportedDirective("UNSUPPORTED_CONVERSION", " ", 1)); + Assertions.assertThrows(IllegalArgumentException.class, + () -> new UnsupportedDirective("UNSUPPORTED_CONVERSION", "source", 0)); + } + + /** Detects close calls while allowing further input operations. */ + private static final class TrackedStream extends ByteArrayInputStream { + private boolean closed; + + /** + * Creates an encoded fixture stream. + * + * @param content The fixture text. + */ + private TrackedStream(String content) { + super(content.getBytes(StandardCharsets.UTF_8)); + } + + /** {@inheritDoc} */ + @Override + public void close() { + closed = true; + } + } + + /** + * Appends raw bytes to a UTF-8 fixture prefix. + * + * @param prefix The fixture prefix. + * @param bytes The raw suffix. + * @return The combined content. + */ + private byte[] concat(String prefix, byte[] bytes) { + final byte[] encoded = prefix.getBytes(StandardCharsets.UTF_8); + final byte[] result = Arrays.copyOf(encoded, encoded.length + bytes.length); + System.arraycopy(bytes, 0, result, encoded.length, bytes.length); + return result; + } + + /** + * Creates a UTF-8 stream for a fixture. + * + * @param content The fixture text. + * @return The encoded stream. + */ + private ByteArrayInputStream stream(String content) { + return new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8)); + } +} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java index 21daec7e0b..91803c5294 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellManualExampleTest.java @@ -20,10 +20,13 @@ import java.io.ByteArrayInputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.List; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; import opennlp.tools.stemmer.Stemmer; @@ -70,4 +73,44 @@ void testLoadAndStemWorkers() throws IOException { // unknown vocabulary passes through unchanged Assertions.assertEquals("table", stemmer.stem("table").toString()); } + + /** + * Checks the morphological analysis example in the manual. + * + * @throws IOException If fixture loading fails. + */ + @Test + void testAnalyzeWorkers() throws IOException { + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(AFFIX.getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream(WORDS.getBytes(StandardCharsets.UTF_8))); + final HunspellStemmer analyzer = new HunspellStemmer(dictionary); + Assertions.assertEquals(List.of("st:work fl:E fl:S"), analyzer.analyze("workers")); + Assertions.assertEquals(List.of(), analyzer.analyze("table")); + } + + /** + * Checks partial loading and the diagnostic format used in the manual. + * + * @param directory The temporary fixture directory. + * @throws IOException If fixture creation or loading fails. + */ + @Test + void testPartialLoadingDiagnostics(@TempDir Path directory) throws IOException { + final Path affix = directory.resolve("dictionary.aff"); + final Path words = directory.resolve("dictionary.dic"); + Files.writeString(affix, "UNSUPPORTED value\n" + AFFIX); + Files.writeString(words, WORDS); + final HunspellDictionary partial = HunspellDictionary.load( + affix, words, HunspellDictionary.LoadMode.ALLOW_PARTIAL); + + Assertions.assertEquals("work", + new HunspellStemmerFactory(partial).newStemmer().stem("workers").toString()); + Assertions.assertEquals(1, partial.getUnsupportedDirectives().size()); + for (HunspellDictionary.UnsupportedDirective diagnostic : partial.getUnsupportedDirectives()) { + final String message = diagnostic.directive() + " at " + + diagnostic.source() + ":" + diagnostic.lineNumber(); + Assertions.assertEquals("UNSUPPORTED at " + affix + ":1", message); + } + } } diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java deleted file mode 100644 index 9221658ddd..0000000000 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellRealDictionaryTest.java +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package opennlp.tools.stemmer.hunspell; - -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Assumptions; -import org.junit.jupiter.api.Test; - -/** - * Gated checks against published dictionaries, which are never bundled: the tests run - * only when {@code -Dopennlp.hunspell.dict.dir} names a directory holding - * {@code .aff}/{@code .dic} pairs, and each test additionally skips when - * its dictionary pair is absent. The download helper in {@code dev/} fetches the pairs - * together with their license files; see {@code dev/README-hunspell-dictionaries.md}. - * - *

The assertions are limited to morphology stable across dictionary revisions: - * everyday inflections, and for German the decomposability of ordinary compounds.

- */ -public class HunspellRealDictionaryTest { - - private static final String DICT_DIR_PROPERTY = "opennlp.hunspell.dict.dir"; - - /** - * Loads one dictionary pair from the gated directory, skipping the test when the - * gate or the pair is absent. - * - * @param name The dictionary base name, such as {@code en_US}. - * @return A stemmer over the loaded pair. Never {@code null}. - * @throws IOException Thrown if a present pair fails to load, which is a failure, - * not a skip. - */ - private static HunspellStemmer loadOrSkip(String name) throws IOException { - final String dir = System.getProperty(DICT_DIR_PROPERTY); - Assumptions.assumeTrue(dir != null && !dir.isBlank(), - "no " + DICT_DIR_PROPERTY + " given"); - final Path affix = Path.of(dir, name + HunspellDictionary.AFFIX_FILE_SUFFIX); - final Path words = Path.of(dir, name + HunspellDictionary.DICTIONARY_FILE_SUFFIX); - Assumptions.assumeTrue(Files.isReadable(affix) && Files.isReadable(words), - name + " pair not present under " + dir); - return new HunspellStemmer(HunspellDictionary.load(affix, words)); - } - - /** - * Checks everyday English inflections against {@code en_US}, plus the identity - * fallback on vocabulary no dictionary lists. - * - * @throws IOException Thrown if a present dictionary pair fails to load. - */ - @Test - void testEnglishInflections() throws IOException { - final HunspellStemmer stemmer = loadOrSkip("en_US"); - Assertions.assertEquals("worker", stemmer.stem("workers").toString()); - Assertions.assertEquals("cat", stemmer.stem("cats").toString()); - Assertions.assertEquals("unhappy", stemmer.stem("unhappiest").toString()); - Assertions.assertEquals("quick", stemmer.stem("quickly").toString()); - Assertions.assertEquals("look", stemmer.stem("looked").toString()); - // unknown vocabulary degrades to identity - Assertions.assertEquals("zyzzyvax", stemmer.stem("zyzzyvax").toString()); - } - - /** - * Checks everyday German inflections against {@code de_DE_frami}: a plural, an - * umlauted plural, and a superlative. - * - * @throws IOException Thrown if a present dictionary pair fails to load. - */ - @Test - void testGermanInflections() throws IOException { - final HunspellStemmer stemmer = loadOrSkip("de_DE_frami"); - Assertions.assertEquals("Kind", stemmer.stem("Kinder").toString()); - // Haeuser, written with a-umlaut, stems to Haus - Assertions.assertEquals("Haus", stemmer.stem("H\u00E4user").toString()); - Assertions.assertEquals("schnell", stemmer.stem("schnellsten").toString()); - } - - /** - * Checks that ordinary German compounds decompose against {@code de_DE_frami}. Only - * the part count is asserted: the exact part spellings follow the dictionary's own - * entries and may shift between its revisions. - * - * @throws IOException Thrown if a present dictionary pair fails to load. - */ - @Test - void testGermanCompoundsDecompose() throws IOException { - final HunspellStemmer stemmer = loadOrSkip("de_DE_frami"); - // Haustuer, written with u-umlaut, is Haus + Tuer - Assertions.assertTrue(stemmer.stemAll("Haust\u00FCr").size() >= 2); - Assertions.assertTrue(stemmer.stemAll("Kinderzimmer").size() >= 2); - Assertions.assertTrue(stemmer.stemAll("Abbildungsverzeichnis").size() >= 2); - } - - /** - * Checks everyday Hungarian inflections against {@code hu_HU}: a plural and two - * case-suffixed forms. - * - * @throws IOException Thrown if a present dictionary pair fails to load. - */ - @Test - void testHungarianInflections() throws IOException { - final HunspellStemmer stemmer = loadOrSkip("hu_HU"); - // kutyak, written with a-acute, is the plural of kutya - Assertions.assertEquals("kutya", stemmer.stem("kuty\u00E1k").toString()); - Assertions.assertEquals("asztal", stemmer.stem("asztalon").toString()); - // konyveket, written with o-umlaut, is an inflected form of konyv - Assertions.assertEquals("k\u00F6nyv", stemmer.stem("k\u00F6nyveket").toString()); - } -} diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java index 312d785483..8d2a35170c 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java @@ -1182,8 +1182,9 @@ void testAffixWithoutPermitFlagStaysAtTheEdge() throws IOException { } /** - * Verifies CHECKCOMPOUNDDUP: a part must not repeat its left neighbor, while the - * same dictionary without the declaration accepts the repetition. + * Verifies CHECKCOMPOUNDDUP: the closing part must not repeat the part before it, + * while an earlier repetition passes, as in the reference implementation, and the + * same dictionary without the declaration accepts both. * * @throws IOException Thrown if a fixture fails to load. */ @@ -1192,10 +1193,11 @@ void testCheckCompoundDupForbidsRepeatedParts() throws IOException { final String words = "2\ndog/Z\nhouse/Z\n"; final HunspellStemmer checked = new HunspellStemmer(load( "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\nCHECKCOMPOUNDDUP\n", words)); - Assertions.assertEquals(List.of("dogdoghouse"), checked.stemAll("dogdoghouse")); + Assertions.assertEquals(List.of("doghousehouse"), checked.stemAll("doghousehouse")); + Assertions.assertEquals(List.of("dog", "house"), checked.stemAll("dogdoghouse")); final HunspellStemmer unchecked = new HunspellStemmer(load( "COMPOUNDFLAG Z\nCOMPOUNDMIN 3\n", words)); - Assertions.assertEquals(List.of("dog", "house"), unchecked.stemAll("dogdoghouse")); + Assertions.assertEquals(List.of("dog", "house"), unchecked.stemAll("doghousehouse")); } /** @@ -1249,38 +1251,27 @@ void testForbiddenEntryBlocksItsDecomposition() throws IOException { } /** - * Verifies that directives outside the affix-stemming subset do not prevent use of - * the rules this implementation supports. + * Checks supported rules when partial loading skips unsupported directives. * * @param line The affix file line. */ @ParameterizedTest @ValueSource(strings = { - "ICONV 1", - "OCONV 1", - "COMPLEXPREFIXES", - "COMPOUNDRULE 1", - "COMPOUNDMORESUFFIXES", - "COMPOUNDROOT R", - "CHECKCOMPOUNDREP", - "SIMPLIFIEDTRIPLE", - "CHECKCOMPOUNDPATTERN 1", - "FORCEUCASE U", - "COMPOUNDSYLLABLE 6 aeiou", - "SYLLABLENUM ABC", - "LANG tr", - "CHECKSHARPS", - "BREAK 1", - "FORBIDWARN", - "IGNORE x", - "KEEPCASE k" + "UNSUPPORTED_SYLLABLES ABC", + "UNSUPPORTED_LEMMA L", + "UNSUPPORTED value" }) void testUnsupportedDirectiveDoesNotBlockSupportedRules(String line) throws IOException { - final HunspellStemmer stemmer = new HunspellStemmer(load( - line + "\nSFX A Y 1\nSFX A 0 s .\n", "1\ndog/A\n")); + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream((line + "\nSFX A Y 1\nSFX A 0 s .\n") + .getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream("1\ndog/A\n".getBytes(StandardCharsets.UTF_8)), + HunspellDictionary.LoadMode.ALLOW_PARTIAL); + final HunspellStemmer stemmer = new HunspellStemmer(dictionary); Assertions.assertEquals("dog", stemmer.stem("dogs").toString()); + Assertions.assertEquals(1, dictionary.getUnsupportedDirectives().size()); } /** @@ -1493,13 +1484,19 @@ void testCrossProductSupportsTwofoldSuffixes() throws IOException { Assertions.assertEquals("foo", stemmer.stem("unfoosbar").toString()); } - /** Verifies that an unrecognized directive does not block supported affix rules. */ + /** Checks supported rules when partial loading skips an unknown directive. */ @Test void testUnknownAffixDirectiveIsSkipped() throws IOException { - final HunspellStemmer stemmer = new HunspellStemmer(load( - "UNRECOGNIZED value\nSFX A Y 1\nSFX A 0 s .\n", "1\ndog/A\n")); + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream("UNRECOGNIZED value\nSFX A Y 1\nSFX A 0 s .\n" + .getBytes(StandardCharsets.UTF_8)), + new ByteArrayInputStream("1\ndog/A\n".getBytes(StandardCharsets.UTF_8)), + HunspellDictionary.LoadMode.ALLOW_PARTIAL); + final HunspellStemmer stemmer = new HunspellStemmer(dictionary); Assertions.assertEquals("dog", stemmer.stem("dogs").toString()); + Assertions.assertEquals("UNRECOGNIZED", + dictionary.getUnsupportedDirectives().get(0).directive()); } /** Verifies validation of the {@code AF} count line. */ @@ -1535,12 +1532,12 @@ void testAliasTableCountIsValidated(String fixture) { * @param flag The invalid numeric flag. */ @ParameterizedTest - @ValueSource(strings = {"-1", "0", "65001"}) + @ValueSource(strings = {"-1", "0", "65536"}) void testNumericFlagOutsideRangeIsRejected(String flag) { final IOException e = Assertions.assertThrows(IOException.class, () -> load("FLAG num\n", "1\nword/" + flag + "\n")); - Assertions.assertEquals("numeric flag outside 1..65000 at line 2: " + flag, + Assertions.assertEquals("numeric flag outside 1..65535 at line 2: " + flag, e.getMessage()); } @@ -1697,18 +1694,23 @@ void testForbiddenSurfaceOverridesAffixAnalysis() throws IOException { } /** - * Verifies that a forbidden homonym blocks affix analysis even when another entry - * for the same surface is valid as a standalone entry. + * Verifies that the first listed homonym decides whether a surface form is + * forbidden, as in the reference implementation: a forbidden first homonym blocks the + * standalone and affix analyses, while a valid first homonym keeps both. * * @throws IOException Thrown if the fixture fails to load. */ @Test - void testForbiddenHomonymOverridesStandaloneEntry() throws IOException { - final HunspellStemmer stemmer = new HunspellStemmer(load( + void testForbiddenFirstHomonymOverridesStandaloneEntry() throws IOException { + final HunspellStemmer forbiddenFirst = new HunspellStemmer(load( + "FORBIDDENWORD X\nSFX A Y 1\nSFX A 0 s .\n", + "3\nfoo/A\nfoos/X\nfoos\n")); + Assertions.assertEquals(List.of("foos"), forbiddenFirst.stemAll("foos")); + Assertions.assertEquals(List.of(), forbiddenFirst.analyze("foos")); + final HunspellStemmer validFirst = new HunspellStemmer(load( "FORBIDDENWORD X\nSFX A Y 1\nSFX A 0 s .\n", "3\nfoo/A\nfoos\nfoos/X\n")); - - Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos")); + Assertions.assertEquals(List.of("foos", "foo"), validFirst.stemAll("foos")); } /** diff --git a/opennlp-docs/src/docbkx/stemmer.xml b/opennlp-docs/src/docbkx/stemmer.xml index d2367684b5..0b8601dc18 100644 --- a/opennlp-docs/src/docbkx/stemmer.xml +++ b/opennlp-docs/src/docbkx/stemmer.xml @@ -73,11 +73,12 @@ new CachingStemmer(factory).stem("running"); // "run"]]>
Hunspell dictionaries - opennlp.tools.stemmer.hunspell implements the documented - Hunspell dictionary format: a user-supplied + opennlp.tools.stemmer.hunspell loads a user-supplied .aff affix file and its .dic word list. OpenNLP bundles no dictionary data; dictionaries are downloaded separately, and - each states its own license. The dictionary is immutable and safe to share; + each provides license information. Retain the dictionary's copyright notices and + license text with downloaded files. OpenNLP's Apache License does not + relicense those files. The dictionary is immutable and safe to share; HunspellStemmerFactory creates a fresh stemmer per call. HunspellManualExampleTest asserts the behavior shown here. -Dopennlp.download.remote=true, and fetches through the digest-verified ResourceInstaller path. A file that already exists in the target is not replaced. Remove old files before refreshing a dictionary. - Directives outside the supported affix-stemming subset are skipped, so - published dictionaries can still use their supported rules. Conversion, - suggestion, and advanced compound behavior from skipped directives is not - applied to the returned stems. - This includes ICONV, OCONV, - COMPLEXPREFIXES, COMPOUNDRULE, - IGNORE, and KEEPCASE. Results can differ from - Hunspell for words that need these rules. + Loading uses HunspellDictionary.LoadMode.STRICT by default. + Unsupported affix directives cause an IOException with the + directive name and source line. File-based loading also includes the path. + Unknown directive names are rejected. Metadata and suggestion settings + that do not affect stemming or analysis are ignored. + Applications can select ALLOW_PARTIAL to use the supported + rules and inspect skipped directives: + + + The diagnostic list is immutable and contains the first source location + for each unsupported directive, in file order. Partial loading does not + apply behavior from skipped directives. Strict affix loading does not + establish complete Hunspell compatibility. + ICONV and OCONV convert input and output; + IGNORE removes configured characters. KEEPCASE, + CHECKSHARPS, and language-specific lowercase mappings govern + case variants. COMPLEXPREFIXES permits 2 prefixes and 1 + suffix instead of 1 prefix and 2 suffixes. Compound rules, boundary + checks, simplified junctions, and recursive BREAK separators + return recognized part stems in order, without duplicates. + Morphological st: fields supply explicit stems, + ds: suffixes make the derived form the stem, and sp: + fields prepend surface material. AM expands morphology aliases. + Rules that add and strip no material are undone like any other rule. The + first listed homonym decides whether a spelling is forbidden, and a + forbidden direct or affixed reading also blocks compound and + BREAK readings. + SYLLABLENUM applies Hungarian compound syllable adjustments. + The deprecated LEMMA_PRESENT directive is accepted without + changing results, consistent with the pinned native reference. + + + HunspellStemmer.analyze returns an immutable list of distinct + morphological analyses. Each analysis contains space-separated fields + from the dictionary entry and applied affixes, in the field order of the + native reference. A missing st: field defaults to the entry + text. A suffix without morphological fields contributes fl: + followed by the flag after the entry fields; a prefix without them + contributes its affix text before the stem, such as + un st:done fl:U. Compound components begin with + pa:. Analysis preserves field text without + OCONV output conversion. Unknown input returns an empty list: + + + These examples use the miniature dictionary above. OpenNLP does not + generate inflected forms or spelling suggestions. + + + Native Hunspell's spelling, stemming, and analysis operations can return + different results from one another. OpenNLP has one engine, so it follows + the spell checker for recognition and the analyzer for stems and fields + wherever the two agree, and resolves each disagreement the same way for + every caller. The results deviate from the native library in the + following cases. HunspellCompatibilityEval in + opennlp-eval-tests records each expected difference together + with the reference output. + + + + + Compound stems are the stems of the recognized parts, in order. The + native library concatenates the parts and stems the last one, so + riverboats gives river and + boat here and riverboat natively. To obtain + the native form, join the pa: surfaces of every part but + the last from analyze and append the last part's + st: value. + + + + + Purely numeric tokens such as 1.5 or 42-42 + are accepted by the native spell checker before any dictionary lookup. + Here they are unknown input and are returned unchanged. Recognize + numbers in the tokenizer or before stemming. + + + + + A word starting with a dotted capital I is lowercased with Unicode + mapping outside the Turkic languages, so İmply is + recognized when imply is listed. The native checker + never tries the lowercase form of such a word without a Turkic + LANG. Declare LANG tr, az, or + crh for Turkic data, where both implementations keep the + dotted and dotless letters apart. + + + + + A compound of three or more parts that is one edit away from a listed + word, such as foobarbaz next to goobarbaz, is + rejected natively by running the spelling suggester, which OpenNLP + does not provide. Such compounds are recognized here. An application + that needs the rejection can validate multi-part compounds with a + spelling checker, or limit COMPOUNDWORDMAX in the affix + file where the language allows. + + + + + Analyses keep the dictionary field text. The native analyzer applies + OCONV to the whole analysis string and, under + COMPLEXPREFIXES, returns field text reversed. Use + stemAll for converted stems. + + + + + Where the native spell checker and analyzer disagree, recognition + follows the spell checker and output follows the analyzer. This + affects all-uppercase Turkish words such as İPEK, which + the native checker rejects while its analyzer stems them; mixed-case + words with a capitalized initial such as NoLinda and + BREAK forms, which the native analyzer does not handle; + and forbidden affixed compounds, which the native analyzer stems. In + each case OpenNLP recognizes the word when the native checker does and + returns the analyzer's stems. + + + + + An affix file without SET is read as UTF-8 and rejected + when it is not valid UTF-8; the native default is ISO-8859-1. Add + SET ISO8859-1 to such a file. + + + + + Malformed declarations are rejected where the native parser guesses: + a multi-character flag under the default flag mode, an AF + or AM table whose entry count does not match its header, + and a numeric flag outside 1 through 65535. Correct the declaration; + ALLOW_PARTIAL skips unsupported directives, not malformed + ones. + + + + FLAG and AF declarations apply to the complete affix file, including rules listed before those declarations. Parsing rejects malformed text in parsed rules, invalid counts, numeric flags - outside the range 1 through 65000, and COMPOUNDMIN values + outside the range 1 through 65535, and COMPOUNDMIN values that cannot be doubled safely. Comments and unused metadata may retain a legacy encoding. Default and long flag modes preserve raw one-byte flag values in UTF-8 files. Compound length and boundary checks @@ -119,6 +261,14 @@ stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]> declares FULLSTRIP, as in Hunspell itself. Each affix or dictionary stream is rejected when it exceeds HunspellDictionary.MAX_STREAM_BYTES (64 MiB). + Search permits at most 64 compound components, 2048 compound candidate + checks per case variant, and 2048 word-break attempts per input. Sharp-s + expansion permits at most 64 case variants. Output is limited to 2048 + distinct results. Compound-rule patterns permit 4096 flag elements. + Simplified triple letters can be restored at multiple junctions; + CHECKCOMPOUNDPATTERN replacement applies at one junction. + These limits can exclude valid analyses; candidates must pass validation + before inclusion in the result.
diff --git a/opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellCompatibilityEval.java b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellCompatibilityEval.java new file mode 100644 index 0000000000..78f17e3230 --- /dev/null +++ b/opennlp-eval-tests/src/test/java/opennlp/tools/eval/HunspellCompatibilityEval.java @@ -0,0 +1,530 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.eval; + +import java.io.File; +import java.math.BigInteger; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.Callable; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; +import org.junit.jupiter.api.TestReporter; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; + +import opennlp.tools.stemmer.hunspell.HunspellDictionary; +import opennlp.tools.stemmer.hunspell.HunspellStemmer; + +/** + * Evaluates the Hunspell stemmer with the LibreOffice English, German, and Hungarian + * dictionaries under the {@code hunspell} directory of {@code OPENNLP_DATA_DIR}: strict + * loading, expected inflections and compounds, concurrent use, and agreement with the + * stems, analyses, and recognition recorded from the reference implementation as + * described in {@code dev/README-hunspell-dictionaries.md}. + */ +@TestInstance(TestInstance.Lifecycle.PER_CLASS) +public class HunspellCompatibilityEval extends AbstractEvalTest { + + private static final String DATA_DIRECTORY = "hunspell"; + private static final int THREADS = 4; + private static final int REPETITIONS = 10; + private static final int TIMEOUT_SECONDS = 60; + private static final String UNKNOWN = "zyzzyvax"; + + private final Map stemmers = new EnumMap<>(Dictionary.class); + + /** The external dictionaries, their MD5 digests, and the evaluated inputs. */ + private enum Dictionary { + ENGLISH("en_US", "bbb118ea006c22ebe9ef7dbfe0dbfc2a", "7e671db5244b0496f9888e9f0176c360", + List.of("workers", "cats", "unhappiest", "quickly", "looked", "reading", + "dogs", "books", "walked", "walking", "talked", "talking", "played", + "playing", "helped", "helping", "houses", "children", "feet", "better", + "Workers", "WORKERS", "cAtS", "worker's", "well-known", "unhappy", "undone", UNKNOWN)), + GERMAN("de_DE_frami", "9fd6eb96145bdccb2dc0213e1df5a46e", "07c5fd0780eca6ba448cba8c7be170cd", + List.of("gegangen", "Kinder", "Häuser", "schnellsten", "Freunden", "Vorschläge", + "Haustür", "Kinderzimmer", "Abbildungsverzeichnis", "Haus", "Baum", "Buch", "schnell", UNKNOWN)), + HUNGARIAN("hu_HU", "c13482742489921ce2d8d19ff5122986", "66bab5478e829ba5e33afcde24a7259c", + List.of("kutyák", "asztalon", "könyveket", "házak", "emberek", "kutyáknak", UNKNOWN)); + + private final String id; + private final BigInteger affixChecksum; + private final BigInteger dictionaryChecksum; + private final List inputs; + + /** + * Describes one external dictionary. + * + * @param id The file name without suffix. + * @param affixChecksum The MD5 digest of the affix file, in hexadecimal. + * @param dictionaryChecksum The MD5 digest of the word list, in hexadecimal. + * @param inputs The evaluated inputs. + */ + Dictionary(String id, String affixChecksum, String dictionaryChecksum, List inputs) { + this.id = id; + this.affixChecksum = new BigInteger(affixChecksum, 16); + this.dictionaryChecksum = new BigInteger(dictionaryChecksum, 16); + this.inputs = inputs; + } + } + + /** + * The distinct stems and analyses of one input. + * + * @param stems The stems. + * @param analyses The morphological analyses. + */ + private record Result(Set stems, Set analyses) { } + + /** + * The reference implementation's recorded outcome for one input. + * + * @param accepted Whether the reference spell checker accepted the input. + * @param stems The reference stems. + * @param analyses The reference analyses with separator whitespace normalized. + */ + private record Recorded(boolean accepted, Set stems, Set analyses) { + + /** {@return the recorded stems and analyses as a result} */ + Result result() { + return new Result(stems, analyses); + } + } + + /** The classification of one comparison. */ + private enum Outcome { + EXACT, EXPECTED_DIFFERENCE, IDENTITY_FALLBACK, UNEXPECTED + } + + /** Checks the comparison classification on synthetic results. */ + @Test + void comparisonValidation() { + final Result complete = new Result(Set.of("card"), Set.of("st:card")); + final Result empty = new Result(Set.of(), Set.of()); + final Result identity = new Result(Set.of(UNKNOWN), Set.of()); + final Recorded accepted = new Recorded(true, complete.stems(), complete.analyses()); + final Recorded acceptedEmpty = new Recorded(true, Set.of(), Set.of()); + final Recorded rejected = new Recorded(false, Set.of(), Set.of()); + Assertions.assertAll( + () -> Assertions.assertEquals(Outcome.EXACT, classify("card", accepted, complete, null)), + () -> Assertions.assertEquals(Outcome.EXPECTED_DIFFERENCE, + classify("card", acceptedEmpty, complete, complete)), + () -> Assertions.assertEquals(Outcome.IDENTITY_FALLBACK, classify(UNKNOWN, rejected, identity, null)), + () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", accepted, empty, null)), + () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", acceptedEmpty, complete, null)), + () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", rejected, complete, null)), + () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", accepted, complete, complete)), + () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify("card", acceptedEmpty, empty, complete)), + () -> Assertions.assertEquals(Outcome.UNEXPECTED, classify(UNKNOWN, accepted, identity, null))); + } + + /** + * Loads each dictionary strictly and confirms that partial loading skips nothing. + * + * @param dictionary The external dictionary. + * @throws Exception If a file is missing, changed, or malformed. + */ + @ParameterizedTest + @EnumSource(Dictionary.class) + void strictLoading(Dictionary dictionary) throws Exception { + stemmer(dictionary); + final HunspellDictionary partial = HunspellDictionary.load( + file(dictionary, HunspellDictionary.AFFIX_FILE_SUFFIX), + file(dictionary, HunspellDictionary.DICTIONARY_FILE_SUFFIX), + HunspellDictionary.LoadMode.ALLOW_PARTIAL); + Assertions.assertTrue(partial.getUnsupportedDirectives().isEmpty()); + } + + /** + * Checks expected stems and compound decompositions. + * + * @param dictionary The external dictionary. + * @throws Exception If loading fails. + */ + @ParameterizedTest + @EnumSource(Dictionary.class) + void expectedInflections(Dictionary dictionary) throws Exception { + final HunspellStemmer stemmer = stemmer(dictionary); + final Map expected = switch (dictionary) { + case ENGLISH -> Map.of("workers", "worker", "cats", "cat", "unhappiest", "unhappy", + "quickly", "quick", "looked", "look"); + case GERMAN -> Map.of("Kinder", "Kind", "Häuser", "Haus", "schnellsten", "schnell"); + case HUNGARIAN -> Map.of("kutyák", "kutya", "asztalon", "asztal", "könyveket", "könyv"); + }; + expected.forEach((word, stem) -> Assertions.assertEquals(stem, + stemmer.stem(word).toString(), word)); + Assertions.assertEquals(List.of(UNKNOWN), stemmer.stemAll(UNKNOWN)); + Assertions.assertTrue(stemmer.analyze(UNKNOWN).isEmpty()); + if (dictionary == Dictionary.GERMAN) { + for (String word : List.of("Haustür", "Kinderzimmer", "Abbildungsverzeichnis")) { + Assertions.assertTrue(stemmer.stemAll(word).size() >= 2, word); + Assertions.assertTrue(stemmer.analyze(word).stream() + .anyMatch(analysis -> analysis.startsWith("pa:")), word); + } + } + } + + /** + * Compares stems, analyses, and recognition with the recorded reference results and + * fails on any result that is neither exact, an expected difference, nor an identity + * fallback for an input the reference rejects. + * + * @param dictionary The external dictionary. + * @param reporter The test reporter for the counts. + * @throws Exception If loading fails. + */ + @ParameterizedTest + @EnumSource(Dictionary.class) + void referenceCompatibility(Dictionary dictionary, TestReporter reporter) throws Exception { + final HunspellStemmer stemmer = stemmer(dictionary); + final Map recorded = recorded(dictionary); + int exact = 0; + int differences = 0; + int fallback = 0; + final List failures = new ArrayList<>(); + for (String word : dictionary.inputs) { + final Recorded reference = recorded.get(word); + Assertions.assertNotNull(reference, "no recorded reference result for " + word); + final Result javaResult = result(stemmer, word); + switch (classify(word, reference, javaResult, expectedDifference(dictionary, word))) { + case EXACT -> exact++; + case EXPECTED_DIFFERENCE -> differences++; + case IDENTITY_FALLBACK -> fallback++; + case UNEXPECTED -> failures.add(word + ": reference=" + reference + ", OpenNLP=" + javaResult); + } + } + final String summary = "inputs=" + dictionary.inputs.size() + ", exact=" + exact + + ", expectedDifferences=" + differences + ", identityFallback=" + fallback + + ", unexpected=" + failures.size(); + reporter.publishEntry(dictionary.id, summary); + System.out.println(dictionary.id + ": " + summary); + Assertions.assertTrue(failures.isEmpty(), () -> String.join("\n", failures)); + } + + /** + * Repeats the evaluated inputs from several threads on one shared stemmer and + * compares each result with the single-threaded result. + * + * @param dictionary The external dictionary. + * @throws Exception If loading fails or a task fails. + */ + @ParameterizedTest + @EnumSource(Dictionary.class) + void concurrentAnalysis(Dictionary dictionary) throws Exception { + final HunspellStemmer stemmer = stemmer(dictionary); + final Map expected = new LinkedHashMap<>(); + dictionary.inputs.forEach(word -> expected.put(word, result(stemmer, word))); + final List> tasks = new ArrayList<>(); + for (int thread = 0; thread < THREADS; thread++) { + final int offset = thread; + tasks.add(() -> { + for (int repeat = 0; repeat < REPETITIONS; repeat++) { + for (int index = 0; index < dictionary.inputs.size(); index++) { + final String word = dictionary.inputs.get((index + offset + repeat) % dictionary.inputs.size()); + Assertions.assertEquals(expected.get(word), result(stemmer, word), word); + } + } + return null; + }); + } + try (var executor = Executors.newFixedThreadPool(THREADS)) { + for (var future : executor.invokeAll(tasks, TIMEOUT_SECONDS, TimeUnit.SECONDS)) { + Assertions.assertFalse(future.isCancelled(), "concurrent analysis timed out"); + future.get(); + } + } + } + + /** + * Loads a dictionary once after verifying its digests. + * + * @param dictionary The external dictionary. + * @return The shared stemmer. + * @throws Exception If a file is missing, changed, or malformed. + */ + private HunspellStemmer stemmer(Dictionary dictionary) throws Exception { + if (!stemmers.containsKey(dictionary)) { + final Path affix = file(dictionary, HunspellDictionary.AFFIX_FILE_SUFFIX); + final Path words = file(dictionary, HunspellDictionary.DICTIONARY_FILE_SUFFIX); + verifyFileChecksum(affix, dictionary.affixChecksum); + verifyFileChecksum(words, dictionary.dictionaryChecksum); + final HunspellDictionary loaded = HunspellDictionary.load(affix, words); + Assertions.assertTrue(loaded.getUnsupportedDirectives().isEmpty()); + stemmers.put(dictionary, new HunspellStemmer(loaded)); + } + return stemmers.get(dictionary); + } + + /** + * Locates a dictionary file under the evaluation data directory. + * + * @param dictionary The external dictionary. + * @param suffix The file suffix. + * @return The file path. + * @throws Exception If the data directory is not configured or does not exist. + */ + private Path file(Dictionary dictionary, String suffix) throws Exception { + return new File(getOpennlpDataDir(), DATA_DIRECTORY + File.separator + dictionary.id + suffix).toPath(); + } + + /** + * Collects the distinct stems and analyses of one input. + * + * @param stemmer The stemmer. + * @param word The input. + * @return The result. + */ + private Result result(HunspellStemmer stemmer, String word) { + return new Result(new LinkedHashSet<>(stemmer.stemAll(word).stream() + .map(CharSequence::toString).toList()), new LinkedHashSet<>(stemmer.analyze(word))); + } + + /** + * Classifies one comparison. An expected difference must match the recorded + * OpenNLP output exactly and must still differ from the reference, so a stale entry + * is reported. + * + * @param word The input. + * @param reference The recorded reference outcome. + * @param javaResult The OpenNLP result. + * @param expected The recorded OpenNLP output for a known difference, or {@code null}. + * @return The classification. + */ + private Outcome classify(String word, Recorded reference, Result javaResult, Result expected) { + if (expected != null) { + return reference.accepted() && !reference.result().equals(javaResult) + && expected.equals(javaResult) ? Outcome.EXPECTED_DIFFERENCE : Outcome.UNEXPECTED; + } + if (reference.accepted() && reference.result().equals(javaResult)) { + return Outcome.EXACT; + } + if (!reference.accepted() && reference.stems().isEmpty() && reference.analyses().isEmpty() + && javaResult.stems().equals(Set.of(word)) && javaResult.analyses().isEmpty()) { + return Outcome.IDENTITY_FALLBACK; + } + return Outcome.UNEXPECTED; + } + + /** + * Specifies the OpenNLP output for inputs whose recorded reference output differs: + * compound and break parts against concatenated or missing reference stems, and + * the reference analyzer's lowercase readings of capitalized German nouns. + * + * @param dictionary The external dictionary. + * @param word The input. + * @return The expected OpenNLP output, or {@code null} for an exact comparison. + */ + private Result expectedDifference(Dictionary dictionary, String word) { + if (dictionary == Dictionary.ENGLISH && word.equals("well-known")) { + return new Result(Set.of("well", "known"), Set.of("pa:well st:well pa:known st:known")); + } + if (dictionary != Dictionary.GERMAN) { + return null; + } + return switch (word) { + case "Kinder" -> new Result(Set.of("Kind"), Set.of("st:Kind fl:R")); + case "Häuser" -> new Result(Set.of("Haus"), Set.of("st:Haus fl:p")); + case "schnellsten" -> new Result(Set.of("schnell"), Set.of("st:schnell fl:C")); + case "Freunden" -> new Result(Set.of("freunden", "Freund"), Set.of("st:freunden", "st:Freund fl:P")); + case "Vorschläge" -> new Result(Set.of("Vor", "schlag"), + Set.of("pa:Vor st:Vor fl:j pa:schläge st:schlag fl:p")); + case "Haustür" -> new Result(Set.of("Haus", "tür"), Set.of("pa:Haus st:Haus fl:j pa:tür")); + case "Kinderzimmer" -> new Result(Set.of("Kinder", "zimmer"), + Set.of("pa:Kinder st:Kinder fl:j pa:zimmer")); + case "Abbildungsverzeichnis" -> new Result(Set.of("Abbildungs", "verzeichnis"), + Set.of("pa:Abbildungs st:Abbildungs fl:j pa:verzeichnis")); + case "Haus" -> new Result(Set.of("Haus"), Set.of("st:Haus")); + case "Baum" -> new Result(Set.of("Baum"), Set.of("st:Baum")); + case "Buch" -> new Result(Set.of("Buch"), Set.of("st:Buch")); + case "schnell" -> new Result(Set.of("schnell"), Set.of("st:schnell")); + default -> null; + }; + } + + /** + * The results recorded from the reference implementation for the evaluated inputs, + * keyed by input. + * + * @param dictionary The external dictionary. + * @return The recorded outcomes. + */ + private static Map recorded(Dictionary dictionary) { + return switch (dictionary) { + case ENGLISH -> Map.ofEntries( + Map.entry("workers", new Recorded(true, + Set.of("worker"), + Set.of("st:worker fl:S"))), + Map.entry("cats", new Recorded(true, + Set.of("cat"), + Set.of("st:cat fl:S"))), + Map.entry("unhappiest", new Recorded(true, + Set.of("unhappy"), + Set.of("st:unhappy fl:T"))), + Map.entry("quickly", new Recorded(true, + Set.of("quick"), + Set.of("st:quick fl:Y"))), + Map.entry("looked", new Recorded(true, + Set.of("look"), + Set.of("st:look fl:D"))), + Map.entry("reading", new Recorded(true, + Set.of("reading", "read"), + Set.of("st:reading", "st:read fl:G"))), + Map.entry("dogs", new Recorded(true, + Set.of("dog"), + Set.of("st:dog fl:S"))), + Map.entry("books", new Recorded(true, + Set.of("book"), + Set.of("st:book fl:S"))), + Map.entry("walked", new Recorded(true, + Set.of("walk"), + Set.of("st:walk fl:D"))), + Map.entry("walking", new Recorded(true, + Set.of("walking", "walk"), + Set.of("st:walking", "st:walk fl:G"))), + Map.entry("talked", new Recorded(true, + Set.of("talk"), + Set.of("st:talk fl:D"))), + Map.entry("talking", new Recorded(true, + Set.of("talk"), + Set.of("st:talk fl:G"))), + Map.entry("played", new Recorded(true, + Set.of("play"), + Set.of("st:play fl:D"))), + Map.entry("playing", new Recorded(true, + Set.of("play"), + Set.of("st:play fl:G"))), + Map.entry("helped", new Recorded(true, + Set.of("help"), + Set.of("st:help fl:D"))), + Map.entry("helping", new Recorded(true, + Set.of("helping", "help"), + Set.of("st:helping", "st:help fl:G"))), + Map.entry("houses", new Recorded(true, + Set.of("house"), + Set.of("st:house fl:S"))), + Map.entry("children", new Recorded(true, + Set.of("children"), + Set.of("st:children"))), + Map.entry("feet", new Recorded(true, + Set.of("feet"), + Set.of("st:feet"))), + Map.entry("better", new Recorded(true, + Set.of("better"), + Set.of("st:better"))), + Map.entry("Workers", new Recorded(true, + Set.of("worker"), + Set.of("st:worker fl:S"))), + Map.entry("WORKERS", new Recorded(true, + Set.of("worker"), + Set.of("st:worker fl:S"))), + Map.entry("cAtS", new Recorded(false, + Set.of(), + Set.of())), + Map.entry("worker's", new Recorded(true, + Set.of("worker"), + Set.of("st:worker fl:M"))), + Map.entry("well-known", new Recorded(true, + Set.of(), + Set.of())), + Map.entry("unhappy", new Recorded(true, + Set.of("unhappy", "happy"), + Set.of("st:unhappy", "un st:happy fl:U"))), + Map.entry("undone", new Recorded(true, + Set.of("done"), + Set.of("un st:done fl:U"))), + Map.entry("zyzzyvax", new Recorded(false, + Set.of(), + Set.of()))); + case GERMAN -> Map.ofEntries( + Map.entry("gegangen", new Recorded(true, + Set.of("gegangen"), + Set.of("st:gegangen"))), + Map.entry("Kinder", new Recorded(true, + Set.of("kinder", "kind", "Kind"), + Set.of("st:kinder fl:k", "st:kind fl:R", "st:Kind fl:R"))), + Map.entry("Häuser", new Recorded(true, + Set.of("häuser", "haus", "Haus"), + Set.of("st:häuser fl:k", "st:haus fl:p", "st:Haus fl:p"))), + Map.entry("schnellsten", new Recorded(true, + Set.of("schnell"), + Set.of("fl:k st:schnell fl:C", "st:schnell fl:C"))), + Map.entry("Freunden", new Recorded(true, + Set.of("freunden", "freund", "Freund"), + Set.of("st:freunden", "st:freund fl:P", "st:Freund fl:P"))), + Map.entry("Vorschläge", new Recorded(true, + Set.of(), + Set.of())), + Map.entry("Haustür", new Recorded(true, + Set.of(), + Set.of("pa:tür"))), + Map.entry("Kinderzimmer", new Recorded(true, + Set.of(), + Set.of("pa:zimmer"))), + Map.entry("Abbildungsverzeichnis", new Recorded(true, + Set.of(), + Set.of("pa:verzeichnis"))), + Map.entry("Haus", new Recorded(true, + Set.of("haus", "Haus"), + Set.of("st:haus fl:k", "st:Haus"))), + Map.entry("Baum", new Recorded(true, + Set.of("baum", "Baum"), + Set.of("st:baum fl:k", "st:Baum"))), + Map.entry("Buch", new Recorded(true, + Set.of("buch", "Buch"), + Set.of("st:buch fl:k", "st:Buch"))), + Map.entry("schnell", new Recorded(true, + Set.of("schnell"), + Set.of("st:schnell", "st:schnell fl:k"))), + Map.entry("zyzzyvax", new Recorded(false, + Set.of(), + Set.of()))); + case HUNGARIAN -> Map.ofEntries( + Map.entry("kutyák", new Recorded(true, + Set.of("kutya"), + Set.of("st:kutya po:noun ts:NOM is:PLUR is:NOM"))), + Map.entry("asztalon", new Recorded(true, + Set.of("asztal"), + Set.of("st:asztal po:noun ts:NOM is:SUE"))), + Map.entry("könyveket", new Recorded(true, + Set.of("könyv"), + Set.of("st:könyv po:noun ts:NOM is:PLUR is:ACC"))), + Map.entry("házak", new Recorded(true, + Set.of("ház"), + Set.of("st:ház po:noun ts:PLUR ts:NOM"))), + Map.entry("emberek", new Recorded(true, + Set.of("ember"), + Set.of("st:ember po:noun ts:NOM is:PLUR is:NOM"))), + Map.entry("kutyáknak", new Recorded(true, + Set.of("kutya"), + Set.of("st:kutya po:noun ts:NOM is:PLUR is:DAT"))), + Map.entry("zyzzyvax", new Recorded(false, + Set.of(), + Set.of()))); + }; + } +}