diff --git a/dev/README-hunspell-dictionaries.md b/dev/README-hunspell-dictionaries.md index 8723a4164a..711435d948 100644 --- a/dev/README-hunspell-dictionaries.md +++ b/dev/README-hunspell-dictionaries.md @@ -87,17 +87,22 @@ accordingly; no conversion is required. ## Testing against real dictionaries -The in-tree tests run against project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, additionally checks everyday morphology against published dictionaries when pointed at a directory of `.aff`/`.dic` pairs (each test skips when its pair is absent): +The in-tree tests use project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, also checks everyday morphology with the LibreOffice `en_US`, `de_DE_frami`, and `hu_HU` dictionaries. Point it at one directory containing all listed `.aff` and `.dic` files. A missing dictionary skips the associated test; a dictionary that cannot be loaded fails it. ``` -./mvnw test -pl opennlp-core/opennlp-runtime -Dtest=HunspellRealDictionaryTest \ +./mvnw test -pl opennlp-core/opennlp-runtime -am \ + -Dtest=HunspellRealDictionaryTest -Dsurefire.failIfNoSpecifiedTests=false \ -Dopennlp.hunspell.dict.dir=/tmp/hunspell-dicts ``` ## What the engine supports -Supported affix features include `PFX` and `SFX` rules, continuation classes, -compound flags, blocking flags, `CIRCUMFIX`, and `FULLSTRIP`. The parser rejects -directives that would change stems if ignored. It skips cosmetic tables that do not -affect stemming. Malformed files report the relevant line number. Each affix or -dictionary stream is limited to 64 MiB. +The engine applies `PFX` and `SFX` rules with strip strings and character-class conditions. It supports a prefix and suffix cross-product, a double suffix sequence connected by continuation classes, identity rules in continuation paths, file-wide `FLAG` modes, file-wide `AF` aliases, and the `SET` encoding declaration. Numeric flags range from 1 through 65000. + +Compound decomposition supports `COMPOUNDFLAG`, `COMPOUNDBEGIN`, `COMPOUNDMIDDLE`, `COMPOUNDEND`, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBIDFLAG`, `CHECKCOMPOUNDDUP`, `CHECKCOMPOUNDCASE`, and `CHECKCOMPOUNDTRIPLE`. Compound boundaries and minimum lengths use Unicode code points. `NEEDAFFIX` (also named `PSEUDOROOT`), `ONLYINCOMPOUND`, `FORBIDDENWORD`, `CIRCUMFIX`, and `FULLSTRIP` control whether an analysis is accepted. + +Other directives are skipped. Their conversion, suggestion, or advanced compound behavior is not applied by this affix stemmer. Comments and unused metadata may contain legacy-encoded bytes even when the file uses UTF-8. Parsed rules and dictionary text are decoded strictly. Default and `long` flag modes preserve raw one-byte flag values used by published UTF-8 dictionaries. Invalid rule counts, aliases, flags, and compound limits fail during loading. Each affix or dictionary stream is rejected when it exceeds `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB). + +Skipped directives include `ICONV`, `OCONV`, `COMPLEXPREFIXES`, `COMPOUNDRULE`, +`IGNORE`, and `KEEPCASE`. Loading a dictionary does not apply these rules; +results can differ from Hunspell for words that need them. diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java index ab24a68853..2bb06935d5 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionary.java @@ -17,9 +17,13 @@ package opennlp.tools.stemmer.hunspell; +import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.charset.CharacterCodingException; import java.nio.charset.Charset; +import java.nio.charset.CodingErrorAction; import java.nio.charset.IllegalCharsetNameException; import java.nio.charset.StandardCharsets; import java.nio.charset.UnsupportedCharsetException; @@ -35,34 +39,25 @@ import opennlp.tools.util.StringUtil; /** - * An immutable, in-memory Hunspell-format dictionary: the word list of a {@code .dic} - * file and the prefix and suffix rules of its {@code .aff} companion, loaded from - * user-supplied files. The engine implements the documented format directly; no - * dictionary data is bundled, dictionaries are supplied by the user. + * An immutable, in-memory Hunspell-format dictionary loaded from user-supplied + * {@code .aff} and {@code .dic} files. OpenNLP includes no dictionary data. * - *

Supported affix features: {@code PFX} and {@code SFX} rules with strip strings, - * character-class conditions, and cross-product combination of one prefix with one - * suffix; twofold suffixes through the continuation classes on suffix rules; - * {@code FLAG} modes {@code char} (default), {@code UTF-8}, {@code long}, and - * {@code num}; the {@code AF} flag alias table; the {@code SET} encoding declaration; - * compound decomposition under {@code COMPOUNDFLAG}, the positional - * {@code COMPOUNDBEGIN}/{@code COMPOUNDMIDDLE}/{@code COMPOUNDEND} flags, + *

Supported affix features are {@code PFX} and {@code SFX} rules with strip + * strings, character-class conditions, cross-product combinations, and a double suffix + * connected by continuation classes; {@code FLAG} modes {@code char}, {@code UTF-8}, + * {@code long}, and {@code num}; the {@code AF} alias table; and the {@code SET} + * encoding declaration. Compound decomposition supports {@code COMPOUNDFLAG}, + * {@code COMPOUNDBEGIN}, {@code COMPOUNDMIDDLE}, {@code COMPOUNDEND}, * {@code COMPOUNDMIN}, {@code COMPOUNDWORDMAX}, {@code COMPOUNDPERMITFLAG}, - * {@code COMPOUNDFORBIDFLAG}, and the {@code CHECKCOMPOUNDDUP}, - * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE} declarations, with - * compound parts standing on their entries alone or on an entry plus one affix; the - * blocking flags - * {@code NEEDAFFIX} (with its historical alias {@code PSEUDOROOT}), - * {@code ONLYINCOMPOUND}, and {@code FORBIDDENWORD}, which suppress analyses the - * dictionary marks as virtual stems, compound-only parts, or forbidden words; and - * {@code CIRCUMFIX}, which binds marked prefix and suffix halves to one another; and - * the {@code FULLSTRIP} declaration, without which a rule that strips a whole stem is - * not applied, matching hunspell. - * Directives that would change stems when ignored ({@code ICONV}, {@code OCONV}, - * {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE}, {@code IGNORE}, - * {@code KEEPCASE}) are rejected at load time. Cosmetic tables such as - * {@code REP}, {@code MAP}, and {@code KEY} are skipped, so analyses that would need - * them are missed rather than invented.

+ * {@code COMPOUNDFORBIDFLAG}, {@code CHECKCOMPOUNDDUP}, + * {@code CHECKCOMPOUNDCASE}, and {@code CHECKCOMPOUNDTRIPLE}. The blocking flags + * {@code NEEDAFFIX} (also named {@code PSEUDOROOT}), {@code ONLYINCOMPOUND}, and + * {@code FORBIDDENWORD}, plus {@code CIRCUMFIX} and {@code FULLSTRIP}, are also + * applied.

+ * + *

Other directives are skipped. Their spelling, conversion, suggestion, or + * advanced compound behavior is not applied by this affix stemmer. Dictionary + * morphology fields are also ignored.

* *

Instances are immutable and safe to share between threads.

* @@ -134,6 +129,12 @@ enum CompoundPosition { /** The line tag of a suffix block and of every rule line inside it. */ private static final String SUFFIX_TAG = "SFX"; + /** The directive that selects the file-wide flag representation. */ + private static final String FLAG_TAG = "FLAG"; + + /** The directive that defines the file-wide flag alias table. */ + private static final String ALIAS_TAG = "AF"; + /** Prefix used by comment lines. */ private static final String COMMENT_PREFIX = "#"; @@ -149,6 +150,16 @@ enum CompoundPosition { /** The affix format's marker for absent strip or affix material. */ private static final String NO_MATERIAL = "0"; + /** + * Largest flag value permitted by {@code FLAG num}, as specified by the + * + * Hunspell format manual. + */ + private static final int MAX_NUMERIC_FLAG = 65_000; + + /** Largest {@code COMPOUNDMIN} value that can be doubled without overflow. */ + private static final int MAX_COMPOUND_MIN = Integer.MAX_VALUE / 2; + private final Map> entries; private final BoundaryIndex suffixesByLast; private final List suffixesWithoutMaterial; @@ -319,16 +330,282 @@ public static HunspellDictionary load(InputStream affixStream, if (dictionaryStream == null) { throw new IllegalArgumentException("dictionaryStream must not be null"); } - final byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream"); + byte[] affixBytes = readBounded(affixStream, MAX_STREAM_BYTES, "affix stream"); final Charset charset = declaredCharset(affixBytes); - final AffixFile affix = parseAffix(new String(affixBytes, charset)); + maskIgnoredAffixLines(affixBytes); + final boolean rawUtf8Flags = StandardCharsets.UTF_8.equals(charset) + && !usesUnicodeOrNumericFlags(affixBytes); + affixBytes = normalizeUtf8ByteFlags(affixBytes, charset); + final AffixFile affix = parseAffix(decode(affixBytes, charset, "affix stream")); + byte[] dictionaryBytes = readBounded(dictionaryStream, MAX_STREAM_BYTES, + "dictionary stream"); + if (rawUtf8Flags) { + dictionaryBytes = normalizeDictionaryByteFlags(dictionaryBytes); + } final Map> entries = parseWordList( - new String(readBounded(dictionaryStream, MAX_STREAM_BYTES, "dictionary stream"), - charset), + decode(dictionaryBytes, charset, "dictionary stream"), affix.flagMode, affix.flagAliases); return new HunspellDictionary(entries, affix); } + /** + * Replaces comments and unused directive lines with ASCII spaces before strict + * decoding. Published dictionaries sometimes retain legacy-encoded metadata despite + * a {@code SET UTF-8} declaration. Line endings and byte positions remain unchanged, + * while malformed bytes in parsed directives are still reported. + * + * @param bytes The buffered affix file, modified in place. + */ + private static void maskIgnoredAffixLines(byte[] bytes) { + int lineStart = 0; + for (int i = 0; i <= bytes.length; i++) { + if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') { + int fieldStart = lineStart; + while (fieldStart < i && isAsciiFieldSpace(bytes[fieldStart])) { + fieldStart++; + } + int fieldEnd = fieldStart; + while (fieldEnd < i && !isAsciiFieldSpace(bytes[fieldEnd])) { + fieldEnd++; + } + if (fieldStart < fieldEnd && bytes[fieldStart] != '#') { + final String directive = new String(bytes, fieldStart, + fieldEnd - fieldStart, StandardCharsets.US_ASCII); + if (isParsedAffixDirective(directive)) { + maskInlineComment(bytes, fieldEnd, i); + lineStart = i + 1; + continue; + } + } + Arrays.fill(bytes, lineStart, i, (byte) ' '); + lineStart = i + 1; + } + } + } + + /** {@return whether a byte separates fields in an affix line} */ + private static boolean isAsciiFieldSpace(byte value) { + return value == ' ' || value == '\t' || value == '\f'; + } + + /** Replaces an inline comment that starts after an affix field separator. */ + private static void maskInlineComment(byte[] bytes, int from, int to) { + boolean fieldStart = false; + for (int i = from; i < to; i++) { + if (isAsciiFieldSpace(bytes[i])) { + fieldStart = true; + } else if (fieldStart && bytes[i] == '#') { + Arrays.fill(bytes, i, to, (byte) ' '); + return; + } else { + fieldStart = false; + } + } + } + + /** {@return whether this implementation parses a directive's fields} */ + private static boolean isParsedAffixDirective(String directive) { + return switch (directive) { + case SET_TAG, FLAG_TAG, ALIAS_TAG, PREFIX_TAG, SUFFIX_TAG, + "COMPOUNDFLAG", "COMPOUNDBEGIN", "COMPOUNDMIDDLE", "COMPOUNDEND", + "COMPOUNDPERMITFLAG", "COMPOUNDFORBIDFLAG", "NEEDAFFIX", "PSEUDOROOT", + "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX", "COMPOUNDMIN", + "COMPOUNDWORDMAX", "CHECKCOMPOUNDDUP", "CHECKCOMPOUNDCASE", + "CHECKCOMPOUNDTRIPLE", "FULLSTRIP" -> true; + default -> false; + }; + } + + /** + * Converts raw one-byte flags in a UTF-8 affix file to equivalent Unicode code + * points before decoding. Hunspell's default and {@code long} flag modes operate on + * bytes, and published UTF-8 dictionaries can therefore contain non-UTF-8 bytes in + * flag fields. Text, conditions, and affix material remain subject to strict UTF-8 + * decoding. + * + * @param bytes The affix file after unused lines have been masked. + * @param charset The encoding selected by {@code SET}. + * @return The content with raw flag bytes represented as valid UTF-8. + */ + private static byte[] normalizeUtf8ByteFlags(byte[] bytes, Charset charset) { + if (!StandardCharsets.UTF_8.equals(charset) || usesUnicodeOrNumericFlags(bytes)) { + return bytes; + } + final ByteArrayOutputStream normalized = new ByteArrayOutputStream(bytes.length); + int lineStart = 0; + for (int i = 0; i <= bytes.length; i++) { + if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') { + writeNormalizedFlagLine(normalized, bytes, lineStart, i); + if (i < bytes.length) { + normalized.write(bytes[i]); + } + lineStart = i + 1; + } + } + return normalized.toByteArray(); + } + + /** {@return whether {@code FLAG UTF-8} or {@code FLAG num} selects non-byte flags} */ + private static boolean usesUnicodeOrNumericFlags(byte[] bytes) { + final int[] starts = new int[5]; + final int[] fieldEnds = new int[5]; + int lineStart = 0; + for (int i = 0; i <= bytes.length; i++) { + if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') { + final int count = findAsciiFields(bytes, lineStart, i, starts, fieldEnds); + if (count >= 2 && FLAG_TAG.equals( + asciiField(bytes, starts[0], fieldEnds[0]))) { + final String mode = asciiField(bytes, starts[1], fieldEnds[1]); + return "UTF-8".equals(mode) || "num".equals(mode); + } + lineStart = i + 1; + } + } + return false; + } + + /** Writes one affix line, converting high bytes only within raw flag fields. */ + private static void writeNormalizedFlagLine(ByteArrayOutputStream target, + byte[] bytes, int lineStart, int lineEnd) { + final int[] starts = new int[5]; + final int[] fieldEnds = new int[5]; + final int count = findAsciiFields(bytes, lineStart, lineEnd, starts, fieldEnds); + int firstFlagStart = -1; + int firstFlagEnd = -1; + int continuationStart = -1; + int continuationEnd = -1; + if (count >= 2) { + final String directive = asciiField(bytes, starts[0], fieldEnds[0]); + if (ALIAS_TAG.equals(directive) || PREFIX_TAG.equals(directive) + || SUFFIX_TAG.equals(directive) || isSingleFlagDirective(directive)) { + firstFlagStart = starts[1]; + firstFlagEnd = fieldEnds[1]; + } + if (count >= 4 && (PREFIX_TAG.equals(directive) || SUFFIX_TAG.equals(directive))) { + for (int i = starts[3]; i < fieldEnds[3]; i++) { + if (bytes[i] == '/') { + continuationStart = i + 1; + continuationEnd = fieldEnds[3]; + break; + } + } + } + } + for (int i = lineStart; i < lineEnd; i++) { + final boolean flagByte = i >= firstFlagStart && i < firstFlagEnd + || i >= continuationStart && i < continuationEnd; + writeNormalizedByte(target, bytes[i], flagByte); + } + } + + /** + * Converts raw flag bytes after the flag separator of each dictionary entry. + * Word text and morphology fields remain subject to strict UTF-8 decoding. + * + * @param bytes The buffered dictionary file. + * @return The content with raw flag bytes represented as valid UTF-8. + */ + private static byte[] normalizeDictionaryByteFlags(byte[] bytes) { + final ByteArrayOutputStream normalized = new ByteArrayOutputStream(bytes.length); + int lineStart = 0; + for (int i = 0; i <= bytes.length; i++) { + if (i == bytes.length || bytes[i] == '\n' || bytes[i] == '\r') { + int flagStart = -1; + int flagEnd = -1; + for (int cursor = lineStart + 1; cursor < i; cursor++) { + if (bytes[cursor] == '/' && bytes[cursor - 1] != '\\') { + flagStart = cursor + 1; + flagEnd = flagStart; + while (flagEnd < i && bytes[flagEnd] != ' ' && bytes[flagEnd] != '\t') { + flagEnd++; + } + break; + } + } + for (int cursor = lineStart; cursor < i; cursor++) { + writeNormalizedByte(normalized, bytes[cursor], + cursor >= flagStart && cursor < flagEnd); + } + if (i < bytes.length) { + normalized.write(bytes[i]); + } + lineStart = i + 1; + } + } + return normalized.toByteArray(); + } + + /** Writes a raw byte, converting a high flag byte to the matching UTF-8 code point. */ + private static void writeNormalizedByte(ByteArrayOutputStream target, byte source, + boolean flagByte) { + final int value = source & 0xff; + if (flagByte && value >= 0x80) { + target.write(value < 0xc0 ? 0xc2 : 0xc3); + target.write(value < 0xc0 ? value : value - 0x40); + } else { + target.write(value); + } + } + + /** {@return whether the directive value is one Hunspell flag} */ + private static boolean isSingleFlagDirective(String directive) { + return switch (directive) { + case "COMPOUNDFLAG", "COMPOUNDBEGIN", "COMPOUNDMIDDLE", "COMPOUNDEND", + "COMPOUNDPERMITFLAG", "COMPOUNDFORBIDFLAG", "NEEDAFFIX", "PSEUDOROOT", + "ONLYINCOMPOUND", "FORBIDDENWORD", "CIRCUMFIX" -> true; + default -> false; + }; + } + + /** Finds the fields needed to classify one raw line. */ + private static int findAsciiFields(byte[] bytes, int from, int to, + int[] starts, int[] fieldEnds) { + int count = 0; + int cursor = from; + while (cursor < to && count < starts.length) { + while (cursor < to && isAsciiFieldSpace(bytes[cursor])) { + cursor++; + } + if (cursor == to) { + break; + } + starts[count] = cursor; + while (cursor < to && !isAsciiFieldSpace(bytes[cursor])) { + cursor++; + } + fieldEnds[count] = cursor; + count++; + } + return count; + } + + /** Returns one raw ASCII field. */ + private static String asciiField(byte[] bytes, int from, int to) { + return new String(bytes, from, to - from, StandardCharsets.US_ASCII); + } + + /** + * Converts file content without replacing malformed or unmappable input. + * + * @param bytes The encoded file content. + * @param charset The selected character encoding. + * @param label The file label used in the exception message. + * @return The decoded content. + * @throws IOException Thrown if {@code bytes} are invalid in {@code charset}. + */ + private static String decode(byte[] bytes, Charset charset, String label) + throws IOException { + try { + return charset.newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT) + .decode(ByteBuffer.wrap(bytes)) + .toString(); + } catch (CharacterCodingException e) { + throw new IOException(label + " is not valid " + charset.name(), e); + } + } + /** * Reads an input stream into a byte array, failing when more than {@code maxBytes} * arrive. @@ -461,9 +738,9 @@ private int positionalFlag(CompoundPosition position) { /** * Checks whether a listed word may stand at a compound position: some homonym's - * flag set carries the general compounding flag or the position's dedicated flag - * and is not forbidden. A compound-only or virtual-stem homonym may take the - * position; that is what those flags permit. + * flag set contains the general compound flag or the position's dedicated flag + * and is not forbidden. An {@code ONLYINCOMPOUND} entry is valid here, while a + * {@code NEEDAFFIX} entry still requires an affix. * * @param flagSets The word's flag sets from {@link #lookup(String)}. * @param position The part's place in the compound. @@ -473,7 +750,8 @@ boolean mayStand(List flagSets, CompoundPosition position) { final int positional = positionalFlag(position); for (final int[] flags : flagSets) { if ((contains(flags, compoundFlag) || contains(flags, positional)) - && !contains(flags, forbiddenWord) && !contains(flags, needAffix)) { + && !contains(flags, forbiddenWord) && !contains(flags, needAffix) + && !forbiddenAtCompoundPosition(flags, position)) { return true; } } @@ -481,22 +759,23 @@ boolean mayStand(List flagSets, CompoundPosition position) { } /** - * Checks whether some homonym supports an affixed compound part: its flag set - * carries the removed affix's flag, is not forbidden, and either the affix itself - * admits the position or the set carries the compounding or positional flag. + * Checks whether some homonym supports an affixed compound part. The flag set + * contains the removed affix flag, is not forbidden, and either the affix permits + * the position or the set contains the compound or positional flag. * * @param flagSets The part stem's flag sets from {@link #lookup(String)}. * @param affixFlag The removed affix's flag. * @param position The part's place in the compound. * @param affixAdmits Whether the affix's continuation classes admit the position, * from {@link #affixAdmits(Affix, CompoundPosition)}. - * @return {@code true} if some homonym stands affixed at the position. + * @return {@code true} if some homonym permits the affixed form at the position. */ boolean supportsPart(List flagSets, int affixFlag, CompoundPosition position, boolean affixAdmits) { final int positional = positionalFlag(position); for (final int[] flags : flagSets) { if (contains(flags, affixFlag) && !contains(flags, forbiddenWord) + && !forbiddenAtCompoundPosition(flags, position) && (affixAdmits || contains(flags, compoundFlag) || contains(flags, positional))) { return true; @@ -545,6 +824,23 @@ boolean forbidsInCompound(Affix affix) { return compoundForbid != 0 && affix.allowsContinuation(compoundForbid); } + /** + * Checks whether an entry marked with {@code COMPOUNDFORBIDFLAG} is barred from + * this compound position. Hunspell permits such an entry only as the last part, as + * specified by the + * + * format manual and the + * + * regression fixture. + * + * @param flags One entry's flag set. + * @param position The part's place in the compound. + * @return {@code true} if the entry may not stand at the position. + */ + private boolean forbiddenAtCompoundPosition(int[] flags, CompoundPosition position) { + return position != CompoundPosition.END && contains(flags, compoundForbid); + } + /** * Checks whether any of a word's flag sets is forbidden, which a dictionary uses * to block one specific ill-formed compound while its parts stay productive. @@ -633,20 +929,24 @@ boolean supports(List flagSets, int flag) { } /** - * Checks whether some homonym supports a cross-product analysis: one flag set - * carries both removed affixes' flags and is neither compound-only nor forbidden. - * The two flags must sit in the same set, because homonyms are separate words and - * each removal must be licensed by the same one. + * Checks whether some homonym supports a cross-product analysis. The root can contain + * both affix flags, or one root flag can select an affix with continuation flags that + * select the other affix. * * @param flagSets The stem's flag sets from {@link #lookup(String)}. - * @param prefixFlag The removed prefix's flag. - * @param suffixFlag The removed suffix's flag. - * @return {@code true} if some homonym carries both flags and may stand affixed. + * @param prefix The removed prefix. + * @param suffix The removed suffix. + * @return {@code true} if some homonym licenses both affixes. */ - boolean supports(List flagSets, int prefixFlag, int suffixFlag) { + boolean supportsCrossProduct(List flagSets, Affix prefix, Affix suffix) { for (final int[] flags : flagSets) { - if (contains(flags, prefixFlag) && contains(flags, suffixFlag) - && !contains(flags, onlyInCompound) && !contains(flags, forbiddenWord)) { + final boolean rootHasPrefix = contains(flags, prefix.flag()); + final boolean rootHasSuffix = contains(flags, suffix.flag()); + final boolean licensesBoth = (rootHasPrefix + && (rootHasSuffix || prefix.allowsContinuation(suffix.flag()))) + || (rootHasSuffix && suffix.allowsContinuation(prefix.flag())); + if (licensesBoth && !contains(flags, onlyInCompound) + && !contains(flags, forbiddenWord)) { return true; } } @@ -733,7 +1033,6 @@ private static final class AffixFile { private final List prefixes = new ArrayList<>(); private final List suffixes = new ArrayList<>(); private final List flagAliases = new ArrayList<>(); - private boolean aliasHeaderSeen; private FlagMode flagMode = FlagMode.CHAR; private int compoundFlag; private int compoundBegin; @@ -756,37 +1055,32 @@ private static final class AffixFile { /** * Parses the affix file: the {@code FLAG} declaration, the {@code AF} flag alias * table, the compound and blocking flag declarations, and the {@code PFX} and - * {@code SFX} blocks. Result-altering unsupported directives fail loud; - * cosmetic ones are skipped. + * {@code SFX} blocks. Other directives are skipped because this class implements + * affix stemming, not the complete Hunspell spell-checking engine. * * @param content The decoded affix file content. * @return The parsed rules and flag mode. Never {@code null}. - * @throws IOException Thrown if a supported directive is malformed, or if - * {@code ICONV}, {@code OCONV}, {@code COMPLEXPREFIXES}, {@code COMPOUNDRULE}, - * {@code IGNORE}, or {@code KEEPCASE} appears. + * @throws IOException Thrown if a supported directive is malformed. */ private static AffixFile parseAffix(String content) throws IOException { final AffixFile result = new AffixFile(); - final String[] lines = splitLines(content); + final String[] lines = splitLines(withoutByteOrderMark(content)); + final String[][] fieldsByLine = new String[lines.length][]; + for (int i = 0; i < lines.length; i++) { + fieldsByLine[i] = split(lines[i]); + } + result.flagMode = readFlagMode(fieldsByLine); + result.flagAliases.addAll(readFlagAliases(fieldsByLine, result.flagMode)); int i = 0; while (i < lines.length) { - final String[] fields = split(lines[i]); + final String[] fields = fieldsByLine[i]; if (fields.length == 0 || fields[0].startsWith(COMMENT_PREFIX)) { i++; continue; } switch (fields[0]) { - case "FLAG": - if (fields.length < 2) { - throw new IOException("FLAG line without a mode at line " + (i + 1)); - } - result.flagMode = switch (fields[1]) { - case "long" -> FlagMode.LONG; - case "num" -> FlagMode.NUM; - case "UTF-8" -> FlagMode.CHAR; - default -> throw new IOException( - "unsupported FLAG mode '" + fields[1] + "' at line " + (i + 1)); - }; + case FLAG_TAG: + // The file-wide declaration was parsed before any rule fields. i++; break; case "COMPOUNDFLAG": @@ -822,11 +1116,23 @@ private static AffixFile parseAffix(String content) throws IOException { i++; break; case "COMPOUNDMIN": - result.compoundMin = Math.max(1, parseValue(fields, i + 1)); + final int compoundMin = parseValue(fields, i + 1); + if (compoundMin < 0) { + throw new IOException("negative COMPOUNDMIN at line " + (i + 1)); + } + if (compoundMin > MAX_COMPOUND_MIN) { + throw new IOException("COMPOUNDMIN exceeds " + MAX_COMPOUND_MIN + + " at line " + (i + 1)); + } + result.compoundMin = Math.max(1, compoundMin); i++; break; case "COMPOUNDWORDMAX": - result.compoundWordMax = Math.max(0, parseValue(fields, i + 1)); + final int compoundWordMax = parseValue(fields, i + 1); + if (compoundWordMax < 0) { + throw new IOException("negative COMPOUNDWORDMAX at line " + (i + 1)); + } + result.compoundWordMax = compoundWordMax; i++; break; case "CHECKCOMPOUNDDUP": @@ -845,33 +1151,14 @@ private static AffixFile parseAffix(String content) throws IOException { result.fullStrip = true; i++; break; - case "AF": - // the first AF line declares the alias count; every further AF line is one - // alias, a flag run whose 1-based position numeric dictionary flags refer to - if (fields.length >= 2) { - if (!result.aliasHeaderSeen) { - result.aliasHeaderSeen = true; - } else { - result.flagAliases.add(parseFlags(fields[1], result.flagMode, i + 1)); - } - } + case ALIAS_TAG: + // The file-wide table was parsed before continuation and entry flags. i++; break; case PREFIX_TAG: case SUFFIX_TAG: - i = parseAffixBlock(lines, i, fields, result); + i = parseAffixBlock(fieldsByLine, i, fields, result); break; - case "ICONV": - case "OCONV": - case "COMPLEXPREFIXES": - // COMPOUNDRULE licenses pattern compounds, IGNORE drops characters before - // matching, and KEEPCASE forbids the case variants this stemmer analyzes; - // ignoring any of them would change stems with no signal - case "COMPOUNDRULE": - case "IGNORE": - case "KEEPCASE": - throw new IOException("unsupported affix directive '" + fields[0] - + "' at line " + (i + 1)); default: i++; break; @@ -880,6 +1167,80 @@ private static AffixFile parseAffix(String content) throws IOException { return result; } + /** + * Finds the file-wide flag mode before parsing directives that contain flags. + * + * @param fieldsByLine The affix file fields, indexed by source line. + * @return The selected flag mode, or character mode when no declaration is present. + * @throws IOException Thrown if the declaration is missing a mode, unsupported, or + * repeated. + */ + private static FlagMode readFlagMode(String[][] fieldsByLine) throws IOException { + FlagMode mode = FlagMode.CHAR; + boolean foundMode = false; + for (int i = 0; i < fieldsByLine.length; i++) { + final String[] fields = fieldsByLine[i]; + if (fields.length == 0 || fields[0].startsWith(COMMENT_PREFIX) + || !FLAG_TAG.equals(fields[0])) { + continue; + } + if (foundMode) { + throw new IOException("multiple FLAG directives at line " + (i + 1)); + } + if (fields.length < 2) { + throw new IOException("FLAG line without a mode at line " + (i + 1)); + } + mode = switch (fields[1]) { + case "long" -> FlagMode.LONG; + case "num" -> FlagMode.NUM; + case "UTF-8" -> FlagMode.CHAR; + default -> throw new IOException( + "unsupported FLAG mode '" + fields[1] + "' at line " + (i + 1)); + }; + foundMode = true; + } + return mode; + } + + /** + * Parses the file-wide flag alias table before parsing affix continuation flags. + * + * @param fieldsByLine The affix file fields, indexed by source line. + * @param mode The file's flag encoding. + * @return The aliases in their one-based reference order. + * @throws IOException Thrown if the table header, size, or an alias is malformed. + */ + private static List readFlagAliases(String[][] fieldsByLine, FlagMode mode) + throws IOException { + final List aliases = new ArrayList<>(); + int expected = -1; + for (int i = 0; i < fieldsByLine.length; i++) { + final String[] fields = fieldsByLine[i]; + if (fields.length == 0 || fields[0].startsWith(COMMENT_PREFIX) + || !ALIAS_TAG.equals(fields[0])) { + continue; + } + if (fields.length < 2) { + throw new IOException("AF line without a value at line " + (i + 1)); + } + if (expected < 0) { + // The AF header gives the alias count. Later AF lines contain flag runs. + // Numeric dictionary flags reference the one-based position of a run. + expected = parseValue(fields, i + 1); + if (expected < 0) { + throw new IOException("negative AF count at line " + (i + 1)); + } + } else { + aliases.add(parseFlags(fields[1], mode, i + 1)); + } + } + if (expected >= 0 && aliases.size() != expected) { + throw new IOException("AF header specifies " + expected + " aliases but found " + + aliases.size()); + } + return aliases; + } + /** * Parses the integer value of a directive that carries exactly one. * @@ -904,20 +1265,23 @@ private static int parseValue(String[] fields, int lineNumber) throws IOExceptio * cross-product marker, and the rule count, followed by exactly that many rule * lines. * - * @param lines All lines of the affix file. + * @param fieldsByLine All affix file fields, indexed by source line. * @param index The line index of the block header. * @param header The already-split header fields. * @param result The parse target the rules are added to. * @return The index of the first line after the block. * @throws IOException Thrown if the header or a rule line is malformed. */ - private static int parseAffixBlock(String[] lines, int index, String[] header, + private static int parseAffixBlock(String[][] fieldsByLine, int index, String[] header, AffixFile result) throws IOException { if (header.length < 4) { throw new IOException("malformed affix header at line " + (index + 1)); } final boolean suffix = SUFFIX_TAG.equals(header[0]); final int flag = parseFlag(header[1], result.flagMode, index + 1); + if (!"Y".equals(header[2]) && !"N".equals(header[2])) { + throw new IOException("invalid cross-product marker at line " + (index + 1)); + } final boolean crossProduct = "Y".equals(header[2]); final int count; try { @@ -925,21 +1289,28 @@ private static int parseAffixBlock(String[] lines, int index, String[] header, } catch (NumberFormatException e) { throw new IOException("malformed affix rule count at line " + (index + 1), e); } + if (count < 0) { + throw new IOException("negative affix rule count at line " + (index + 1)); + } int line = index + 1; for (int rule = 0; rule < count; rule++, line++) { - if (line >= lines.length) { + if (line >= fieldsByLine.length) { throw new IOException("affix block truncated at line " + (line + 1)); } - final String[] fields = split(lines[line]); + final String[] fields = fieldsByLine[line]; if (fields.length < 5 || !fields[0].equals(header[0])) { throw new IOException("malformed affix rule at line " + (line + 1)); } + if (parseFlag(fields[1], result.flagMode, line + 1) != flag) { + throw new IOException("affix rule flag does not match header at line " + (line + 1)); + } final String strip = NO_MATERIAL.equals(fields[2]) ? "" : fields[2]; String affixText = fields[3]; int[] continuation = new int[0]; final int slash = affixText.indexOf('/'); if (slash >= 0) { - continuation = parseFlags(affixText.substring(slash + 1), result.flagMode, line + 1); + continuation = parseAliasedFlags(affixText.substring(slash + 1), + result.flagMode, result.flagAliases, line + 1); affixText = affixText.substring(0, slash); } if (NO_MATERIAL.equals(affixText)) { @@ -975,7 +1346,7 @@ private static int parseAffixBlock(String[] lines, int index, String[] header, */ private static Map> parseWordList(String content, FlagMode flagMode, List flagAliases) throws IOException { - final String[] lines = splitLines(content); + final String[] lines = splitLines(withoutByteOrderMark(content)); final Map> entries = new HashMap<>(); int start = 0; if (lines.length > 0 && isCount(trim(lines[0]))) { @@ -1003,22 +1374,7 @@ private static Map> parseWordList(String content, break; } } - if (!flagAliases.isEmpty() && isCount(flagRun)) { - final int alias; - try { - alias = Integer.parseInt(flagRun); - } catch (NumberFormatException e) { - throw new IOException("malformed flag alias '" + flagRun + "' at line " - + (i + 1), e); - } - if (alias < 1 || alias > flagAliases.size()) { - throw new IOException("flag alias " + alias + " at line " + (i + 1) - + " is outside the AF table of " + flagAliases.size() + " aliases"); - } - flags = flagAliases.get(alias - 1); - } else { - flags = parseFlags(flagRun, flagMode, i + 1); - } + flags = parseAliasedFlags(flagRun, flagMode, flagAliases, i + 1); } entries.computeIfAbsent(word.replace("\\/", "/"), key -> new ArrayList<>(1)) .add(flags); @@ -1026,6 +1382,17 @@ private static Map> parseWordList(String content, return entries; } + /** + * Removes a Unicode byte-order mark decoded at the start of a file. + * + * @param content The decoded file content. + * @return The content without an initial byte-order mark. + */ + private static String withoutByteOrderMark(String content) { + return !content.isEmpty() && content.charAt(0) == '\uFEFF' + ? content.substring(1) : content; + } + /** * Checks whether a line consists purely of decimal digits, which identifies the * optional entry-count header of a word list. @@ -1045,6 +1412,37 @@ private static boolean isCount(String line) { return true; } + /** + * Resolves a numeric {@code AF} alias or parses a direct flag run when no alias + * applies. + * + * @param text The flag field without the leading slash. + * @param mode The selected flag encoding. + * @param aliases The affix file's alias table. + * @param lineNumber The source line, for error messages. + * @return The resolved or parsed flags. + * @throws IOException Thrown if the alias is malformed or outside the table, or the + * direct flags do not fit {@code mode}. + */ + private static int[] parseAliasedFlags(String text, FlagMode mode, + List aliases, int lineNumber) throws IOException { + if (!aliases.isEmpty() && isCount(text)) { + final int alias; + try { + alias = Integer.parseInt(text); + } catch (NumberFormatException e) { + throw new IOException("malformed flag alias '" + text + "' at line " + + lineNumber, e); + } + if (alias < 1 || alias > aliases.size()) { + throw new IOException("flag alias " + alias + " at line " + lineNumber + + " is outside the AF table of " + aliases.size() + " aliases"); + } + return aliases.get(alias - 1); + } + return parseFlags(text, mode, lineNumber); + } + /** * Finds the first {@code /} that is not escaped as {@code \/}, which separates the * word from its flag run in a word-list entry. @@ -1053,8 +1451,8 @@ private static boolean isCount(String line) { * @return The index of the separator, or {@code -1} when the entry has no flags. */ private static int unescapedSlash(String line) { - for (int i = 0; i < line.length(); i++) { - if (line.charAt(i) == '/' && (i == 0 || line.charAt(i - 1) != '\\')) { + for (int i = 1; i < line.length(); i++) { + if (line.charAt(i) == '/' && line.charAt(i - 1) != '\\') { return i; } } @@ -1149,11 +1547,16 @@ private static int[] parseFlags(String text, FlagMode mode, int lineNumber) final String[] parts = splitOn(text, ','); final int[] flags = new int[parts.length]; for (int i = 0; i < parts.length; i++) { + final String value = trim(parts[i]); try { - flags[i] = Integer.parseInt(trim(parts[i])); + flags[i] = Integer.parseInt(value); } catch (NumberFormatException e) { throw new IOException("malformed numeric flag at line " + lineNumber, e); } + if (flags[i] < 1 || flags[i] > MAX_NUMERIC_FLAG) { + throw new IOException("numeric flag outside 1.." + MAX_NUMERIC_FLAG + " at line " + + lineNumber + ": " + value); + } } return flags; } diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java index 656064da37..c5699fa149 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellStemmer.java @@ -31,7 +31,8 @@ /** * A dictionary-backed {@link Stemmer} over a {@link HunspellDictionary}: a surface form * is reduced to the dictionary words it can be derived from by removing one suffix, one - * prefix, or a cross-product combination of both. + * prefix, a cross-product combination of both, or an additional suffix licensed by a + * continuation class. * *

{@link #stem(CharSequence)} returns the first analysis, preferring the word's own * dictionary entry; {@link #stemAll(CharSequence)} returns every distinct analysis. A @@ -137,16 +138,22 @@ private List variants(String surface) { * Adds every analysis of one case variant to the result set: the word's own * dictionary entry, single suffix removal, twofold suffix removal through * continuation classes, single prefix removal, and cross-product removal of one - * prefix together with one suffix. Insertion order into the set fixes the + * prefix together with one suffix and an optional continuation suffix. Insertion + * order into the set fixes the * preference order reported by {@link #stemAll(CharSequence)}. * * @param word The case variant to analyze. * @param analyses The mutable, insertion-ordered set collecting the stems found. */ private void analyze(String word, Set analyses) { - final List own = dictionary.lookup(word); - if (own != null && dictionary.validStandalone(own)) { - analyses.add(word); + final List entries = dictionary.lookup(word); + if (entries != null) { + if (dictionary.anyForbidden(entries)) { + return; + } + if (dictionary.validStandalone(entries)) { + analyses.add(word); + } } for (final Affix suffix : dictionary.suffixesEndingWith( word.codePointBefore(word.length()))) { @@ -181,22 +188,30 @@ private void analyze(String word, Set analyses) { * @param analyses The mutable, insertion-ordered set collecting the part stems. */ private void decompose(String word, String surface, Set analyses) { - final List own = dictionary.lookup(word); - if (own != null && dictionary.anyForbidden(own)) { + final List entries = dictionary.lookup(word); + if (entries != null && dictionary.anyForbidden(entries)) { return; } - if (word.length() < 2 * dictionary.compoundMin()) { + final int codePointCount = word.codePointCount(0, word.length()); + if (codePointCount < 2 * dictionary.compoundMin()) { return; } + final int[] codePointOffsets = new int[codePointCount + 1]; + int offset = 0; + for (int i = 0; i < codePointCount; i++) { + codePointOffsets[i] = offset; + offset += Character.charCount(word.codePointAt(offset)); + } + codePointOffsets[codePointCount] = word.length(); // lowercasing may change the length in exceptional mappings, in which case the // offsets no longer align and the variant itself is the only usable case source final String caseSource = surface.length() == word.length() ? surface : word; - search(word, caseSource, 0, new ArrayList<>(), new ArrayList<>(), analyses, - new int[] {PART_CHECK_BUDGET}); + search(word, caseSource, codePointOffsets, 0, new ArrayList<>(), + new ArrayList<>(), analyses, new int[] {PART_CHECK_BUDGET}); } /** - * Extends a partial decomposition with the part starting at {@code from}, trying + * Extends a partial decomposition with the part starting at {@code fromPoint}, trying * every admissible length and recursing on the remainder. The boundary into this * part honors the {@code CHECKCOMPOUNDCASE} and {@code CHECKCOMPOUNDTRIPLE} * declarations, a part repeating its left neighbor honors @@ -206,28 +221,34 @@ private void decompose(String word, String surface, Set analyses) { * @param word The case variant under decomposition. * @param caseSource The character-case source for junction checks, the surface * form when its offsets align with the variant. - * @param from The index the next part starts at. + * @param codePointOffsets UTF-16 offsets for each code point boundary. + * @param fromPoint The code point index where the next part starts. * @param surfaces The surface strings of the parts taken so far. * @param stems The licensed stems of the parts taken so far, one list per part. * @param analyses The mutable, insertion-ordered set collecting the part stems. * @param budget The remaining part-licensing attempts, counted down in place. */ - private void search(String word, String caseSource, int from, List surfaces, - List> stems, Set analyses, int[] budget) { + private void search(String word, String caseSource, int[] codePointOffsets, + int fromPoint, List surfaces, List> stems, + Set analyses, int[] budget) { + final int from = codePointOffsets[fromPoint]; if (from > 0 && violatesBoundaryChecks(word, caseSource, from)) { return; } final int min = dictionary.compoundMin(); final int max = dictionary.compoundWordMax(); final boolean first = from == 0; + final int remaining = codePointOffsets.length - 1 - fromPoint; // every split leaving room for a further part; a first-position part must also // leave the closing part, so the whole word is never one part - if (max == 0 || surfaces.size() + 2 <= max) { - for (int end = from + min; end <= word.length() - min; end++) { + if (remaining >= 2 * min && (max == 0 || surfaces.size() + 2 <= max)) { + final int lastEndPoint = codePointOffsets.length - 1 - min; + for (int endPoint = fromPoint + min; endPoint <= lastEndPoint; endPoint++) { if (budget[0] <= 0) { return; } budget[0]--; + final int end = codePointOffsets[endPoint]; final String part = word.substring(from, end); if (duplicatesNeighbor(part, surfaces)) { continue; @@ -239,13 +260,14 @@ private void search(String word, String caseSource, int from, List surfa } surfaces.add(part); stems.add(partStems); - search(word, caseSource, end, surfaces, stems, analyses, budget); + search(word, caseSource, codePointOffsets, endPoint, surfaces, stems, + analyses, budget); surfaces.remove(surfaces.size() - 1); stems.remove(stems.size() - 1); } } // the closing part takes the whole remainder; a compound has at least two parts - if (first || word.length() - from < min + if (first || remaining < min || (max > 0 && surfaces.size() + 1 > max) || budget[0] <= 0) { return; } @@ -289,17 +311,20 @@ private boolean duplicatesNeighbor(String part, List surfaces) { * @return {@code true} if a declaration forbids this junction. */ private boolean violatesBoundaryChecks(String word, String caseSource, int from) { - final char before = word.charAt(from - 1); - final char after = word.charAt(from); + final int before = word.codePointBefore(from); + final int after = word.codePointAt(from); if (dictionary.checkCompoundCase() - && (Character.isUpperCase(caseSource.charAt(from - 1)) - || Character.isUpperCase(caseSource.charAt(from)))) { + && (Character.isUpperCase(caseSource.codePointBefore(from)) + || Character.isUpperCase(caseSource.codePointAt(from)))) { return true; } - if (dictionary.checkCompoundTriple() && before == after - && ((from >= 2 && word.charAt(from - 2) == after) - || (from + 1 < word.length() && word.charAt(from + 1) == after))) { - return true; + if (dictionary.checkCompoundTriple() && before == after) { + final int beforeStart = from - Character.charCount(before); + final int afterEnd = from + Character.charCount(after); + if ((beforeStart > 0 && word.codePointBefore(beforeStart) == after) + || (afterEnd < word.length() && word.codePointAt(afterEnd) == after)) { + return true; + } } return false; } @@ -347,8 +372,8 @@ private List partStems(String part, CompoundPosition position, */ private void collectPartStems(String part, CompoundPosition position, boolean first, boolean last, Set stems) { - final List own = dictionary.lookup(part); - if (own != null && dictionary.mayStand(own, position)) { + final List entries = dictionary.lookup(part); + if (entries != null && dictionary.mayStand(entries, position)) { stems.add(part); } for (final Affix suffix : dictionary.suffixesEndingWith( @@ -408,20 +433,18 @@ private void collectAffixedPartStem(String part, Affix affix, boolean suffix, * @return The candidate stem, or {@code null} when the rule does not apply. */ private String removeAffixInCompound(String part, Affix affix, boolean suffix) { - if (affix.affix().isEmpty() && affix.strip().isEmpty()) { - return affix.condition().matches(part) ? part : null; - } - return suffix ? removeSuffix(part, affix) : removePrefix(part, affix); + return suffix + ? removeSuffixAllowingIdentity(part, affix) + : removePrefixAllowingIdentity(part, affix); } /** * Undoes one suffix rule and, through continuation classes, one further suffix on - * the intermediate stem, adding every dictionary-confirmed analysis. A rule that - * applies only inside compounds or only as half of a circumfix is not undone at all, - * the latter because no prefix accompanies it on this path; a rule marked as needing - * a further affix yields no single-removal analysis, because the surface form it - * makes alone is a virtual stem; its twofold analyses stand, the inner affix being - * exactly the further one required. + * the intermediate stem, adding dictionary-confirmed analyses. A rule that applies + * only inside compounds or requires the matching circumfix member is not undone + * because no prefix accompanies this path. A rule requiring a further affix produces + * no single-removal analysis. An identity rule also produces no single-removal + * analysis, but it can complete a two-suffix analysis through continuation classes. * * @param word The case variant under analysis. * @param suffix The suffix rule to undo. @@ -431,11 +454,12 @@ private void undoSuffix(String word, Affix suffix, Set analyses) { if (dictionary.compoundOnly(suffix) || dictionary.circumfixOnly(suffix)) { return; } - final String stem = removeSuffix(word, suffix); + final boolean identity = isIdentityRule(suffix); + final String stem = removeSuffixAllowingIdentity(word, suffix); if (stem == null) { return; } - if (!dictionary.needsFurtherAffix(suffix)) { + if (!identity && !dictionary.needsFurtherAffix(suffix)) { final List flagSets = dictionary.lookup(stem); if (flagSets != null && dictionary.supports(flagSets, suffix.flag())) { analyses.add(stem); @@ -451,8 +475,9 @@ private void undoSuffix(String word, Affix suffix, Set analyses) { } /** - * Undoes the second suffix of a twofold removal when the inner rule's continuation - * classes allow it after the outer one. + * Undoes the inner suffix of a twofold removal when the rule's continuation + * classes allow the outer one. The continuation-linked combination satisfies a + * {@code NEEDAFFIX} marker on either rule. * * @param stem The intermediate stem after the outer removal. * @param outer The already-undone outer suffix rule. @@ -465,7 +490,7 @@ private void undoInnerSuffix(String stem, Affix outer, Affix inner, || dictionary.circumfixOnly(inner)) { return; } - final String doubleStem = removeSuffix(stem, inner); + final String doubleStem = removeSuffixAllowingIdentity(stem, inner); if (doubleStem == null) { return; } @@ -477,11 +502,11 @@ private void undoInnerSuffix(String stem, Affix outer, Affix inner, /** * Undoes one prefix rule and, for cross-product rules, one further suffix on the - * intermediate stem, adding every dictionary-confirmed analysis. A rule that + * intermediate stem, adding dictionary-confirmed analyses. A rule that * applies only inside compounds is not undone at all. A rule marked as needing a - * further affix or as half of a circumfix yields no single-removal analysis; its - * cross-product analyses stand, the suffix being exactly the further affix or the - * other circumfix half required. + * further affix or the matching circumfix member produces no single-removal analysis. + * An identity rule also produces no single-removal analysis. A valid cross-product + * suffix can combine with either kind of rule. * * @param word The case variant under analysis. * @param prefix The prefix rule to undo. @@ -491,11 +516,13 @@ private void undoPrefix(String word, Affix prefix, Set analyses) { if (dictionary.compoundOnly(prefix)) { return; } - final String stem = removePrefix(word, prefix); + final boolean identity = isIdentityRule(prefix); + final String stem = removePrefixAllowingIdentity(word, prefix); if (stem == null) { return; } - if (!dictionary.needsFurtherAffix(prefix) && !dictionary.circumfixOnly(prefix)) { + if (!identity && !dictionary.needsFurtherAffix(prefix) + && !dictionary.circumfixOnly(prefix)) { final List flagSets = dictionary.lookup(stem); if (flagSets != null && dictionary.supports(flagSets, prefix.flag())) { analyses.add(stem); @@ -534,12 +561,50 @@ private void undoCrossProductSuffix(String stem, Affix prefix, Affix suffix, if (doubleStem == null) { return; } - // a needs-further-affix marker on either rule is satisfied by the other rule, - // so no such check applies here; both flags must sit in one homonym's flag set + // One member can satisfy the other member's needs-further-affix marker. Both rule + // flags must occur in one homonym's flag set. final List both = dictionary.lookup(doubleStem); - if (both != null && dictionary.supports(both, prefix.flag(), suffix.flag())) { + if (both != null && dictionary.supportsCrossProduct(both, prefix, suffix) + && !(dictionary.needsFurtherAffix(prefix) + && dictionary.needsFurtherAffix(suffix))) { analyses.add(doubleStem); } + for (final Affix inner : dictionary.suffixesEndingWith( + doubleStem.codePointBefore(doubleStem.length()))) { + undoCrossProductInnerSuffix(doubleStem, prefix, suffix, inner, analyses); + } + for (final Affix inner : dictionary.suffixesWithoutMaterial()) { + undoCrossProductInnerSuffix(doubleStem, prefix, suffix, inner, analyses); + } + } + + /** + * Undoes an inner suffix after a prefix and an outer suffix have been removed. + * The inner suffix must license the outer suffix through the continuation flags. + * The suffix combination satisfies {@code NEEDAFFIX} markers in the derivation. + * + * @param stem The intermediate stem after the prefix and outer suffix removal. + * @param prefix The already-undone prefix rule. + * @param outer The already-undone outer suffix rule. + * @param inner The candidate inner suffix rule. + * @param analyses The mutable, insertion-ordered set collecting the stems found. + */ + private void undoCrossProductInnerSuffix(String stem, Affix prefix, Affix outer, + Affix inner, Set analyses) { + if (!inner.crossProduct() || !inner.allowsContinuation(outer.flag()) + || dictionary.compoundOnly(inner) + || dictionary.circumfixOnly(outer) + || dictionary.circumfixOnly(prefix) != dictionary.circumfixOnly(inner)) { + return; + } + final String root = removeSuffixAllowingIdentity(stem, inner); + if (root == null) { + return; + } + final List flagSets = dictionary.lookup(root); + if (flagSets != null && dictionary.supportsCrossProduct(flagSets, prefix, inner)) { + analyses.add(root); + } } /** @@ -568,6 +633,46 @@ private String removeSuffix(String word, Affix suffix) { return suffix.condition().matches(stem) ? stem : null; } + /** + * Undoes a suffix in a continuation sequence, including a rule that changes no + * material. An identity rule still has to satisfy the condition. + * + * @param word The surface form at this point in the sequence. + * @param suffix The rule to undo. + * @return The candidate stem, or {@code null} when the rule does not apply. + */ + private String removeSuffixAllowingIdentity(String word, Affix suffix) { + if (isIdentityRule(suffix)) { + return suffix.condition().matches(word) ? word : null; + } + return removeSuffix(word, suffix); + } + + /** + * Undoes a prefix in a continuation sequence, including a rule that changes no + * material. An identity rule still has to satisfy the condition. + * + * @param word The surface form at this point in the sequence. + * @param prefix The rule to undo. + * @return The candidate stem, or {@code null} when the rule does not apply. + */ + private String removePrefixAllowingIdentity(String word, Affix prefix) { + if (isIdentityRule(prefix)) { + return prefix.condition().matches(word) ? word : null; + } + return removePrefix(word, prefix); + } + + /** + * Checks whether an affix rule adds and strips no material. + * + * @param affix The rule to inspect. + * @return {@code true} if applying the rule does not change the spelling. + */ + private boolean isIdentityRule(Affix affix) { + return affix.affix().isEmpty() && affix.strip().isEmpty(); + } + /** * Undoes one prefix rule: cuts the affix material off the start of the word, * restores the strip string the rule removed on application, and checks the rule's diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java index 8657b7e8c0..312d785483 100644 --- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/hunspell/HunspellStemmerTest.java @@ -40,6 +40,8 @@ */ public class HunspellStemmerTest { + private static final byte TRUNCATED_UTF8_LEAD_BYTE = (byte) 0xC3; + private static final String AFFIX = String.join("\n", "# project-authored test fixture", "SET UTF-8", @@ -367,6 +369,10 @@ void testMalformedFlagDeclarationMessages() { e = Assertions.assertThrows(IOException.class, () -> load("FLAG short\n", "0\n")); Assertions.assertEquals("unsupported FLAG mode 'short' at line 1", e.getMessage()); + + e = Assertions.assertThrows(IOException.class, + () -> load("FLAG num\nFLAG UTF-8\n", "0\n")); + Assertions.assertEquals("multiple FLAG directives at line 2", e.getMessage()); } /** @@ -389,10 +395,22 @@ void testMalformedAffixBlockMessages() { () -> load("SFX S Y 2\nSFX S 0 s .", "0\n")); Assertions.assertEquals("affix block truncated at line 3", e.getMessage()); + e = Assertions.assertThrows(IOException.class, + () -> load("SFX S X 1\nSFX S 0 s .\n", "0\n")); + Assertions.assertEquals("invalid cross-product marker at line 1", e.getMessage()); + e = Assertions.assertThrows(IOException.class, () -> load("SFX S Y 1\nPFX S 0 s .\n", "0\n")); Assertions.assertEquals("malformed affix rule at line 2", e.getMessage()); + e = Assertions.assertThrows(IOException.class, + () -> load("SFX S Y 1\nSFX T 0 s .\n", "0\n")); + Assertions.assertEquals("affix rule flag does not match header at line 2", e.getMessage()); + + e = Assertions.assertThrows(IOException.class, + () -> load("SFX S Y -1\n", "0\n")); + Assertions.assertEquals("negative affix rule count at line 1", e.getMessage()); + e = Assertions.assertThrows(IOException.class, () -> load("SFX S Y 1\nSFX S 0 s [ab\n", "0\n")); Assertions.assertEquals("unterminated character class at line 2", e.getMessage()); @@ -1231,25 +1249,38 @@ void testForbiddenEntryBlocksItsDecomposition() throws IOException { } /** - * Verifies that result-altering unsupported affix directives fail at load time. - * Ignoring {@code ICONV}, {@code OCONV}, {@code COMPLEXPREFIXES}, - * {@code COMPOUNDRULE}, {@code IGNORE}, or {@code KEEPCASE} would change stems - * with no signal. + * Verifies that directives outside the affix-stemming subset do not prevent use of + * the rules this implementation supports. + * + * @param line The affix file line. */ @ParameterizedTest - @CsvSource({ - "ICONV, ICONV 1", - "OCONV, OCONV 1", - "COMPLEXPREFIXES, COMPLEXPREFIXES", - "COMPOUNDRULE, COMPOUNDRULE 1", - "IGNORE, IGNORE x", - "KEEPCASE, KEEPCASE k" + @ValueSource(strings = { + "ICONV 1", + "OCONV 1", + "COMPLEXPREFIXES", + "COMPOUNDRULE 1", + "COMPOUNDMORESUFFIXES", + "COMPOUNDROOT R", + "CHECKCOMPOUNDREP", + "SIMPLIFIEDTRIPLE", + "CHECKCOMPOUNDPATTERN 1", + "FORCEUCASE U", + "COMPOUNDSYLLABLE 6 aeiou", + "SYLLABLENUM ABC", + "LANG tr", + "CHECKSHARPS", + "BREAK 1", + "FORBIDWARN", + "IGNORE x", + "KEEPCASE k" }) - void testResultAlteringUnsupportedDirectiveFailsLoud(String name, String line) { - final IOException e = Assertions.assertThrows(IOException.class, - () -> load(line + "\n", "0\n")); - Assertions.assertEquals("unsupported affix directive '" + name + "' at line 1", - e.getMessage()); + void testUnsupportedDirectiveDoesNotBlockSupportedRules(String line) + throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + line + "\nSFX A Y 1\nSFX A 0 s .\n", "1\ndog/A\n")); + + Assertions.assertEquals("dog", stemmer.stem("dogs").toString()); } /** @@ -1397,4 +1428,489 @@ void testCosmeticUnsupportedDirectiveIsSkipped() throws IOException { final HunspellDictionary dictionary = load("REP 1\nREP alot a lot\n", "1\nlock\n"); Assertions.assertNotNull(dictionary.lookup("lock")); } + + /** + * Verifies that an {@code AF} table applies to affix rules above the table. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testAliasTableAppliesToAffixesThatPrecedeIt() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "FLAG num", + "SFX 1 Y 1", + "SFX 1 0 er/1 .", + "SFX 2 Y 1", + "SFX 2 0 s .", + "AF 2", + "AF 2", + "AF 1", + ""), + "1\nkind/2\n")); + + Assertions.assertEquals(List.of("kind"), stemmer.stemAll("kinders")); + } + + /** + * Verifies that {@code COMPOUNDMIN} counts Unicode code points. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testCompoundMinCountsSupplementaryCharactersOnce() throws IOException { + final String first = "\uD840\uDC00"; + final String rightPart = "\uD840\uDC01"; + final String words = "2\n" + first + "/Z\n" + rightPart + "/Z\n"; + + final HunspellStemmer minimumTwo = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 2\n", words)); + Assertions.assertEquals(List.of(first + rightPart), minimumTwo.stemAll(first + rightPart)); + + final HunspellStemmer minimumOne = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 1\n", words)); + Assertions.assertEquals(List.of(first, rightPart), minimumOne.stemAll(first + rightPart)); + } + + /** + * Verifies cross-product analysis with stacked suffixes. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testCrossProductSupportsTwofoldSuffixes() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "PFX U Y 1", + "PFX U 0 un .", + "SFX A Y 1", + "SFX A 0 s/B .", + "SFX B Y 1", + "SFX B 0 bar .", + ""), + "1\nfoo/AU\n")); + + Assertions.assertEquals("foo", stemmer.stem("unfoosbar").toString()); + } + + /** Verifies that an unrecognized directive does not block supported affix rules. */ + @Test + void testUnknownAffixDirectiveIsSkipped() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "UNRECOGNIZED value\nSFX A Y 1\nSFX A 0 s .\n", "1\ndog/A\n")); + + Assertions.assertEquals("dog", stemmer.stem("dogs").toString()); + } + + /** Verifies validation of the {@code AF} count line. */ + @ParameterizedTest + @ValueSource(strings = {"count-mismatch", "malformed", "negative"}) + void testAliasTableCountIsValidated(String fixture) { + final String affix; + final String message; + switch (fixture) { + case "count-mismatch" -> { + affix = "AF 2\nAF A\n"; + message = "AF header specifies 2 aliases but found 1"; + } + case "malformed" -> { + affix = "AF count\n"; + message = "malformed AF at line 1"; + } + case "negative" -> { + affix = "AF -1\n"; + message = "negative AF count at line 1"; + } + default -> throw new AssertionError(fixture); + } + final IOException exception = Assertions.assertThrows(IOException.class, + () -> load(affix, "0\n")); + + Assertions.assertEquals(message, exception.getMessage()); + } + + /** + * Verifies the numeric flag range. + * + * @param flag The invalid numeric flag. + */ + @ParameterizedTest + @ValueSource(strings = {"-1", "0", "65001"}) + void testNumericFlagOutsideRangeIsRejected(String flag) { + final IOException e = Assertions.assertThrows(IOException.class, + () -> load("FLAG num\n", "1\nword/" + flag + "\n")); + + Assertions.assertEquals("numeric flag outside 1..65000 at line 2: " + flag, + e.getMessage()); + } + + /** + * Verifies that an identity suffix can license an outer suffix. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testZeroMaterialInnerSuffixLicensesOuterSuffix() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "SFX A Y 1", + "SFX A 0 0/B .", + "SFX B Y 1", + "SFX B 0 baz .", + ""), + "1\nbar/A\n")); + + Assertions.assertEquals(List.of("bar"), stemmer.stemAll("barbaz")); + } + + /** + * Verifies that both {@code NEEDAFFIX} markers cannot satisfy one another. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testCrossProductNeedAffixMarkersDoNotSatisfyEachOther() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "NEEDAFFIX X", + "PFX P Y 1", + "PFX P 0 pseudo/X .", + "SFX A Y 1", + "SFX A 0 pseudo/X .", + ""), + "1\nfoo/AP\n")); + + Assertions.assertEquals(List.of("pseudofoopseudo"), + stemmer.stemAll("pseudofoopseudo")); + } + + /** + * Verifies that a slash at the start of a dictionary entry is word text. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testLeadingSlashIsPartOfWord() throws IOException { + final HunspellStemmer slashWord = new HunspellStemmer(load( + "SFX X Y 1\nSFX X 0 s .\n", + "2\n/foo\n/foo/X\n")); + + Assertions.assertEquals(List.of("/foo"), slashWord.stemAll("/foos")); + } + + /** + * Verifies that an identity continuation completes a virtual suffix. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testZeroMaterialContinuationCompletesVirtualSuffix() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "PSEUDOROOT X", + "SFX A Y 1", + "SFX A 0 0 .", + "SFX C Y 1", + "SFX C 0 baz/XA .", + ""), + "1\nbar/C\n")); + + Assertions.assertEquals(List.of("bar"), stemmer.stemAll("barbaz")); + } + + /** + * Verifies cross-product licensing from either member's continuation flags. + * + * @param licensingRule The member that identifies the partner. + * @throws IOException Thrown if a fixture fails to load. + */ + @ParameterizedTest + @ValueSource(strings = {"prefix", "suffix"}) + void testContinuationFlagLicensesCrossProductPartner(String licensingRule) + throws IOException { + final boolean prefixLicenses = "prefix".equals(licensingRule); + final String affix = prefixLicenses + ? String.join("\n", + "PFX P Y 1", + "PFX P 0 un/S .", + "SFX S Y 1", + "SFX S 0 s .", + "") + : String.join("\n", + "PFX P Y 1", + "PFX P 0 un .", + "SFX R Y 1", + "SFX R 0 able/P .", + ""); + final String words = prefixLicenses ? "1\nlock/P\n" : "1\ndrink/R\n"; + final String surface = prefixLicenses ? "unlocks" : "undrinkable"; + final String expected = prefixLicenses ? "lock" : "drink"; + + final HunspellStemmer stemmer = new HunspellStemmer(load(affix, words)); + + Assertions.assertEquals(expected, stemmer.stem(surface).toString()); + } + + /** + * Verifies that {@code FLAG} applies to affix rules above the declaration. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testFlagModeAppliesToRulesThatPrecedeTheDeclaration() throws IOException { + final HunspellStemmer lateFlagMode = new HunspellStemmer(load(String.join("\n", + "SFX 1 Y 1", + "SFX 1 0 s .", + "FLAG num", + ""), String.join("\n", "1", "dog/1", ""))); + + Assertions.assertEquals("dog", lateFlagMode.stem("dogs").toString()); + } + + /** + * Verifies that {@code COMPOUNDFORBIDFLAG} rejects nonfinal dictionary entries. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testCompoundForbidFlagBarsDictionaryEntryBeforeEnd() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDFORBIDFLAG F\nCOMPOUNDMIN 3\n", + "2\ndog/ZF\nhouse/Z\n")); + + Assertions.assertEquals(List.of("doghouse"), stemmer.stemAll("doghouse")); + Assertions.assertEquals(List.of("house", "dog"), stemmer.stemAll("housedog")); + } + + /** + * Verifies that a forbidden surface form is not analyzed through an affix rule. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testForbiddenSurfaceOverridesAffixAnalysis() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "FORBIDDENWORD X\nSFX A Y 1\nSFX A 0 s .\n", + "2\nfoo/A\nfoos/X\n")); + + Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos")); + } + + /** + * Verifies that a forbidden homonym blocks affix analysis even when another entry + * for the same surface is valid as a standalone entry. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testForbiddenHomonymOverridesStandaloneEntry() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + "FORBIDDENWORD X\nSFX A Y 1\nSFX A 0 s .\n", + "3\nfoo/A\nfoos\nfoos/X\n")); + + Assertions.assertEquals(List.of("foos"), stemmer.stemAll("foos")); + } + + /** + * Verifies that malformed UTF-8 is rejected in semantic affix content and in the + * dictionary file. + * + * @param file The malformed input file. + */ + @ParameterizedTest + @ValueSource(strings = {"affix", "dictionary"}) + void testMalformedFileEncodingIsRejected(String file) { + final byte[] affix = "SET UTF-8\n".getBytes(StandardCharsets.UTF_8); + final byte[] words = Arrays.copyOf("1\n".getBytes(StandardCharsets.UTF_8), 3); + words[2] = TRUNCATED_UTF8_LEAD_BYTE; + final byte[] affixPrefix = "SET UTF-8\nSFX A Y 1\nSFX A 0 " + .getBytes(StandardCharsets.UTF_8); + final byte[] malformedAffix = Arrays.copyOf(affixPrefix, affixPrefix.length + 1); + malformedAffix[malformedAffix.length - 1] = TRUNCATED_UTF8_LEAD_BYTE; + final byte[] selectedAffix = "affix".equals(file) ? malformedAffix : affix; + final byte[] selectedWords = "dictionary".equals(file) + ? words : "0\n".getBytes(StandardCharsets.UTF_8); + + final IOException exception = Assertions.assertThrows(IOException.class, + () -> HunspellDictionary.load(new ByteArrayInputStream(selectedAffix), + new ByteArrayInputStream(selectedWords))); + + Assertions.assertEquals(file + " stream is not valid UTF-8", exception.getMessage()); + } + + /** + * Verifies that invalid bytes in a comment do not prevent loading an otherwise valid + * UTF-8 affix file. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testMalformedCommentEncodingIsIgnored() throws IOException { + final byte[] prefix = "SET UTF-8\n# ".getBytes(StandardCharsets.UTF_8); + final byte[] affix = Arrays.copyOf(prefix, prefix.length + 1); + affix[affix.length - 1] = TRUNCATED_UTF8_LEAD_BYTE; + + final HunspellDictionary dictionary = HunspellDictionary.load( + new ByteArrayInputStream(affix), + new ByteArrayInputStream("1\ndog\n".getBytes(StandardCharsets.UTF_8))); + + Assertions.assertNotNull(dictionary.lookup("dog")); + } + + /** + * Verifies raw one-byte flags in a file where word text uses UTF-8. + * + * @param representation Whether the dictionary entry uses an alias or a direct flag. + * @throws IOException Thrown if the fixture fails to load. + */ + @ParameterizedTest + @ValueSource(strings = {"alias", "direct"}) + void testDefaultFlagModePreservesRawBytesInUtf8File(String representation) + throws IOException { + final boolean alias = "alias".equals(representation); + final String aliasTable = alias ? "AF 1\nAF \u00D7\n" : ""; + final byte[] affix = ("SET UTF-8\n" + aliasTable + + "SFX \u00D7 Y 1\nSFX \u00D7 0 s .\n") + .getBytes(StandardCharsets.ISO_8859_1); + final byte[] words = (alias ? "1\ndog/1\n" : "1\ndog/\u00D7\n") + .getBytes(alias ? StandardCharsets.UTF_8 : StandardCharsets.ISO_8859_1); + final HunspellStemmer stemmer = new HunspellStemmer(HunspellDictionary.load( + new ByteArrayInputStream(affix), new ByteArrayInputStream(words))); + + Assertions.assertEquals("dog", stemmer.stem("dogs").toString()); + } + + /** + * Verifies {@code AF} references in affix continuation fields. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testAffixContinuationFlagsResolveThroughTheAliasTable() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "AF 2", + "AF AB", + "AF A", + "SFX A Y 1", + "SFX A 0 x .", + "SFX B Y 1", + "SFX B 0 y/2 .", + ""), + "1\nfoo/1\n")); + + Assertions.assertEquals(List.of("foo"), stemmer.stemAll("fooyx")); + } + + /** + * Verifies that a stacked suffix satisfies {@code NEEDAFFIX} in a cross-product. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testStackedSuffixSatisfiesNeedAffixWithinCrossProduct() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "NEEDAFFIX X", + "PFX P Y 1", + "PFX P 0 pseudo/X .", + "SFX A Y 1", + "SFX A 0 pseudo/XB .", + "SFX B Y 1", + "SFX B 0 bar/X .", + ""), + "1\nfoo/AP\n")); + + Assertions.assertEquals(List.of("foo"), + stemmer.stemAll("pseudofoopseudobar")); + } + + /** + * Verifies cross-product analysis with an identity inner suffix. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testZeroMaterialInnerSuffixSupportsCrossProduct() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "PFX P Y 1", + "PFX P 0 un .", + "SFX A Y 1", + "SFX A 0 0/B .", + "SFX B Y 1", + "SFX B 0 baz .", + ""), + "1\nbar/AP\n")); + + Assertions.assertEquals(List.of("bar"), stemmer.stemAll("unbarbaz")); + } + + /** + * Verifies that compound limits cannot be negative. + * + * @param directive The compound limit directive. + */ + @ParameterizedTest + @ValueSource(strings = {"COMPOUNDMIN", "COMPOUNDWORDMAX"}) + void testNegativeCompoundLimitIsRejected(String directive) { + final IOException e = Assertions.assertThrows(IOException.class, + () -> load(directive + " -1\n", "0\n")); + + Assertions.assertEquals("negative " + directive + " at line 1", e.getMessage()); + } + + /** Verifies that {@code COMPOUNDMIN} cannot overflow the doubled length check. */ + @Test + void testCompoundMinAboveSafeRangeIsRejected() { + final IOException exception = Assertions.assertThrows(IOException.class, + () -> load("COMPOUNDMIN 1073741824\n", "0\n")); + + Assertions.assertEquals("COMPOUNDMIN exceeds 1073741823 at line 1", + exception.getMessage()); + } + + /** + * Verifies cross-product analysis with an identity prefix. + * + * @throws IOException Thrown if the fixture fails to load. + */ + @Test + void testZeroMaterialPrefixParticipatesInCrossProduct() throws IOException { + final HunspellStemmer stemmer = new HunspellStemmer(load( + String.join("\n", + "PFX P Y 1", + "PFX P 0 0/S .", + "SFX S Y 1", + "SFX S 0 s .", + ""), + "1\nroot/P\n")); + + Assertions.assertEquals(List.of("root"), stemmer.stemAll("roots")); + } + + /** + * Verifies supplementary characters in compound boundary checks. + * + * @param check The boundary check to exercise. + * @throws IOException Thrown if a fixture fails to load. + */ + @ParameterizedTest + @ValueSource(strings = {"case", "triple"}) + void testCompoundBoundaryChecksUseCodePoints(String check) throws IOException { + final boolean triple = "triple".equals(check); + final String codePoint = triple ? "\uD840\uDC00" : "\uD801\uDC00"; + final String words = triple + ? "2\na" + codePoint + codePoint + "/Z\n" + codePoint + "b/Z\n" + : "2\na/Z\n" + codePoint + "b/Z\n"; + final String declaration = triple ? "CHECKCOMPOUNDTRIPLE" : "CHECKCOMPOUNDCASE"; + final String surface = triple + ? "a" + codePoint + codePoint + codePoint + "b" + : "a" + codePoint + "b"; + final HunspellStemmer stemmer = new HunspellStemmer(load( + "COMPOUNDFLAG Z\nCOMPOUNDMIN 1\n" + declaration + "\n", words)); + + Assertions.assertEquals(List.of(surface), stemmer.stemAll(surface)); + } + } diff --git a/opennlp-docs/src/docbkx/stemmer.xml b/opennlp-docs/src/docbkx/stemmer.xml index ddb5056411..d2367684b5 100644 --- a/opennlp-docs/src/docbkx/stemmer.xml +++ b/opennlp-docs/src/docbkx/stemmer.xml @@ -99,10 +99,22 @@ stemmer.stem("table"); // "table" (unknown vocabulary is unchanged)]]> -Dopennlp.download.remote=true, and fetches through the digest-verified ResourceInstaller path. A file that already exists in the target is not replaced. Remove old files before refreshing a dictionary. - Directives that would change stems when ignored - (ICONV, OCONV, COMPLEXPREFIXES, - COMPOUNDRULE, IGNORE, KEEPCASE) - fail at load time; cosmetic tables such as REP are skipped. + Directives outside the supported affix-stemming subset are skipped, so + published dictionaries can still use their supported rules. Conversion, + suggestion, and advanced compound behavior from skipped directives is not + applied to the returned stems. + This includes ICONV, OCONV, + COMPLEXPREFIXES, COMPOUNDRULE, + IGNORE, and KEEPCASE. Results can differ from + Hunspell for words that need these rules. + FLAG and AF declarations apply to the complete + affix file, including rules listed before those declarations. Parsing + rejects malformed text in parsed rules, invalid counts, numeric flags + outside the range 1 through 65000, and COMPOUNDMIN values + that cannot be doubled safely. Comments and unused metadata may retain a + legacy encoding. Default and long flag modes preserve raw + one-byte flag values in UTF-8 files. Compound length and boundary checks + count Unicode code points. A rule that strips a whole stem applies only when the affix file declares FULLSTRIP, as in Hunspell itself. Each affix or dictionary stream is rejected when it exceeds