OPENNLP-1893: Support Hunspell affix dictionaries for stemming

dev/README-hunspell-dictionaries.md

@@ -0,0 +1,89 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+# Hunspell dictionaries for the affix stemmer
+
+The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) implements the documented Hunspell dictionary format: a `.dic` word list plus its `.aff` affix companion, both supplied by the user. Apache OpenNLP bundles no dictionary data; whichever dictionary you download, its license is stated in the readme shipped alongside it.
+
+## Where dictionaries come from
+
+The LibreOffice project maintains a large collection of Hunspell dictionaries, one directory per language, at `github.com/LibreOffice/dictionaries`. Licenses differ per dictionary, which is why nothing is bundled: for example, the `en_US` dictionary derives from SCOWL and states its terms in `README_en_US.txt` in the same directory. Many other sources work too; the engine only cares that the pair follows the Hunspell format.
+
+OpenNLP does not ship a URL catalog. Applications that manage downloads can keep a
+properties file with an entry id followed by `.url`, `.sha512`, and optionally
+`.filename` keys. Pin each URL to a stable release or commit.
+
+## Option A: application catalog
+
+Catalog downloads stay inactive until you set `-Dopennlp.download.remote=true`.
+
+```java
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import opennlp.tools.stemmer.hunspell.HunspellDictionaryDownload;
+import opennlp.tools.util.DictionaryCatalog;
+
+// JVM flag: -Dopennlp.download.remote=true
+try (InputStream in = Files.newInputStream(Path.of("dictionary-catalog.properties"))) {
+  DictionaryCatalog catalog = DictionaryCatalog.load(in);
+  HunspellDictionaryDownload.downloadFromCatalog(
+      catalog, "en_US", Path.of("/tmp/hunspell-en_US"));
+}
+```
+
+For `en_US`, the catalog ids are `hunspell.en_US.aff`, `hunspell.en_US.dic`, and
+optionally `hunspell.en_US.readme`. A complete catalog example lives at
+`opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/util/dictionary-catalog.properties`.
+The download test uses local file URLs to exercise this flow without network access.
+
+## Option B: your own files
+
+Fetch `.aff` / `.dic` (and the license readme) with any tool, or with
+`DownloadUtil.download(uri, path, sha512)`, then load them:
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.stemmer.Stemmer;
+import opennlp.tools.stemmer.hunspell.HunspellDictionary;
+import opennlp.tools.stemmer.hunspell.HunspellStemmerFactory;
+
+HunspellDictionary dictionary = HunspellDictionary.load(
+    Path.of("/tmp/hunspell-en_US/en_US.aff"),
+    Path.of("/tmp/hunspell-en_US/en_US.dic"));
+HunspellStemmerFactory factory = new HunspellStemmerFactory(dictionary);
+
+Stemmer stemmer = factory.newStemmer();
+CharSequence stem = stemmer.stem("workers");
+```
+
+What `stem` evaluates to is decided by the dictionary you loaded, and this project ships no dictionary data, so no result is claimed here for `en_US`. The same load-and-stem flow is pinned by `HunspellManualExampleTest` (miniature in-memory dictionary, asserted stems for `workers` and `worker`) and by `HunspellStemmerFactoryTest#testEndToEndUsageFromFiles` (the same pair written to disk). The developer manual chapter `stemmer.xml` cites `HunspellManualExampleTest`.
+
+The dictionary is immutable and safe to share between threads; the factory hands out a fresh stemmer per call, so each thread takes its own from `newStemmer()`. A dictionary that declares a non-UTF-8 encoding through the `SET` directive in its `.aff` file is decoded accordingly; nothing needs converting beforehand.
+
+## Testing against real dictionaries
+
+The in-tree tests run against project-authored fixtures only. An opt-in test class, `HunspellRealDictionaryTest`, additionally checks everyday morphology against published dictionaries when pointed at a directory of `<name>.aff`/`<name>.dic` pairs (each test skips when its pair is absent):
+
+```
+./mvnw test -pl opennlp-core/opennlp-runtime -Dtest=HunspellRealDictionaryTest \
+    -Dopennlp.hunspell.dict.dir=/tmp/hunspell-dicts
+```
+
+## What the engine supports
+
+Supported affix features: `PFX` and `SFX` rules with strip strings, character-class conditions, cross-product combination of one prefix with one suffix, twofold suffixes through continuation classes, `FLAG` modes `char`, `UTF-8`, `long`, and `num`, the `AF` flag alias table, the `SET` encoding declaration, compound decomposition under `COMPOUNDFLAG`, the positional `COMPOUNDBEGIN`/`COMPOUNDMIDDLE`/`COMPOUNDEND` flags, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBIDFLAG`, and the `CHECKCOMPOUNDDUP`/`CHECKCOMPOUNDCASE`/`CHECKCOMPOUNDTRIPLE` declarations (compound parts stand on their entries alone or on an entry plus one affix, the zero and dash suffixes dictionaries position linking forms with included), the blocking flags `NEEDAFFIX` (alias `PSEUDOROOT`), `ONLYINCOMPOUND`, and `FORBIDDENWORD`, which keep virtual stems, compound-only parts, and forbidden words out of the reported analyses, and `CIRCUMFIX`, which binds marked prefix and suffix halves to one another as in the German `ge...t` participle, and the `FULLSTRIP` declaration, without which a rule that strips a whole stem is not applied, matching Hunspell. Directives that would change stems when ignored (`ICONV`, `OCONV`, `COMPLEXPREFIXES`, `COMPOUNDRULE`, `IGNORE`, `KEEPCASE`) fail at load time. Cosmetic tables such as `REP`, `MAP`, and `KEY` are skipped, so analyses that would need them are missed rather than invented. A malformed `.aff` file fails loudly at load time with the offending line number in the message. Each affix or dictionary stream is rejected when it exceeds `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB).

opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/AffixCondition.java

@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer.hunspell;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * One parsed affix condition: a fixed-length sequence of literal code points and
+ * bracketed character classes, matched with a single scan and no regular expressions.
+ * A suffix condition anchors at the end of the candidate stem, a prefix condition at
+ * its start; the condition {@code .} matches everything. Positions are Unicode code
+ * points so supplementary characters agree with {@code FLAG UTF-8} flag reading.
+ */
+final class AffixCondition {
+
+  /** The shared instance for the condition {@code .}, which accepts every stem. */
+  private static final AffixCondition ANY = new AffixCondition(new int[0][], null, true);
+
+  /** Per position: the accepted code points, or {@code null} for any code point. */
+  private final int[][] accepted;
+  /** Per position with a class: whether the class is negated; {@code null} rows unused. */
+  private final boolean[] negated;
+  /** Whether the owning rule is a suffix rule, which anchors the condition at the end. */
+  private final boolean suffix;
+
+  /**
+   * Initializes the condition.
+   *
+   * @param accepted The accepted code points per position.
+   * @param negated The negation marker per position.
+   * @param suffix Whether the owning rule is a suffix rule.
+   */
+  private AffixCondition(int[][] accepted, boolean[] negated, boolean suffix) {
+    this.accepted = accepted;
+    this.negated = negated;
+    this.suffix = suffix;
+  }
+
+  /**
+   * Parses a condition field. Each pattern position is a literal code point, a
+   * {@code .} matching any code point, or a bracketed class such as {@code [sx]}; a
+   * class starting with {@code ^} is negated and matches any code point outside it.
+   *
+   * @param pattern The condition text from the affix rule.
+   * @param suffix Whether the owning rule is a suffix rule.
+   * @param lineNumber The affix file line, for error messages.
+   * @return The parsed condition. Never {@code null}.
+   * @throws IOException Thrown if a character class is unterminated.
+   */
+  static AffixCondition parse(String pattern, boolean suffix, int lineNumber)
+      throws IOException {
+    if (".".equals(pattern)) {
+      return ANY;
+    }
+    final List<int[]> positions = new ArrayList<>();
+    final List<Boolean> negations = new ArrayList<>();
+    int i = 0;
+    while (i < pattern.length()) {
+      final int codePoint = pattern.codePointAt(i);
+      if (codePoint == '[') {
+        final int end = pattern.indexOf(']', i + 1);
+        if (end < 0) {
+          throw new IOException("unterminated character class at line " + lineNumber);
+        }
+        String members = pattern.substring(i + 1, end);
+        boolean negate = false;
+        if (members.startsWith("^")) {
+          negate = true;
+          members = members.substring(1);
+        }
+        positions.add(toCodePoints(members));
+        negations.add(negate);
+        i = end + 1;
+      } else if (codePoint == '.') {
+        positions.add(null);
+        negations.add(false);
+        i++;
+      } else {
+        positions.add(new int[] {codePoint});
+        negations.add(false);
+        i += Character.charCount(codePoint);
+      }
+    }
+    final int[][] accepted = positions.toArray(new int[0][]);
+    final boolean[] negated = new boolean[accepted.length];
+    for (int p = 0; p < negated.length; p++) {
+      negated[p] = negations.get(p);
+    }
+    return new AffixCondition(accepted, negated, suffix);
+  }
+
+  /**
+   * Collects the code points of a character-class body.
+   *
+   * @param members The class body text.
+   * @return The code points in order. Never {@code null}.
+   */
+  private static int[] toCodePoints(String members) {
+    final int[] codePoints = new int[members.codePointCount(0, members.length())];
+    int i = 0;
+    int out = 0;
+    while (i < members.length()) {
+      final int codePoint = members.codePointAt(i);
+      codePoints[out++] = codePoint;
+      i += Character.charCount(codePoint);
+    }
+    return codePoints;
+  }
+
+  /**
+   * Tests a candidate stem against the condition at its anchored side: the last
+   * positions of the stem for a suffix condition, the first positions for a prefix
+   * condition. A stem shorter than the condition never matches. Length is in code
+   * points.
+   *
+   * @param stem The candidate stem after affix removal and strip restoration.
+   * @return {@code true} if the stem satisfies the condition.
+   */
+  boolean matches(String stem) {
+    if (accepted.length == 0) {
+      return true;
+    }
+    final int stemPoints = stem.codePointCount(0, stem.length());
+    if (stemPoints < accepted.length) {
+      return false;
+    }
+    int offset = suffix ? stem.offsetByCodePoints(0, stemPoints - accepted.length) : 0;
+    for (int p = 0; p < accepted.length; p++) {
+      final int[] members = accepted[p];
+      final int codePoint = stem.codePointAt(offset);
+      offset += Character.charCount(codePoint);
+      if (members == null) {
+        continue;
+      }
+      boolean member = false;
+      for (final int candidate : members) {
+        if (candidate == codePoint) {
+          member = true;
+          break;
+        }
+      }
+      if (member == negated[p]) {
+        return false;
+      }
+    }
+    return true;
+  }
+}