OPENNLP-1909: General verified installer for user-supplied third-party resources

dev/README-hunspell-dictionaries.md

@@ -17,15 +17,21 @@
 
 # Hunspell dictionaries for the affix stemmer
 
-The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) implements the documented Hunspell dictionary format: a `.dic` word list plus its `.aff` affix companion, both supplied by the user. Apache OpenNLP bundles no dictionary data; whichever dictionary you download, its license is stated in the readme shipped alongside it.
+The Hunspell stemmer (`opennlp.tools.stemmer.hunspell`) reads a user-supplied
+`.dic` word list and its `.aff` affix file. Apache OpenNLP bundles no dictionary
+data. The dictionary's readme states its license.
 
 ## Where dictionaries come from
 
-The LibreOffice project maintains a large collection of Hunspell dictionaries, one directory per language, at `github.com/LibreOffice/dictionaries`. Licenses differ per dictionary, which is why nothing is bundled: for example, the `en_US` dictionary derives from SCOWL and states its terms in `README_en_US.txt` in the same directory. Many other sources work too; the engine only cares that the pair follows the Hunspell format.
+The LibreOffice project maintains Hunspell dictionaries by language at
+`github.com/LibreOffice/dictionaries`. Each dictionary has a separate license.
+For example, SCOWL is the source for the `en_US` dictionary, with terms in
+`README_en_US.txt`. Other sources can be used when the `.aff` and `.dic` files
+follow the Hunspell format.
 
 OpenNLP does not ship a URL catalog. Applications that manage downloads can keep a
 properties file with an entry id followed by `.url`, `.sha512`, and optionally
-`.filename` keys. Pin each URL to a stable release or commit.
+`.filename` keys. Use a URL for a stable release or commit.
 
 ## Option A: application catalog
 
@@ -54,7 +60,7 @@ The download test uses local file URLs to exercise this flow without network acc
 ## Option B: your own files
 
 Fetch `.aff` / `.dic` (and the license readme) with any tool, or with
-`DownloadUtil.download(uri, path, sha512)`, then load them:
+`ResourceInstaller.install(uri, directory, sha512)`, then load them:
 
 ```java
 import java.nio.file.Path;
@@ -71,9 +77,13 @@ Stemmer stemmer = factory.newStemmer();
 CharSequence stem = stemmer.stem("workers");
 ```
 
-What `stem` evaluates to is decided by the dictionary you loaded, and this project ships no dictionary data, so no result is claimed here for `en_US`. The same load-and-stem flow is pinned by `HunspellManualExampleTest` (miniature in-memory dictionary, asserted stems for `workers` and `worker`) and by `HunspellStemmerFactoryTest#testEndToEndUsageFromFiles` (the same pair written to disk). The developer manual chapter `stemmer.xml` cites `HunspellManualExampleTest`.
+The result depends on the loaded dictionary. The in-tree manual example uses a
+small dictionary and checks that `workers` stems to `worker`.
 
-The dictionary is immutable and safe to share between threads; the factory hands out a fresh stemmer per call, so each thread takes its own from `newStemmer()`. A dictionary that declares a non-UTF-8 encoding through the `SET` directive in its `.aff` file is decoded accordingly; nothing needs converting beforehand.
+The dictionary is immutable and safe to share between threads. The factory creates a
+new stemmer for each call, so each thread can use its own instance. A dictionary that
+declares a non-UTF-8 encoding through the `SET` directive in its `.aff` file is decoded
+accordingly; no conversion is required.
 
 ## Testing against real dictionaries
 
@@ -86,4 +96,8 @@ The in-tree tests run against project-authored fixtures only. An opt-in test cla
 
 ## What the engine supports
 
-Supported affix features: `PFX` and `SFX` rules with strip strings, character-class conditions, cross-product combination of one prefix with one suffix, twofold suffixes through continuation classes, `FLAG` modes `char`, `UTF-8`, `long`, and `num`, the `AF` flag alias table, the `SET` encoding declaration, compound decomposition under `COMPOUNDFLAG`, the positional `COMPOUNDBEGIN`/`COMPOUNDMIDDLE`/`COMPOUNDEND` flags, `COMPOUNDMIN`, `COMPOUNDWORDMAX`, `COMPOUNDPERMITFLAG`, `COMPOUNDFORBIDFLAG`, and the `CHECKCOMPOUNDDUP`/`CHECKCOMPOUNDCASE`/`CHECKCOMPOUNDTRIPLE` declarations (compound parts stand on their entries alone or on an entry plus one affix, the zero and dash suffixes dictionaries position linking forms with included), the blocking flags `NEEDAFFIX` (alias `PSEUDOROOT`), `ONLYINCOMPOUND`, and `FORBIDDENWORD`, which keep virtual stems, compound-only parts, and forbidden words out of the reported analyses, and `CIRCUMFIX`, which binds marked prefix and suffix halves to one another as in the German `ge...t` participle, and the `FULLSTRIP` declaration, without which a rule that strips a whole stem is not applied, matching Hunspell. Directives that would change stems when ignored (`ICONV`, `OCONV`, `COMPLEXPREFIXES`, `COMPOUNDRULE`, `IGNORE`, `KEEPCASE`) fail at load time. Cosmetic tables such as `REP`, `MAP`, and `KEY` are skipped, so analyses that would need them are missed rather than invented. A malformed `.aff` file fails loudly at load time with the offending line number in the message. Each affix or dictionary stream is rejected when it exceeds `HunspellDictionary.MAX_STREAM_BYTES` (64 MiB).
+Supported affix features include `PFX` and `SFX` rules, continuation classes,
+compound flags, blocking flags, `CIRCUMFIX`, and `FULLSTRIP`. The parser rejects
+directives that would change stems if ignored. It skips cosmetic tables that do not
+affect stemming. Malformed files report the relevant line number. Each affix or
+dictionary stream is limited to 64 MiB.

dev/README-mecab-dictionaries.md

@@ -17,53 +17,75 @@
 
 # CJK dictionaries for the lattice tokenizer
 
-The lattice tokenizer (`opennlp.tools.tokenize.lattice`) segments Japanese and Korean over a MeCab-format dictionary, and the unigram segmenter handles Chinese over a plain word-frequency lexicon. Apache OpenNLP bundles no dictionary data. Download a dictionary from its project and read the license file inside the archive before use.
+The lattice tokenizer (`opennlp.tools.tokenize.lattice`) segments Japanese and Korean over a MeCab-format dictionary, and the unigram segmenter handles Chinese over a plain word-frequency lexicon. Apache OpenNLP bundles no dictionary data: you download a dictionary from the project of your choice, and each dictionary contains its own license. Read the license file inside the archive before use.
 
 ## Known MeCab-format dictionary projects
 
-| Dictionary | Language | Encoding |
-|---|---|---|
-| IPADIC 2.7.0 | Japanese | EUC-JP |
-| mecab-ko-dic 2.1.1 | Korean | UTF-8 |
+| Catalog id | Dictionary | Language | Encoding |
+|---|---|---|---|
+| `mecab.ipadic` | IPADIC 2.7.0 | Japanese | EUC-JP |
+| `mecab.ko-dic` | mecab-ko-dic 2.1.1 | Korean | UTF-8 |
 
-Download a release archive directly from the dictionary project. The installer reads
-gzip-compressed ustar archives.
+Example download URLs and SHA-512 digests for those ids live in the test resource
+`opennlp-core/opennlp-runtime/src/test/resources/opennlp/tools/util/dictionary-catalog.properties`. Both archives are
+gzip-compressed tars; `MecabDictionaryInstaller` reads the ustar, pax, and GNU
+formats through `ResourceInstaller`.
 
 The installer extracts only the dictionary payload: the `*.csv` and `*.def` files a
 `MecabDictionary` reads, plus the `dicrc` configuration file the distributions ship
-alongside them. It flattens the entries into the target directory, and by the same
-flattening makes it impossible for an archive path to escape that directory. The
-returned count is the number of dictionary files extracted. Tar headers are
-checksum-validated, and files are staged on the target filesystem before publication.
-The installer does not replace files already present in the target directory.
+alongside them. `ResourceInstaller` rejects paths outside the staging directory,
+then `MecabDictionaryInstaller` flattens the selected files into the target. The
+returned value is the number of dictionary files installed.
 
-## Install a local archive
+## Option A: opt-in catalog install
+
+Applications supply the catalog. Catalog URLs are inactive until you set
+`-Dopennlp.download.remote=true` or the equivalent system property in code.
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.MecabDictionaryInstaller;
+import opennlp.tools.util.DictionaryCatalog;
+
+// JVM flag: -Dopennlp.download.remote=true
+DictionaryCatalog catalog = DictionaryCatalog.load(catalogProperties);
+int files = MecabDictionaryInstaller.installFromCatalog(
+    catalog, "mecab.ipadic", Path.of("ipadic"));
+```
+
+## Option B: your own URL and digest
 
 ```java
+import java.net.URI;
 import java.nio.file.Path;
 import opennlp.tools.tokenize.lattice.MecabDictionaryInstaller;
 
-Path localArchive = Path.of("mecab-ipadic-2.7.0-20070801.tar.gz");
-int files = MecabDictionaryInstaller.install(localArchive.toUri(), Path.of("ipadic"));
+String expectedSha512 = "..."; // the 128-hex SHA-512 of the archive
+int files = MecabDictionaryInstaller.install(
+    URI.create("https://example.example/dict.tar.gz"),
+    Path.of("dict"),
+    expectedSha512);
 ```
 
-`MecabDictionaryInstaller.install` accepts trusted local `file:` URIs. Remote download
-and verification are outside this API.
+A local `file:` URI may omit the digest:
+`MecabDictionaryInstaller.install(localArchive.toUri(), targetDirectory)`.
+HTTP and HTTPS sources require a digest. Other URI schemes are rejected.
 
 ## Size budgets for larger dictionaries
 
-Extraction is bounded so a crafted archive cannot fill the disk. By default one
-extracted tar entry is limited to 512 MiB and the total extracted payload to 2 GiB.
-IPADIC and mecab-ko-dic fit within these limits. For larger dictionaries, such as
-UniDic, raise the limits at JVM startup:
+Fetching and unpacking go through `ResourceInstaller` and are bounded so a crafted
+archive cannot fill the disk: by default one download is capped at 1 GiB, the
+unpacked payload at 4 GiB, and the archive at 100000 entries. IPADIC and
+mecab-ko-dic fit comfortably. For larger dictionaries, such as UniDic, raise the
+limits at JVM startup:
 
 ```bash
--Dopennlp.install.max.entry.bytes=4294967296 \
--Dopennlp.install.max.total.bytes=8589934592
+-Dopennlp.download.max.bytes=4294967296 \
+-Dopennlp.install.max.total.bytes=8589934592 \
+-Dopennlp.install.max.entries=200000
 ```
 
-Values must be positive byte counts; anything absent or invalid falls back to the
-default.
+Missing, invalid, and nonpositive property values use the default limits.
 
 ## Load and tokenize
 
@@ -101,4 +123,4 @@ UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt"));
 String[] tokens = segmenter.tokenize("\u6211\u6765\u5230\u5317\u4EAC\u5929\u5B89\u95E8");
 ```
 
-As with the dictionaries, the lexicon has its own license; no lexicon data is bundled.
+The lexicon archive includes its license; OpenNLP bundles no data.

opennlp-api/src/main/java/opennlp/tools/util/ResourceLimits.java

@@ -58,22 +58,6 @@ public final class ResourceLimits {
   public static final int MAX_MATRIX_CELLS =
       initLimit(MAX_MATRIX_CELLS_PROPERTY, 134_217_728);
 
-  /** System property for the maximum size of one extracted archive entry. */
-  public static final String MAX_ARCHIVE_ENTRY_BYTES_PROPERTY =
-      "opennlp.install.max.entry.bytes";
-
-  /** Maximum size of one extracted archive entry, 512 MiB by default. */
-  public static final long MAX_ARCHIVE_ENTRY_BYTES =
-      initLimit(MAX_ARCHIVE_ENTRY_BYTES_PROPERTY, 512L * 1024 * 1024);
-
-  /** System property for the maximum total size extracted from one archive. */
-  public static final String MAX_ARCHIVE_TOTAL_BYTES_PROPERTY =
-      "opennlp.install.max.total.bytes";
-
-  /** Maximum total size extracted from one archive, 2 GiB by default. */
-  public static final long MAX_ARCHIVE_TOTAL_BYTES =
-      initLimit(MAX_ARCHIVE_TOTAL_BYTES_PROPERTY, 2L * 1024 * 1024 * 1024);
-
   private ResourceLimits() {
   }
 

opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/hunspell/HunspellDictionaryDownload.java

@@ -32,20 +32,24 @@
  */
 public final class HunspellDictionaryDownload {
 
+  /** Prevents construction of this utility class. */
   private HunspellDictionaryDownload() {
   }
 
   /**
    * Downloads the cataloged {@code .aff}, {@code .dic}, and readme files for
-   * {@code dictionaryId} into {@code targetDirectory}.
+   * {@code dictionaryId} into {@code targetDirectory}. Each file uses its configured
+   * name or source name, for example {@code en_US.aff}. Existing target files are not
+   * replaced, so they must be removed before refreshing a dictionary.
    *
    * @param catalog The application-supplied catalog. Must not be {@code null}.
    * @param dictionaryId The catalog dictionary name, for example {@code en_US}.
    *                     Must not be {@code null}.
    * @param targetDirectory The directory to write into; created when absent. Must not
    *                        be {@code null}.
    * @throws IOException Thrown if remote downloads are disabled, a catalog entry is
-   *         missing, or verification fails.
+   *         missing, verification fails, or the target already contains one of the
+   *         files.
    * @throws IllegalArgumentException Thrown if a parameter is {@code null}.
    */
   public static void downloadFromCatalog(DictionaryCatalog catalog, String dictionaryId,
@@ -60,34 +64,11 @@ public static void downloadFromCatalog(DictionaryCatalog catalog, String diction
       throw new IllegalArgumentException("targetDirectory must not be null");
     }
     final String prefix = "hunspell." + dictionaryId;
-    download(catalog, prefix + HunspellDictionary.AFFIX_FILE_SUFFIX, targetDirectory);
-    download(catalog, prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX, targetDirectory);
+    catalog.install(prefix + HunspellDictionary.AFFIX_FILE_SUFFIX, targetDirectory);
+    catalog.install(prefix + HunspellDictionary.DICTIONARY_FILE_SUFFIX, targetDirectory);
     final String readmeId = prefix + ".readme";
     if (catalog.ids().contains(readmeId)) {
-      download(catalog, readmeId, targetDirectory);
+      catalog.install(readmeId, targetDirectory);
     }
   }
-
-  /**
-   * Downloads one catalog entry into {@code targetDirectory}, named by the entry's
-   * preferred file name or, when absent, by the last segment of its URI path.
-   *
-   * @param catalog The catalog holding {@code id}.
-   * @param id The catalog entry id.
-   * @param targetDirectory The directory to write into.
-   * @throws IOException Thrown if remote downloads are disabled, the entry is missing,
-   *         or the download fails verification.
-   */
-  private static void download(DictionaryCatalog catalog, String id, Path targetDirectory)
-      throws IOException {
-    final DictionaryCatalog.Entry entry = catalog.get(id);
-    final String filename;
-    if (entry.filename() != null) {
-      filename = entry.filename();
-    } else {
-      final String path = entry.uri().getPath();
-      filename = path.substring(path.lastIndexOf('/') + 1);
-    }
-    catalog.download(id, targetDirectory.resolve(filename));
-  }
 }