OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese

dev/README-mecab-dictionaries.md

@@ -0,0 +1,104 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+# CJK dictionaries for the lattice tokenizer
+
+The lattice tokenizer (`opennlp.tools.tokenize.lattice`) segments Japanese and Korean over a MeCab-format dictionary, and the unigram segmenter handles Chinese over a plain word-frequency lexicon. Apache OpenNLP bundles no dictionary data. Download a dictionary from its project and read the license file inside the archive before use.
+
+## Known MeCab-format dictionary projects
+
+| Dictionary | Language | Encoding |
+|---|---|---|
+| IPADIC 2.7.0 | Japanese | EUC-JP |
+| mecab-ko-dic 2.1.1 | Korean | UTF-8 |
+
+Download a release archive directly from the dictionary project. The installer reads
+gzip-compressed ustar archives.
+
+The installer extracts only the dictionary payload: the `*.csv` and `*.def` files a
+`MecabDictionary` reads, plus the `dicrc` configuration file the distributions ship
+alongside them. It flattens the entries into the target directory, and by the same
+flattening makes it impossible for an archive path to escape that directory. The
+returned count is the number of dictionary files extracted. Tar headers are
+checksum-validated, and files are staged on the target filesystem before publication.
+The installer does not replace files already present in the target directory.
+
+## Install a local archive
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.MecabDictionaryInstaller;
+
+Path localArchive = Path.of("mecab-ipadic-2.7.0-20070801.tar.gz");
+int files = MecabDictionaryInstaller.install(localArchive.toUri(), Path.of("ipadic"));
+```
+
+`MecabDictionaryInstaller.install` accepts trusted local `file:` URIs. Remote download
+and verification are outside this API.
+
+## Size budgets for larger dictionaries
+
+Extraction is bounded so a crafted archive cannot fill the disk. By default one
+extracted tar entry is limited to 512 MiB and the total extracted payload to 2 GiB.
+IPADIC and mecab-ko-dic fit within these limits. For larger dictionaries, such as
+UniDic, raise the limits at JVM startup:
+
+```bash
+-Dopennlp.install.max.entry.bytes=4294967296 \
+-Dopennlp.install.max.total.bytes=8589934592
+```
+
+Values must be positive byte counts; anything absent or invalid falls back to the
+default.
+
+## Load and tokenize
+
+`MecabDictionary.load(Path)` assumes UTF-8. IPADIC needs the two-argument overload:
+
+```java
+import java.nio.charset.Charset;
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.LatticeTokenizer;
+import opennlp.tools.tokenize.lattice.MecabDictionary;
+
+MecabDictionary dictionary =
+    MecabDictionary.load(Path.of("ipadic"), Charset.forName("EUC-JP"));
+LatticeTokenizer tokenizer = new LatticeTokenizer(dictionary);
+// "Tokyo-to ni iku" (go to the Tokyo metropolis), escaped to keep this file ASCII
+String[] tokens = tokenizer.tokenize("\u6771\u4EAC\u90FD\u306B\u884C\u304F");
+```
+
+For a UTF-8 dictionary such as mecab-ko-dic, `MecabDictionary.load(Path.of("ko-dic"))`
+is enough. Loaded dictionaries and tokenizers are immutable and safe to share between
+threads, so load once and reuse.
+
+## Chinese: the unigram segmenter needs only a frequency lexicon
+
+`opennlp.tools.tokenize.lattice.UnigramSegmenter` does not use MeCab dictionaries. It
+loads a plain text lexicon, one entry per line: the word, its count, and optionally a
+tag, separated by whitespace. Any word-frequency list you have the rights to use works:
+
+```java
+import java.nio.file.Path;
+import opennlp.tools.tokenize.lattice.UnigramSegmenter;
+
+UnigramSegmenter segmenter = UnigramSegmenter.load(Path.of("words.txt"));
+// "wo laidao Beijing Tian'anmen" (I arrive at Beijing Tiananmen), escaped as above
+String[] tokens = segmenter.tokenize("\u6211\u6765\u5230\u5317\u4EAC\u5929\u5B89\u95E8");
+```
+
+As with the dictionaries, the lexicon has its own license; no lexicon data is bundled.

opennlp-api/src/main/java/opennlp/tools/tokenize/lattice/CategoryTable.java

@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.tokenize.lattice;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.tokenize.lattice.MecabDictionary.Category;
+
+/**
+ * The {@code char.def} code point to category name mapping over the Unicode
+ * code point range.
+ *
+ * <p>The Basic Multilingual Plane is stored in a directly indexed array. The
+ * supplementary planes are stored as a sorted, non-overlapping range table searched by
+ * binary search, because dictionaries map them in a handful of large blocks.</p>
+ */
+final class CategoryTable {
+
+  private final Category[] bmp;
+  private final int[] rangeStart;
+  private final int[] rangeEnd;
+  private final Category[] rangeCategory;
+
+  private CategoryTable(Category[] bmp, int[] rangeStart, int[] rangeEnd,
+      Category[] rangeCategory) {
+    this.bmp = bmp;
+    this.rangeStart = rangeStart;
+    this.rangeEnd = rangeEnd;
+    this.rangeCategory = rangeCategory;
+  }
+
+  /**
+   * Looks up the category a {@code char.def} mapping gives a code point. The table
+   * contains the {@link Category} instances themselves, and two code points of one
+   * category share one instance, so categories may be compared by identity.
+   *
+   * @param codePoint The code point to classify.
+   * @return The category, or {@code null} when no mapping covers the code point.
+   */
+  Category categoryOf(int codePoint) {
+    if (codePoint <= Character.MAX_VALUE) {
+      return bmp[codePoint];
+    }
+    int low = 0;
+    int high = rangeStart.length - 1;
+    while (low <= high) {
+      final int middle = (low + high) >>> 1;
+      if (codePoint < rangeStart[middle]) {
+        high = middle - 1;
+      } else if (codePoint > rangeEnd[middle]) {
+        low = middle + 1;
+      } else {
+        return rangeCategory[middle];
+      }
+    }
+    return null;
+  }
+
+  private static final String CHARACTER_DEFINITION_FILE = "char.def";
+
+  /**
+   * Collects {@code char.def} mappings in file order and builds a
+   * {@link CategoryTable}, giving a later mapping precedence over an earlier one that
+   * covers the same code point, which is what direct indexing does for the BMP.
+   */
+  static final class Builder {
+
+    private final String[] bmp = new String[Character.MAX_VALUE + 1];
+    private final List<int[]> bounds = new ArrayList<>();
+    private final List<String> names = new ArrayList<>();
+
+    /**
+     * Records one inclusive code point range's category.
+     *
+     * @param from The first code point of the range.
+     * @param to The last code point of the range, inclusive.
+     * @param category The category name to give the range. Must not be {@code null}.
+     */
+    void map(int from, int to, String category) {
+      for (int c = from; c <= Math.min(to, Character.MAX_VALUE); c++) {
+        bmp[c] = category;
+      }
+      if (to > Character.MAX_VALUE) {
+        bounds.add(new int[] {Math.max(from, Character.MAX_VALUE + 1), to});
+        names.add(category);
+      }
+    }
+
+    /**
+     * Builds the lookup table from the recorded mappings.
+     *
+     * @param categories The categories the {@code char.def} category section defined,
+     *                   keyed by name.
+     * @return The table. Not {@code null}.
+     * @throws IOException Thrown if a mapping names a category that was not defined.
+     */
+    CategoryTable build(Map<String, Category> categories) throws IOException {
+      // Cut the supplementary ranges at every boundary they introduce, so that each
+      // resulting elementary interval is covered by a single winning range and the
+      // table stays sorted and non-overlapping for binary search.
+      final int[] edges = new int[bounds.size() * 2];
+      for (int i = 0; i < bounds.size(); i++) {
+        edges[i * 2] = bounds.get(i)[0];
+        edges[i * 2 + 1] = bounds.get(i)[1] + 1;
+      }
+      Arrays.sort(edges);
+      final List<int[]> intervals = new ArrayList<>();
+      final List<String> winners = new ArrayList<>();
+      for (int i = 0; i < edges.length - 1; i++) {
+        if (edges[i] == edges[i + 1]) {
+          continue;
+        }
+        final String winner = lastCovering(edges[i]);
+        if (winner == null) {
+          continue;
+        }
+        final int previous = intervals.size() - 1;
+        if (previous >= 0 && intervals.get(previous)[1] == edges[i] - 1
+            && winners.get(previous).equals(winner)) {
+          intervals.get(previous)[1] = edges[i + 1] - 1;
+        } else {
+          intervals.add(new int[] {edges[i], edges[i + 1] - 1});
+          winners.add(winner);
+        }
+      }
+      final int[] starts = new int[intervals.size()];
+      final int[] ends = new int[intervals.size()];
+      for (int i = 0; i < intervals.size(); i++) {
+        starts[i] = intervals.get(i)[0];
+        ends[i] = intervals.get(i)[1];
+      }
+      final Category[] resolvedBmp = new Category[bmp.length];
+      for (int c = 0; c < bmp.length; c++) {
+        if (bmp[c] != null) {
+          resolvedBmp[c] = resolve(bmp[c], categories, c);
+        }
+      }
+      final Category[] resolvedRanges = new Category[winners.size()];
+      for (int i = 0; i < winners.size(); i++) {
+        resolvedRanges[i] = resolve(winners.get(i), categories, starts[i]);
+      }
+      return new CategoryTable(resolvedBmp, starts, ends, resolvedRanges);
+    }
+
+    /**
+     * Resolves a mapped category name against the defined categories. A mapping to an
+     * undefined category fails at load and names the offending code point.
+     *
+     * @param name The category name a mapping line gave.
+     * @param categories The defined categories, keyed by name.
+     * @param codePoint A code point the mapping covers, for the error message.
+     * @return The resolved category. Not {@code null}.
+     * @throws IOException Thrown if no category of that name was defined.
+     */
+    private Category resolve(String name, Map<String, Category> categories,
+        int codePoint) throws IOException {
+      final Category category = categories.get(name);
+      if (category == null) {
+        throw new IOException(String.format(
+            CHARACTER_DEFINITION_FILE + " maps U+%04X to the undefined category %s",
+            codePoint, name));
+      }
+      return category;
+    }
+
+    /**
+     * Finds the category of the last recorded range covering a code point.
+     *
+     * @param codePoint The code point to look up.
+     * @return The category name, or {@code null} when no recorded range covers it.
+     */
+    private String lastCovering(int codePoint) {
+      for (int i = bounds.size() - 1; i >= 0; i--) {
+        final int[] range = bounds.get(i);
+        if (codePoint >= range[0] && codePoint <= range[1]) {
+          return names.get(i);
+        }
+      }
+      return null;
+    }
+  }
+}