OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese - #1191

Merged

rzo1 merged 25 commits into

mainfrom

OPENNLP-1894-lattice-cjk

Sep 4, 2026

Commits on Sep 1, 2026

OPENNLP-1894: Lattice segmentation over user-supplied mecab-format dictionaries
krickert
committed
e6477c8
OPENNLP-1894: Character trie for lattice prefix search
krickert
committed
728f633
OPENNLP-1894: Frequency-driven segmentation over user-supplied lexicons
krickert
committed
127fc51
OPENNLP-1894: Usage example and edge-case tests for the lattice and unigram segmenters, corrected javadoc
krickert
committed
a32dd4b
OPENNLP-1894: Keep lattice test sources ASCII-only via Unicode escapes, add an EUC-JP loading example
krickert
committed
f5cc564
OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper
krickert
committed
c0bbc0a
OPENNLP-1894: Categorize by code point, keep unknown candidates inside their category run, and validate context ids at load
krickert
committed
80a6a2c
OPENNLP-1894: Precompute category runs, unbox the trie, and reject inexpressible dictionary values
krickert
committed
40361dc
OPENNLP-1894: Hold the lexicon in a double-array trie with frequency-recoded labels
krickert
committed
1b49eaa
OPENNLP-1894: Chain lattice nodes intrusively instead of allocating per-position lists
krickert
committed
d554f0c
OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example
krickert
committed
c14f24e
OPENNLP-1894: Apply the review-convention pass and drop unreferenced lexicon accessors
krickert
committed
2665c00
OPENNLP-1894: Trim parsed lines as Unicode whitespace and document the tokenizer overrides
krickert
committed
1fd96ae
OPENNLP-1894: Address review: complete javadoc, hoist constants, and fold fixture duplication
krickert
committed
585ff0e
OPENNLP-1894: Move lattice tokenizer types into opennlp-api
krickert
committed
4c9d187
OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges
krickert
committed
d62a140
OPENNLP-1894: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog
krickert
committed
b024809
OPENNLP-1894: Make download and extraction byte budgets overridable at startup
krickert
committed
200fcad
OPENNLP-1894: Trigger CI for the budget-override commit
krickert
committed
5a3451b
OPENNLP-1894: Address review: cite named formats and align test conventions
krickert
committed
cc55a3a
OPENNLP-1894: Reconcile the shared download test files with the sibling PR
krickert
committed
b092604
OPENNLP-1894: Add a remote-gated end-to-end test over the catalog dictionaries
krickert
committed
64ad31d
OPENNLP-1894: Bound matrix cells separately so real distributions load
krickert
committed
7725b7a
OPENNLP-1894: Extract dictionary files from the archive root only
krickert
committed
f24fbad

Commits on Sep 2, 2026

OPENNLP-1894: Address lattice tokenizer review feedback
krickert
committed
dba1353

Navigation Menu

Uh oh!

OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese - #1191

OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese#1191
rzo1 merged 25 commits into
mainfrom
OPENNLP-1894-lattice-cjk

Commits on Sep 1, 2026

OPENNLP-1894: Lattice segmentation over user-supplied mecab-format dictionaries

OPENNLP-1894: Character trie for lattice prefix search

OPENNLP-1894: Frequency-driven segmentation over user-supplied lexicons

OPENNLP-1894: Usage example and edge-case tests for the lattice and unigram segmenters, corrected javadoc

OPENNLP-1894: Keep lattice test sources ASCII-only via Unicode escapes, add an EUC-JP loading example

OPENNLP-1894: Document dictionary acquisition with a checksum-verifying download helper

OPENNLP-1894: Categorize by code point, keep unknown candidates inside their category run, and validate context ids at load

OPENNLP-1894: Precompute category runs, unbox the trie, and reject inexpressible dictionary values

OPENNLP-1894: Hold the lexicon in a double-array trie with frequency-recoded labels

OPENNLP-1894: Chain lattice nodes intrusively instead of allocating per-position lists

OPENNLP-1894: Document lattice CJK tokenization with a mirror-tested example

OPENNLP-1894: Apply the review-convention pass and drop unreferenced lexicon accessors

OPENNLP-1894: Trim parsed lines as Unicode whitespace and document the tokenizer overrides

OPENNLP-1894: Address review: complete javadoc, hoist constants, and fold fixture duplication

OPENNLP-1894: Move lattice tokenizer types into opennlp-api

OPENNLP-1894: Cap archive extract budgets and harden MeCab load edges

OPENNLP-1894: Verify dictionary downloads by SHA-512 and add an opt-in URL catalog

OPENNLP-1894: Make download and extraction byte budgets overridable at startup

OPENNLP-1894: Trigger CI for the budget-override commit

OPENNLP-1894: Address review: cite named formats and align test conventions

OPENNLP-1894: Reconcile the shared download test files with the sibling PR

OPENNLP-1894: Add a remote-gated end-to-end test over the catalog dictionaries

OPENNLP-1894: Bound matrix cells separately so real distributions load

OPENNLP-1894: Extract dictionary files from the archive root only

Commits on Sep 2, 2026

OPENNLP-1894: Address lattice tokenizer review feedback