OPENNLP-1931: Scan the opennlp-dl JSON vocabulary and id2label without patterns - #1277
Draft
krickert wants to merge 20 commits into
Draft
OPENNLP-1931: Scan the opennlp-dl JSON vocabulary and id2label without patterns#1277krickert wants to merge 20 commits into
krickert wants to merge 20 commits into
Conversation
…in DefaultLemmatizerContextGenerator
…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.
… checks in FeatureGeneratorUtil Add pinning tests for the capPeriod accept and reject sides.
…enPatternFeatureGenerator Add a pinning test that non-letter sub-tokens do not produce st= features.
… char scan Matches (.+)-\w+ semantics: group(1) is everything before the last hyphen, the hyphen must not be at index 0, and the suffix must be non-empty word chars. Add pinning tests for outcomes without hyphen, hyphen at index 0, empty suffix, non-word suffix, and the normal accept case.
…n BrownCluster Replicates String.split(\t) semantics, including dropped trailing empty fields.
…explicit char scans in TokenSampleStream splitOnWhitespace replicates String.split(\\s+): a leading whitespace run yields one empty leading field, runs collapse, and trailing empty fields are dropped.
…mojiCharSequenceNormalizer The replaced pattern contains a high surrogate range, so the regex engine matches whole code points in the flattened range [U+D83C, U+10FC00]. The replacement scans code points, collapses each maximal matching run into a single space, and copies non-matching code points verbatim. Add pinning tests for unpaired surrogates, BMP chars above U+D83C, and supplementary code points beyond U+10FC00.
…xplicit scans in ConlluStream
splitOnHyphen replicates String.split("-"): every hyphen is a boundary,
empty fields between consecutive hyphens are kept, and trailing empty
fields are dropped.
extractTextLang replicates find() of text_([a-z]{2,3}): the first
occurrence of "text_" followed by two to three ASCII lowercase letters,
preferring three.
…ss scans in ParserTool
The two replaceAll passes are replicated by two cursor passes with the
same leftmost-first resume-after-match semantics, which matters for
overlapping pairs such as "x((" or "((a)(b))": a pair starting at the
second char of a match is only reconsidered by the second pass.
…cans in DownloadUtil parseChecksum now scans to the first ASCII whitespace character, replicating split(\s)[0] on the trimmed content. extractLinks replicates find() of the <a href="(.*?)">(.*?)</a> pattern with CASE_INSENSITIVE and DOTALL flags: the href value ends at the first "> and the first case-insensitive </a> closes the match, so nested link markup is swallowed by the outer match.
…meric patterns with explicit scans in ADNameSampleStream
splitOnWhitespace and splitOnUnderscores replicate run-based splitting: a
leading separator run yields one empty leading field, trailing empty
fields are dropped, and an all-separator input yields no fields.
matchHyphenatedToken replicates the three-branch hyphen pattern at code
point granularity, isAlphaNumeric replicates ^[\p{L}\p{Nd}]+$ via
Character.isLetter and Character.isDigit, and tagContent replicates
matches() of <(NER:)?(.*?)> including its optional NER: prefix.
…OSSampleStream
replaceWhitespaceWithEquals replicates replaceAll("=") of the \s+
pattern: every run of ASCII whitespace, including leading and trailing
runs, is replaced by a single equals sign.
…tenceSampleStream parseTextAndParagraph replicates matches() of the ^(?:[a-zA-Z\-]*(\d+)).*?p=(\d+).* pattern: after the optional ASCII letters and hyphens, the text id is the first ASCII digit run and the paragraph id is the digit run after the first "p=" that is followed by at least one digit.
…entenceStream replaceGuillemetPunctuation replicates replaceAll of the »\s+ punct patterns: every run of ASCII whitespace between » and the punctuation character is removed. parsePunctuationLine replicates matches() of the ^(=*)(\W+)$ pattern: the line consists of leading equals signs followed by one or more non-word characters, where a word character is an ASCII letter, digit, or underscore. A line of only equals signs matches, with the last equals sign as lexeme.
Adds StringUtil.isAsciiWhitespace, splitOnAsciiWhitespace, containsAsciiUpperCase, and containsAsciiDigit and removes the copies from the AD streams, the English TokenSampleStream, DownloadUtil, and the POS and lemmatizer context generators. NameFinderME.extractNameType delegates to BioCodec. Cases the new tests found first: the TokenSampleStream split returned one empty token for a whitespace-only line where the original split returned none, and matchHyphenatedToken accepted a single hyphen. BrownCluster.splitTabs now removes all trailing empty fields, as String.split does. Each helper has a test, parameterized where the inputs are a table, with the reject side and the edge cases: empty input, leading and trailing separators, non-ASCII spaces and digits, and supplementary-plane characters. Helpers only called from instance methods are no longer static; the block comments on the helpers are now Javadoc that states the behavior.
This was referenced Sep 7, 2026
The deep-learning module reads two HuggingFace-shaped JSON files, the vocabulary and the model configuration, with regular expressions that look for a string literal, a colon with optional ASCII whitespace, and then either a digit run or another string literal. JsonScan collects the offset helpers those two readers need: the closing quote of a literal honoring backslash escapes, the closing quote of a literal that must stay on one line, a colon surrounded by whitespace, a whitespace run, and a digit run. The class is public so the doccat subpackage can reach it and is marked Internal. The tests cover accept and reject sides of every helper, including non-ASCII spaces and digits, the five line terminators, and supplementary-plane characters.
AbstractDL.loadJsonVocab used a find() loop over a pattern matching a string literal with backslash escapes, optional ASCII whitespace around a colon, and a run of ASCII digits, anywhere in the text. The loop now walks the text with JsonScan: from each quote it finds the closing quote, the colon, and the digit run; on success it records the entry and resumes after the digits, otherwise it resumes at the character after the quote exactly like the matcher did, so a quote inside a skipped literal can open the next candidate. The method is now package-private so tests feed it text directly. The added parameterized tests pin the odd inputs: a value that is not an integer is skipped, a fractional value keeps its integer prefix, escaped and unicode-escaped keys, keys spanning a line, a backslash before a line terminator, an empty key, whitespace and newlines around the colon, a non-ASCII space after the colon, and a later entry overwriting an earlier one.
DocumentCategorizerConfig.fromJson used a DOTALL pattern to cut the text between the brace after "id2label" and the first closing brace, and a second pattern to pull "key": "value" pairs out of that text with the key running to the next quote and the value, lazily, to the next quote on the same line. Both are now cursor scans over JsonScan: the key literal is located with indexOf and retried at the next occurrence when no colon and brace follow it, the content ends at the first closing brace even when that brace belongs to a nested value, and the entry loop resumes after a matched value or at the character after a quote that opened no entry. The parameterized tests pin the nested brace cut, a brace inside a value, a missing or non-object id2label, whitespace and newlines around colons, an escaped quote inside a value, a value spanning a line, an empty key, a numeric value, an overlapping key literal, and supplementary-plane keys and values.
krickert
force-pushed
the
OPENNLP-1931-dl-json-scan
branch
from
September 7, 2026 13:18
9ba4095 to
944ab39
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #1275, and independent of the other parts of the epic. It can be reviewed and merged in any order relative to them. This diff carries the #1275 commits until that one merges; the commits of this change alone: ai-pipestream/opennlp@OPENNLP-1928-regex-removal-trivial...OPENNLP-1931-dl-json-scan
AbstractDL.loadJsonVocabandDocumentCategorizerConfig.fromJsonlocated string keys and values with regular expressions. Both now use a small cursor helper,opennlp.dl.JsonScan(public, marked internal), that finds string literals honoring backslash escapes, skips ASCII whitespace, and reads digit runs.Behavior is unchanged, including the quirks of the old patterns, which the tests record: a value with a decimal fraction gives its integer prefix, a negative id is skipped, the id2label object is cut at the first closing brace even inside a nested value, and a key spanning a newline is accepted while a value with a newline is skipped.
Verification: the old patterns and the new scan were compared over the real vocabulary and configuration resources, 57 hand-written edge cases, and 400,000 generated inputs with zero differences.
JsonScanTestadds 71 cases,LoadVocabTest27,DocumentCategorizerConfigTest24. opennlp-dl: 219 tests with checkstyle, offline,-Dopennlp.forkCount=1.OPENNLP-1931