OPENNLP-1932: Match model finder wildcards without regular expressions - #1278
Draft
krickert wants to merge 20 commits into
Draft
OPENNLP-1932: Match model finder wildcards without regular expressions#1278krickert wants to merge 20 commits into
krickert wants to merge 20 commits into
Conversation
…in DefaultLemmatizerContextGenerator
…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.
… checks in FeatureGeneratorUtil Add pinning tests for the capPeriod accept and reject sides.
…enPatternFeatureGenerator Add a pinning test that non-letter sub-tokens do not produce st= features.
… char scan Matches (.+)-\w+ semantics: group(1) is everything before the last hyphen, the hyphen must not be at index 0, and the suffix must be non-empty word chars. Add pinning tests for outcomes without hyphen, hyphen at index 0, empty suffix, non-word suffix, and the normal accept case.
…n BrownCluster Replicates String.split(\t) semantics, including dropped trailing empty fields.
…explicit char scans in TokenSampleStream splitOnWhitespace replicates String.split(\\s+): a leading whitespace run yields one empty leading field, runs collapse, and trailing empty fields are dropped.
…mojiCharSequenceNormalizer The replaced pattern contains a high surrogate range, so the regex engine matches whole code points in the flattened range [U+D83C, U+10FC00]. The replacement scans code points, collapses each maximal matching run into a single space, and copies non-matching code points verbatim. Add pinning tests for unpaired surrogates, BMP chars above U+D83C, and supplementary code points beyond U+10FC00.
…xplicit scans in ConlluStream
splitOnHyphen replicates String.split("-"): every hyphen is a boundary,
empty fields between consecutive hyphens are kept, and trailing empty
fields are dropped.
extractTextLang replicates find() of text_([a-z]{2,3}): the first
occurrence of "text_" followed by two to three ASCII lowercase letters,
preferring three.
…ss scans in ParserTool
The two replaceAll passes are replicated by two cursor passes with the
same leftmost-first resume-after-match semantics, which matters for
overlapping pairs such as "x((" or "((a)(b))": a pair starting at the
second char of a match is only reconsidered by the second pass.
…cans in DownloadUtil parseChecksum now scans to the first ASCII whitespace character, replicating split(\s)[0] on the trimmed content. extractLinks replicates find() of the <a href="(.*?)">(.*?)</a> pattern with CASE_INSENSITIVE and DOTALL flags: the href value ends at the first "> and the first case-insensitive </a> closes the match, so nested link markup is swallowed by the outer match.
…meric patterns with explicit scans in ADNameSampleStream
splitOnWhitespace and splitOnUnderscores replicate run-based splitting: a
leading separator run yields one empty leading field, trailing empty
fields are dropped, and an all-separator input yields no fields.
matchHyphenatedToken replicates the three-branch hyphen pattern at code
point granularity, isAlphaNumeric replicates ^[\p{L}\p{Nd}]+$ via
Character.isLetter and Character.isDigit, and tagContent replicates
matches() of <(NER:)?(.*?)> including its optional NER: prefix.
…OSSampleStream
replaceWhitespaceWithEquals replicates replaceAll("=") of the \s+
pattern: every run of ASCII whitespace, including leading and trailing
runs, is replaced by a single equals sign.
…tenceSampleStream parseTextAndParagraph replicates matches() of the ^(?:[a-zA-Z\-]*(\d+)).*?p=(\d+).* pattern: after the optional ASCII letters and hyphens, the text id is the first ASCII digit run and the paragraph id is the digit run after the first "p=" that is followed by at least one digit.
…entenceStream replaceGuillemetPunctuation replicates replaceAll of the »\s+ punct patterns: every run of ASCII whitespace between » and the punctuation character is removed. parsePunctuationLine replicates matches() of the ^(=*)(\W+)$ pattern: the line consists of leading equals signs followed by one or more non-word characters, where a word character is an ASCII letter, digit, or underscore. A line of only equals signs matches, with the last equals sign as lexeme.
Adds StringUtil.isAsciiWhitespace, splitOnAsciiWhitespace, containsAsciiUpperCase, and containsAsciiDigit and removes the copies from the AD streams, the English TokenSampleStream, DownloadUtil, and the POS and lemmatizer context generators. NameFinderME.extractNameType delegates to BioCodec. Cases the new tests found first: the TokenSampleStream split returned one empty token for a whitespace-only line where the original split returned none, and matchHyphenatedToken accepted a single hyphen. BrownCluster.splitTabs now removes all trailing empty fields, as String.split does. Each helper has a test, parameterized where the inputs are a table, with the reject side and the edge cases: empty input, leading and trailing separators, non-ASCII spaces and digits, and supplementary-plane characters. Helpers only called from instance methods are no longer static; the block comments on the helpers are now Javadoc that states the behavior.
This was referenced Sep 7, 2026
…ssPathModelFinder The fallback that reads java.class.path split the value with two compiled Patterns, ";" on Windows and ":" elsewhere. A package-private splitClassPath now walks the string once and cuts at the separator character, keeping the String.split result for that separator: a leading separator gives one empty first element, empty elements between consecutive separators stay, trailing empty elements are dropped, separator-only input gives an empty array, and empty input gives a single empty element. The test table covers those cases for both separators and checks each row against String.split. The module gains the managed junit-jupiter-params test dependency for the table.
AbstractClassPathModelFinder turned a wildcard such as "*opennlp-models-*" into a regular expression by escaping "." and rewriting "*" to ".*" and "?" to ".", then compiled it and called matches() on the file part of each URL. The package-private GlobMatcher now walks the wildcard and the input by code point: "*" absorbs any run of characters including none, "?" takes exactly one, and every other character stands for itself. As with the old ".", neither wildcard crosses a line feed, carriage return, next line, line separator, or paragraph separator, and the whole input must be covered. The protected asRegex and matchesPattern(URL, Pattern) are replaced by matchesWildcard(URL, String); DirectoryModelFinder and SimpleClassPathModelFinder keep the plain wildcard strings instead of compiled patterns, and the pattern cache in DirectoryModelFinder goes away because there is nothing left to compile. Characters the old code never escaped, such as parentheses, brackets, braces, "+", "|", "^", "$", and backslash, had regular expression meaning before or made Pattern.compile fail; they are now literal. GlobMatcherTest covers the accept and reject sides, dots, those characters, each line terminator, and supplementary-plane characters.
DirectoryModelFinder had no test. The new test copies the opennlp-models jars from the test class path into a temporary directory one level below the scanned root, runs the shared finder assertions against it, and checks the non-recursive mode, a narrowing jar prefix, an unknown prefix, and the null directory rejection.
krickert
force-pushed
the
OPENNLP-1932-model-resolver-glob
branch
from
September 7, 2026 13:18
5d7315e to
8913d04
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #1275, and independent of the other parts of the epic. It can be reviewed and merged in any order relative to them. This diff carries the #1275 commits until that one merges; the commits of this change alone: ai-pipestream/opennlp@OPENNLP-1928-regex-removal-trivial...OPENNLP-1932-model-resolver-glob
AbstractClassPathModelFinder,DirectoryModelFinder, andSimpleClassPathModelFinderconverted wildcard strings to regular expressions and split the class path with compiled patterns. A package-privateGlobMatchernow matches*(any run) and?(one code point) directly, anchored on the whole name and, as before, not crossing line terminators; the class path split is a character scan withString.splitsemantics.One intended behavior change: characters the old conversion never quoted (
( ) [ ] { } + | ^ $ \) had regex meaning or threwPatternSyntaxException; they now stand for themselves.API: the protected
asRegexandmatchesPattern(URL, Pattern)on the public abstractAbstractClassPathModelFinderare replaced by protectedmatchesWildcard(URL, String). No code in the repository used them.Verification: 39 globs by 46 inputs plus a 300,000-pair fuzz compared with the old regex path; all differences come from the unquoted characters above.
GlobMatcherTestadds 64 cases and the module gains its firstDirectoryModelFinderTest. opennlp-model-resolver: 119 tests in each of its three surefire executions, with checkstyle, offline,-Dopennlp.forkCount=1.OPENNLP-1932