OPENNLP-1933: Replace per-call String regex splits and replacements with scans - #1279
Draft
krickert wants to merge 26 commits into
Draft
OPENNLP-1933: Replace per-call String regex splits and replacements with scans#1279krickert wants to merge 26 commits into
krickert wants to merge 26 commits into
Conversation
…in DefaultLemmatizerContextGenerator
…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.
… checks in FeatureGeneratorUtil Add pinning tests for the capPeriod accept and reject sides.
…enPatternFeatureGenerator Add a pinning test that non-letter sub-tokens do not produce st= features.
… char scan Matches (.+)-\w+ semantics: group(1) is everything before the last hyphen, the hyphen must not be at index 0, and the suffix must be non-empty word chars. Add pinning tests for outcomes without hyphen, hyphen at index 0, empty suffix, non-word suffix, and the normal accept case.
…n BrownCluster Replicates String.split(\t) semantics, including dropped trailing empty fields.
…explicit char scans in TokenSampleStream splitOnWhitespace replicates String.split(\\s+): a leading whitespace run yields one empty leading field, runs collapse, and trailing empty fields are dropped.
…mojiCharSequenceNormalizer The replaced pattern contains a high surrogate range, so the regex engine matches whole code points in the flattened range [U+D83C, U+10FC00]. The replacement scans code points, collapses each maximal matching run into a single space, and copies non-matching code points verbatim. Add pinning tests for unpaired surrogates, BMP chars above U+D83C, and supplementary code points beyond U+10FC00.
…xplicit scans in ConlluStream
splitOnHyphen replicates String.split("-"): every hyphen is a boundary,
empty fields between consecutive hyphens are kept, and trailing empty
fields are dropped.
extractTextLang replicates find() of text_([a-z]{2,3}): the first
occurrence of "text_" followed by two to three ASCII lowercase letters,
preferring three.
…ss scans in ParserTool
The two replaceAll passes are replicated by two cursor passes with the
same leftmost-first resume-after-match semantics, which matters for
overlapping pairs such as "x((" or "((a)(b))": a pair starting at the
second char of a match is only reconsidered by the second pass.
…cans in DownloadUtil parseChecksum now scans to the first ASCII whitespace character, replicating split(\s)[0] on the trimmed content. extractLinks replicates find() of the <a href="(.*?)">(.*?)</a> pattern with CASE_INSENSITIVE and DOTALL flags: the href value ends at the first "> and the first case-insensitive </a> closes the match, so nested link markup is swallowed by the outer match.
…meric patterns with explicit scans in ADNameSampleStream
splitOnWhitespace and splitOnUnderscores replicate run-based splitting: a
leading separator run yields one empty leading field, trailing empty
fields are dropped, and an all-separator input yields no fields.
matchHyphenatedToken replicates the three-branch hyphen pattern at code
point granularity, isAlphaNumeric replicates ^[\p{L}\p{Nd}]+$ via
Character.isLetter and Character.isDigit, and tagContent replicates
matches() of <(NER:)?(.*?)> including its optional NER: prefix.
…OSSampleStream
replaceWhitespaceWithEquals replicates replaceAll("=") of the \s+
pattern: every run of ASCII whitespace, including leading and trailing
runs, is replaced by a single equals sign.
…tenceSampleStream parseTextAndParagraph replicates matches() of the ^(?:[a-zA-Z\-]*(\d+)).*?p=(\d+).* pattern: after the optional ASCII letters and hyphens, the text id is the first ASCII digit run and the paragraph id is the digit run after the first "p=" that is followed by at least one digit.
…entenceStream replaceGuillemetPunctuation replicates replaceAll of the »\s+ punct patterns: every run of ASCII whitespace between » and the punctuation character is removed. parsePunctuationLine replicates matches() of the ^(=*)(\W+)$ pattern: the line consists of leading equals signs followed by one or more non-word characters, where a word character is an ASCII letter, digit, or underscore. A line of only equals signs matches, with the last equals sign as lexeme.
Adds StringUtil.isAsciiWhitespace, splitOnAsciiWhitespace, containsAsciiUpperCase, and containsAsciiDigit and removes the copies from the AD streams, the English TokenSampleStream, DownloadUtil, and the POS and lemmatizer context generators. NameFinderME.extractNameType delegates to BioCodec. Cases the new tests found first: the TokenSampleStream split returned one empty token for a whitespace-only line where the original split returned none, and matchHyphenatedToken accepted a single hyphen. BrownCluster.splitTabs now removes all trailing empty fields, as String.split does. Each helper has a test, parameterized where the inputs are a table, with the reject side and the edge cases: empty input, leading and trailing separators, non-ASCII spaces and digits, and supplementary-plane characters. Helpers only called from instance methods are no longer static; the block comments on the helpers are now Javadoc that states the behavior.
This was referenced Sep 7, 2026
SimpleEventStreamBuilder and RealValueFileEventStream split the context
part of each event line with String.split("\\s+"), which compiles the
pattern on every call. Both now use StringUtil.splitOnAsciiWhitespace,
which yields the same elements: a leading run gives one empty first
element, trailing empty elements are dropped, and only the six ASCII
whitespace characters separate fields, so a no-break space stays inside
a field. New tests pin those cases through the builder and the stream.
RealBasicEventStream split the context part of each event line with
String.split("\\s+"), compiling the pattern per line. It now uses
StringUtil.splitOnAsciiWhitespace, which yields the same elements
including the leading empty element after a double space and the
dropped trailing empties. A new test pins tab and multi-space runs and
a no-break space that must stay inside a field.
BasicContextGenerator.getContext still calls String.split with the
caller supplied separator. Its public constructor accepts any String,
so a caller may pass a pattern today; changing that needs a decision on
the API contract and is left as is.
LeipzigLanguageSampleStream accepted a sentences file when the first three characters of its name matched "[a-z]+", compiling the pattern for every directory entry. A small scan now checks that those characters are ASCII lower case letters, which rejects the same names: capitals, digits, punctuation, and letters outside ASCII. The length three is a named constant shared with the two other places that cut the language code from the file name. Tests cover the scan directly on both sides and read a temporary directory holding accepted and rejected names through the stream.
MascNamedEntityParser, MascPennTagParser, and MascWordParser removed the "ne-n", "penn-n", and "seg-r" prefixes from identifier attributes with String.replaceFirst, which compiles a pattern for every element. The new package-private MascIdentifiers holds the three prefixes as constants and a removeFirst helper that cuts the first occurrence with indexOf and substring and leaves later occurrences in place, as replaceFirst did. Tests check the helper against replaceFirst on both sides, drive each parser over inline annotation XML, and pin that a repeated prefix still fails to parse.
MorfologikDictionaryBuilder.build turned the metadata file name into
the dictionary file name with replaceAll("\\.info$", ".dict"), building
the pattern from the Morfologik extension constant on each call. The
new toDictionaryFileName checks endsWith(".info") and exchanges that
suffix with substring, leaving any other name unchanged. Both suffixes
are named constants. The test compares the helper with the former
replaceAll over names with a repeated, missing, or differently cased
suffix and checks that the built dictionary is written to the metadata
directory under the expected name. The module pom adds the parent
managed junit-jupiter-params test dependency for the parameterized
test.
FrequencyDictionaryLoader split each dictionary line with the pattern "[\t ]+". The new splitColumns walks the line and breaks it on runs of TAB and space only, with the result of Pattern.split: a leading run gives one empty first column, trailing empty columns are dropped, a separator-only line gives an empty array, and an empty line gives one empty column. A no-break space, a vertical tab, or a form feed stays inside a column, as before. The test compares the scan with the former pattern over those cases and reads unigram and bigram lines with mixed TAB and space runs through parseUnigrams and parseBigrams. The module pom adds the parent managed junit-jupiter-params test dependency for the parameterized test. SpellCheckingCharSequenceNormalizer keeps its URL_LIKE pattern: it combines three alternatives with URL schemes, an email shape, and a list of top level domains, which is a heuristic rather than a plain character scan.
The Files.lines call had no close, so the sentences file remained open until garbage collection. On Windows that blocked the new temp-directory test from deleting the files and failed the build with "Failed to close extension context". The count now runs inside try-with-resources.
…file MarkableFileInputStream wraps a FileInputStream without overriding close, so a reader built on MarkableFileInputStreamFactory leaves the file open after it is closed. On Windows this kept the Leipzig temp-directory test from deleting its files. The two close tests fail before the fix; the mark and reset tests document the existing behavior.
Overrides close to close the wrapped FileInputStream. Before this, a PlainTextByLineStream over a MarkableFileInputStreamFactory released the file handle only at garbage collection, which the CLI tools, the format readers, and the eval tests all rely on; on Windows an open handle also blocks deletion of the file.
krickert
force-pushed
the
OPENNLP-1933-string-regex-calls
branch
from
September 7, 2026 13:18
9a8dd2a to
60c0eed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #1275, and independent of the other parts of the epic. It can be reviewed and merged in any order relative to them. This diff carries the #1275 commits until that one merges; the commits of this change alone: ai-pipestream/opennlp@OPENNLP-1928-regex-removal-trivial...OPENNLP-1933-string-regex-calls
String calls that compile a Pattern on each invocation, spread over several modules:
SimpleEventStreamBuilder,RealValueFileEventStream,RealBasicEventStreamsplit("\\s+")StringUtil.splitOnAsciiWhitespaceLeipzigLanguageSampleStreammatches("[a-z]+")on the file name prefixMascNamedEntityParser,MascPennTagParser,MascWordParserreplaceFirstcalls removing id prefixesMascIdentifiers.removeFirst, first occurrence onlyMorfologikDictionaryBuilderreplaceAllof the metadata suffixendsWithandsubstringFrequencyDictionaryLoader[\t ]+column splitString.splitsemanticsEach new helper is tested with the old call as an oracle, and each parser has a test through its public surface. Single-character splits elsewhere (comma, tab, space, semicolon) take the JDK fast path and compile no pattern, so they are not touched.
SpellCheckingCharSequenceNormalizer.URL_LIKEstays; it is a multi-scheme URL and email heuristic and goes on the exempt list in OPENNLP-1935.BasicContextGeneratoris fixed in OPENNLP-1929.Two file-handle leaks surfaced when the new Leipzig test ran on Windows, and both are fixed here.
MarkableFileInputStreamwrapped aFileInputStreamwithout overridingclose, so every reader built onMarkableFileInputStreamFactoryreleased the handle only at garbage collection, and on Windows an open handle blocks deletion of the file.LeipzigSentencesStreamcounted lines with an unclosedFiles.linescall. Each has a test that fails without its fix. Say the word if these belong in a separate issue rather than here.Verification, with checkstyle, offline,
-Dopennlp.forkCount=1: ml-commons 36, ml-maxent 51, formats 463, morfologik 21, spellcheck 156.OPENNLP-1933