OPENNLP-1928: Replace trivial regular expressions with explicit scans - #1275
Draft
krickert wants to merge 17 commits into
Draft
OPENNLP-1928: Replace trivial regular expressions with explicit scans#1275krickert wants to merge 17 commits into
krickert wants to merge 17 commits into
Conversation
…in DefaultLemmatizerContextGenerator
…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.
… checks in FeatureGeneratorUtil Add pinning tests for the capPeriod accept and reject sides.
…enPatternFeatureGenerator Add a pinning test that non-letter sub-tokens do not produce st= features.
… char scan Matches (.+)-\w+ semantics: group(1) is everything before the last hyphen, the hyphen must not be at index 0, and the suffix must be non-empty word chars. Add pinning tests for outcomes without hyphen, hyphen at index 0, empty suffix, non-word suffix, and the normal accept case.
…n BrownCluster Replicates String.split(\t) semantics, including dropped trailing empty fields.
…explicit char scans in TokenSampleStream splitOnWhitespace replicates String.split(\\s+): a leading whitespace run yields one empty leading field, runs collapse, and trailing empty fields are dropped.
…mojiCharSequenceNormalizer The replaced pattern contains a high surrogate range, so the regex engine matches whole code points in the flattened range [U+D83C, U+10FC00]. The replacement scans code points, collapses each maximal matching run into a single space, and copies non-matching code points verbatim. Add pinning tests for unpaired surrogates, BMP chars above U+D83C, and supplementary code points beyond U+10FC00.
…xplicit scans in ConlluStream
splitOnHyphen replicates String.split("-"): every hyphen is a boundary,
empty fields between consecutive hyphens are kept, and trailing empty
fields are dropped.
extractTextLang replicates find() of text_([a-z]{2,3}): the first
occurrence of "text_" followed by two to three ASCII lowercase letters,
preferring three.
…ss scans in ParserTool
The two replaceAll passes are replicated by two cursor passes with the
same leftmost-first resume-after-match semantics, which matters for
overlapping pairs such as "x((" or "((a)(b))": a pair starting at the
second char of a match is only reconsidered by the second pass.
…cans in DownloadUtil parseChecksum now scans to the first ASCII whitespace character, replicating split(\s)[0] on the trimmed content. extractLinks replicates find() of the <a href="(.*?)">(.*?)</a> pattern with CASE_INSENSITIVE and DOTALL flags: the href value ends at the first "> and the first case-insensitive </a> closes the match, so nested link markup is swallowed by the outer match.
…meric patterns with explicit scans in ADNameSampleStream
splitOnWhitespace and splitOnUnderscores replicate run-based splitting: a
leading separator run yields one empty leading field, trailing empty
fields are dropped, and an all-separator input yields no fields.
matchHyphenatedToken replicates the three-branch hyphen pattern at code
point granularity, isAlphaNumeric replicates ^[\p{L}\p{Nd}]+$ via
Character.isLetter and Character.isDigit, and tagContent replicates
matches() of <(NER:)?(.*?)> including its optional NER: prefix.
…OSSampleStream
replaceWhitespaceWithEquals replicates replaceAll("=") of the \s+
pattern: every run of ASCII whitespace, including leading and trailing
runs, is replaced by a single equals sign.
…tenceSampleStream parseTextAndParagraph replicates matches() of the ^(?:[a-zA-Z\-]*(\d+)).*?p=(\d+).* pattern: after the optional ASCII letters and hyphens, the text id is the first ASCII digit run and the paragraph id is the digit run after the first "p=" that is followed by at least one digit.
…entenceStream replaceGuillemetPunctuation replicates replaceAll of the »\s+ punct patterns: every run of ASCII whitespace between » and the punctuation character is removed. parsePunctuationLine replicates matches() of the ^(=*)(\W+)$ pattern: the line consists of leading equals signs followed by one or more non-word characters, where a word character is an ASCII letter, digit, or underscore. A line of only equals signs matches, with the last equals sign as lexeme.
Adds StringUtil.isAsciiWhitespace, splitOnAsciiWhitespace, containsAsciiUpperCase, and containsAsciiDigit and removes the copies from the AD streams, the English TokenSampleStream, DownloadUtil, and the POS and lemmatizer context generators. NameFinderME.extractNameType delegates to BioCodec. Cases the new tests found first: the TokenSampleStream split returned one empty token for a whitespace-only line where the original split returned none, and matchHyphenatedToken accepted a single hyphen. BrownCluster.splitTabs now removes all trailing empty fields, as String.split does. Each helper has a test, parameterized where the inputs are a table, with the reject side and the edge cases: empty input, leading and trailing separators, non-ASCII spaces and digits, and supplementary-plane characters. Helpers only called from instance methods are no longer static; the block comments on the helpers are now Javadoc that states the behavior.
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE:
So 1275's biggest sell is that it improves on StringUtil. Once that is completed all of the below bug + improvements are independent of each other and can be merge at our convenience. Once this one is merged, I will make sure the rest of the tickets are based off main and clean (shouldn't need any rebase cha-cha).
Regex habitually breaks a lot of our features (especially whitespace), is a memory-hog, can be slow, and every language has their own set of bugs too on top of having a syntax that can be difficult to understand. This is an attempt to clean our regex throughout the codebase. Previous attempts were successful and saw a sizable performance boost.
This first ticket focuses on the trivial use cases - all simple quick fixes made to also build up StringUtil.
AI assisted text:
Base of the OPENNLP-1926 work: #1276, #1277, #1278, #1279, #1280, and #1281 are each based on this branch, are independent of each other, and reuse the
StringUtilhelpers added here. #1282 adds the checkstyle guard and has to merge last. Their diffs include these commits until this PR is merged.Replaces trivial regular-expression uses with explicit character scans, keeping the exact behavior of each replaced pattern, including the
String.splitedge cases for leading and trailing separators. Sixteen sites across the POS, lemmatizer, name finder, tokenizer, feature generator, CoNLL-U, AD, parser CLI, download, and emoji normalizer code; each helper has a test, parameterized where the inputs form a table, with reject sides and edge cases (empty input, leading and trailing separators, non-ASCII spaces and digits, supplementary-plane characters). The non-trivial AD patterns and the other modules follow in the stacked PRs. No new regex use and no JDK whitespace predicate is introduced.StringUtilgains four predicates the scans share rather than per-class copies:isAsciiWhitespace,splitOnAsciiWhitespace(theString.split("\\s+")result),containsAsciiUpperCase, andcontainsAsciiDigit.DefaultPOSContextGenerator,DefaultLemmatizerContextGenerator[A-Z],[0-9]capital and digit detectionFeatureGeneratorUtil^[A-ZÄÖÜ]\.$capital-period checkTokenPatternFeatureGenerator[^a-zA-Z]non-letter checkBioCodec,NameFinderME(.+)-\w+typed-outcome splitBrownClusterTokenSampleStream\s+split and[A-Za-z0-9]checkEmojiCharSequenceNormalizerConlluStream-andtext_([a-z]{2,3})language commentParserToolDownloadUtil<a href>link patternADNameSampleStreamADPOSSampleStream,ADSentenceSampleStream,ADSentenceStreamVerification: opennlp-runtime 2,658, opennlp-formats 423, opennlp-cli 53, and opennlp-tools 505 tests with checkstyle, offline,
-Dopennlp.forkCount=1.OPENNLP-1928