OPENNLP-1935: Add a checkstyle guard for java.util.regex in production code - #1282
Draft
krickert wants to merge 39 commits into
Draft
OPENNLP-1935: Add a checkstyle guard for java.util.regex in production code#1282krickert wants to merge 39 commits into
krickert wants to merge 39 commits into
Conversation
…in DefaultLemmatizerContextGenerator
…in DefaultPOSContextGenerator Add pinning tests for the accept and reject sides of both predicates.
… checks in FeatureGeneratorUtil Add pinning tests for the capPeriod accept and reject sides.
…enPatternFeatureGenerator Add a pinning test that non-letter sub-tokens do not produce st= features.
… char scan Matches (.+)-\w+ semantics: group(1) is everything before the last hyphen, the hyphen must not be at index 0, and the suffix must be non-empty word chars. Add pinning tests for outcomes without hyphen, hyphen at index 0, empty suffix, non-word suffix, and the normal accept case.
…n BrownCluster Replicates String.split(\t) semantics, including dropped trailing empty fields.
…explicit char scans in TokenSampleStream splitOnWhitespace replicates String.split(\\s+): a leading whitespace run yields one empty leading field, runs collapse, and trailing empty fields are dropped.
…mojiCharSequenceNormalizer The replaced pattern contains a high surrogate range, so the regex engine matches whole code points in the flattened range [U+D83C, U+10FC00]. The replacement scans code points, collapses each maximal matching run into a single space, and copies non-matching code points verbatim. Add pinning tests for unpaired surrogates, BMP chars above U+D83C, and supplementary code points beyond U+10FC00.
…xplicit scans in ConlluStream
splitOnHyphen replicates String.split("-"): every hyphen is a boundary,
empty fields between consecutive hyphens are kept, and trailing empty
fields are dropped.
extractTextLang replicates find() of text_([a-z]{2,3}): the first
occurrence of "text_" followed by two to three ASCII lowercase letters,
preferring three.
…ss scans in ParserTool
The two replaceAll passes are replicated by two cursor passes with the
same leftmost-first resume-after-match semantics, which matters for
overlapping pairs such as "x((" or "((a)(b))": a pair starting at the
second char of a match is only reconsidered by the second pass.
…cans in DownloadUtil parseChecksum now scans to the first ASCII whitespace character, replicating split(\s)[0] on the trimmed content. extractLinks replicates find() of the <a href="(.*?)">(.*?)</a> pattern with CASE_INSENSITIVE and DOTALL flags: the href value ends at the first "> and the first case-insensitive </a> closes the match, so nested link markup is swallowed by the outer match.
…meric patterns with explicit scans in ADNameSampleStream
splitOnWhitespace and splitOnUnderscores replicate run-based splitting: a
leading separator run yields one empty leading field, trailing empty
fields are dropped, and an all-separator input yields no fields.
matchHyphenatedToken replicates the three-branch hyphen pattern at code
point granularity, isAlphaNumeric replicates ^[\p{L}\p{Nd}]+$ via
Character.isLetter and Character.isDigit, and tagContent replicates
matches() of <(NER:)?(.*?)> including its optional NER: prefix.
…OSSampleStream
replaceWhitespaceWithEquals replicates replaceAll("=") of the \s+
pattern: every run of ASCII whitespace, including leading and trailing
runs, is replaced by a single equals sign.
…tenceSampleStream parseTextAndParagraph replicates matches() of the ^(?:[a-zA-Z\-]*(\d+)).*?p=(\d+).* pattern: after the optional ASCII letters and hyphens, the text id is the first ASCII digit run and the paragraph id is the digit run after the first "p=" that is followed by at least one digit.
…entenceStream replaceGuillemetPunctuation replicates replaceAll of the »\s+ punct patterns: every run of ASCII whitespace between » and the punctuation character is removed. parsePunctuationLine replicates matches() of the ^(=*)(\W+)$ pattern: the line consists of leading equals signs followed by one or more non-word characters, where a word character is an ASCII letter, digit, or underscore. A line of only equals signs matches, with the last equals sign as lexeme.
Adds StringUtil.isAsciiWhitespace, splitOnAsciiWhitespace, containsAsciiUpperCase, and containsAsciiDigit and removes the copies from the AD streams, the English TokenSampleStream, DownloadUtil, and the POS and lemmatizer context generators. NameFinderME.extractNameType delegates to BioCodec. Cases the new tests found first: the TokenSampleStream split returned one empty token for a whitespace-only line where the original split returned none, and matchHyphenatedToken accepted a single hyphen. BrownCluster.splitTabs now removes all trailing empty fields, as String.split does. Each helper has a test, parameterized where the inputs are a table, with the reject side and the edge cases: empty input, leading and trailing separators, non-ASCII spaces and digits, and supplementary-plane characters. Helpers only called from instance methods are no longer static; the block comments on the helpers are now Javadoc that states the behavior.
krickert
force-pushed
the
OPENNLP-1935-regex-guard
branch
2 times, most recently
from
September 7, 2026 12:04
4a3eeb2 to
4e62535
Compare
The tag patterns in the read loop, applied to a full line, accepted an opening tag as the tag name right after the angle bracket, any characters other than a closing angle bracket, and the closing angle bracket as the last character; a closing tag was the exact text of that tag. isOpeningTag and isClosingTag check the same for a given tag name, and the tag names are constants.
The node, leaf, and bizarre leaf patterns in getElement read a line as a level prefix of equals signs and hyphens, a tag with a colon or an equals sign, and then, for leaves, a quoted lemma, secondary tags in angle brackets, a morphological tag, the closing parenthesis, and the lexeme after whitespace. The patterns backtracked in three places: the prefix could give up hyphens to the tag, the lemma took the last quote after which the rest of the line still parsed, and the secondary tags took the longest run of angle-bracket tags that still left a rest. The scan methods keep each of those preferences: scanLevelAndTag hands out the prefix candidates longest first, parseLeafAfterTag tries the lemma ends from the right, and scanSecondaryTags searches tag ends from the right with a note of the positions after which no rest exists. Line terminators end the lemma, the tags, and the lexeme as the dot did. The lexeme check in the fallback branch is isWordWithMarkup. A side by side run of the old patterns and the new scan over the sample corpus, hand written edge lines, mutations of both, and random lines showed no difference; the tests cover the sample lines, the fall through cases, quotes inside lemma and lexeme, the tag groups, and the bizarre form.
ADNameSampleStream compiled a metadata pattern per call, chosen by the text collection: for literary texts the ASCII letters and hyphens before the text id, for CIE the value of the first source attribute, and otherwise the digits of the text id, each only when a paragraph number follows a p= later in the line. ADSentenceSampleStream had a second copy of the last scan. The package-private ADMetadata now contains the one scan for text and paragraph ids with accessors for the parsed values, the text digits, and the letter prefix, plus the source lookup, and both streams call it. As the dot in the patterns did not cross a line terminator, metadata with one is invalid in all cases. The unused Type enum and the commented-out patterns are removed.
The deep-learning module reads two HuggingFace-shaped JSON files, the vocabulary and the model configuration, with regular expressions that look for a string literal, a colon with optional ASCII whitespace, and then either a digit run or another string literal. JsonScan collects the offset helpers those two readers need: the closing quote of a literal honoring backslash escapes, the closing quote of a literal that must stay on one line, a colon surrounded by whitespace, a whitespace run, and a digit run. The class is public so the doccat subpackage can reach it and is marked Internal. The tests cover accept and reject sides of every helper, including non-ASCII spaces and digits, the five line terminators, and supplementary-plane characters.
AbstractDL.loadJsonVocab used a find() loop over a pattern matching a string literal with backslash escapes, optional ASCII whitespace around a colon, and a run of ASCII digits, anywhere in the text. The loop now walks the text with JsonScan: from each quote it finds the closing quote, the colon, and the digit run; on success it records the entry and resumes after the digits, otherwise it resumes at the character after the quote exactly like the matcher did, so a quote inside a skipped literal can open the next candidate. The method is now package-private so tests feed it text directly. The added parameterized tests pin the odd inputs: a value that is not an integer is skipped, a fractional value keeps its integer prefix, escaped and unicode-escaped keys, keys spanning a line, a backslash before a line terminator, an empty key, whitespace and newlines around the colon, a non-ASCII space after the colon, and a later entry overwriting an earlier one.
DocumentCategorizerConfig.fromJson used a DOTALL pattern to cut the text between the brace after "id2label" and the first closing brace, and a second pattern to pull "key": "value" pairs out of that text with the key running to the next quote and the value, lazily, to the next quote on the same line. Both are now cursor scans over JsonScan: the key literal is located with indexOf and retried at the next occurrence when no colon and brace follow it, the content ends at the first closing brace even when that brace belongs to a nested value, and the entry loop resumes after a matched value or at the character after a quote that opened no entry. The parameterized tests pin the nested brace cut, a brace inside a value, a missing or non-object id2label, whitespace and newlines around colons, an escaped quote inside a value, a value spanning a line, an empty key, a numeric value, an overlapping key literal, and supplementary-plane keys and values.
…ssPathModelFinder The fallback that reads java.class.path split the value with two compiled Patterns, ";" on Windows and ":" elsewhere. A package-private splitClassPath now walks the string once and cuts at the separator character, keeping the String.split result for that separator: a leading separator gives one empty first element, empty elements between consecutive separators stay, trailing empty elements are dropped, separator-only input gives an empty array, and empty input gives a single empty element. The test table covers those cases for both separators and checks each row against String.split. The module gains the managed junit-jupiter-params test dependency for the table.
AbstractClassPathModelFinder turned a wildcard such as "*opennlp-models-*" into a regular expression by escaping "." and rewriting "*" to ".*" and "?" to ".", then compiled it and called matches() on the file part of each URL. The package-private GlobMatcher now walks the wildcard and the input by code point: "*" absorbs any run of characters including none, "?" takes exactly one, and every other character stands for itself. As with the old ".", neither wildcard crosses a line feed, carriage return, next line, line separator, or paragraph separator, and the whole input must be covered. The protected asRegex and matchesPattern(URL, Pattern) are replaced by matchesWildcard(URL, String); DirectoryModelFinder and SimpleClassPathModelFinder keep the plain wildcard strings instead of compiled patterns, and the pattern cache in DirectoryModelFinder goes away because there is nothing left to compile. Characters the old code never escaped, such as parentheses, brackets, braces, "+", "|", "^", "$", and backslash, had regular expression meaning before or made Pattern.compile fail; they are now literal. GlobMatcherTest covers the accept and reject sides, dots, those characters, each line terminator, and supplementary-plane characters.
DirectoryModelFinder had no test. The new test copies the opennlp-models jars from the test class path into a temporary directory one level below the scanned root, runs the shared finder assertions against it, and checks the non-recursive mode, a narrowing jar prefix, an unknown prefix, and the null directory rejection.
SimpleEventStreamBuilder and RealValueFileEventStream split the context
part of each event line with String.split("\\s+"), which compiles the
pattern on every call. Both now use StringUtil.splitOnAsciiWhitespace,
which yields the same elements: a leading run gives one empty first
element, trailing empty elements are dropped, and only the six ASCII
whitespace characters separate fields, so a no-break space stays inside
a field. New tests pin those cases through the builder and the stream.
RealBasicEventStream split the context part of each event line with
String.split("\\s+"), compiling the pattern per line. It now uses
StringUtil.splitOnAsciiWhitespace, which yields the same elements
including the leading empty element after a double space and the
dropped trailing empties. A new test pins tab and multi-space runs and
a no-break space that must stay inside a field.
BasicContextGenerator.getContext still calls String.split with the
caller supplied separator. Its public constructor accepts any String,
so a caller may pass a pattern today; changing that needs a decision on
the API contract and is left as is.
LeipzigLanguageSampleStream accepted a sentences file when the first three characters of its name matched "[a-z]+", compiling the pattern for every directory entry. A small scan now checks that those characters are ASCII lower case letters, which rejects the same names: capitals, digits, punctuation, and letters outside ASCII. The length three is a named constant shared with the two other places that cut the language code from the file name. Tests cover the scan directly on both sides and read a temporary directory holding accepted and rejected names through the stream.
MascNamedEntityParser, MascPennTagParser, and MascWordParser removed the "ne-n", "penn-n", and "seg-r" prefixes from identifier attributes with String.replaceFirst, which compiles a pattern for every element. The new package-private MascIdentifiers holds the three prefixes as constants and a removeFirst helper that cuts the first occurrence with indexOf and substring and leaves later occurrences in place, as replaceFirst did. Tests check the helper against replaceFirst on both sides, drive each parser over inline annotation XML, and pin that a repeated prefix still fails to parse.
MorfologikDictionaryBuilder.build turned the metadata file name into
the dictionary file name with replaceAll("\\.info$", ".dict"), building
the pattern from the Morfologik extension constant on each call. The
new toDictionaryFileName checks endsWith(".info") and exchanges that
suffix with substring, leaving any other name unchanged. Both suffixes
are named constants. The test compares the helper with the former
replaceAll over names with a repeated, missing, or differently cased
suffix and checks that the built dictionary is written to the metadata
directory under the expected name. The module pom adds the parent
managed junit-jupiter-params test dependency for the parameterized
test.
FrequencyDictionaryLoader split each dictionary line with the pattern "[\t ]+". The new splitColumns walks the line and breaks it on runs of TAB and space only, with the result of Pattern.split: a leading run gives one empty first column, trailing empty columns are dropped, a separator-only line gives an empty array, and an empty line gives one empty column. A no-break space, a vertical tab, or a form feed stays inside a column, as before. The test compares the scan with the former pattern over those cases and reads unigram and bigram lines with mixed TAB and space runs through parseUnigrams and parseBigrams. The module pom adds the parent managed junit-jupiter-params test dependency for the parameterized test. SpellCheckingCharSequenceNormalizer keeps its URL_LIKE pattern: it combines three alternatives with URL schemes, an email shape, and a list of top level domains, which is a heuristic rather than a plain character scan.
The Files.lines call had no close, so the sentences file remained open until garbage collection. On Windows that blocked the new temp-directory test from deleting the files and failed the build with "Failed to close extension context". The count now runs inside try-with-resources.
…file MarkableFileInputStream wraps a FileInputStream without overriding close, so a reader built on MarkableFileInputStreamFactory leaves the file open after it is closed. On Windows this kept the Leipzig temp-directory test from deleting its files. The two close tests fail before the fix; the mark and reset tests document the existing behavior.
Overrides close to close the wrapped FileInputStream. Before this, a PlainTextByLineStream over a MarkableFileInputStreamFactory released the file handle only at garbage collection, which the CLI tools, the format readers, and the eval tests all rely on; on Windows an open handle also blocks deletion of the file.
… as written The separator constructor documents a separator character, but the current split treats the argument as a regular expression: a pipe splits on each character, a dot matches no boundary at all, and a plus sign or an opening parenthesis throws at prediction time. The test records the documented behavior and fails before the fix.
getContext scans for the separator with indexOf instead of compiling it as a regular expression, keeping the String.split shape: a leading occurrence gives an empty first element and trailing empty elements are removed. The constructor rejects a null or empty separator with an IllegalArgumentException, and the Javadoc states that the separator is taken as written.
…ter set The alphanumeric pattern is a tokenizer model parameter: TokenizerFactory stores it in the manifest as a regex string and reads it back with Pattern.compile, so the Pattern type stays in the API. Matching no longer goes through the regex engine for the shipped defaults: AlphaNumericCheck reads a pattern of the shape ^[...]+$ with literal characters and simple ranges into a character set and tests tokens by lookup. Any other pattern, and any pattern compiled with flags, is still matched by the engine, so the result equals pattern.matcher(token).matches() in all cases. TokenizerME and TokSpanEventStream hold the check instead of the Pattern. The test compares the check with the regex for all built-in language defaults and a set of custom patterns over the same tokens, and pins which shapes run as a set lookup.
Adds the checkstyle checks that keep the policy in place after the epic is merged: NoRegexImport rejects imports of java.util.regex, and NoRegexStringCall flags String.replaceAll, replaceFirst, matches with a string argument, and split with a backslash-escaped argument, since each of those compiles a Pattern per call. Test sources are excluded. The documented exceptions are in checkstyle-suppressions.xml with a reason for each: the regex name finder, the AnCora head rules, the Penn Treebank and name sample markup grammars, and the tokenizer alphanumeric pattern that is part of the model manifest. This commit goes at the top of the stack: the checks pass only after parts 2 to 6 have removed the remaining sites.
krickert
force-pushed
the
OPENNLP-1935-regex-guard
branch
from
September 7, 2026 13:18
4e62535 to
25bc925
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #1275 and merges last. It carries every other part of the epic because the checks it adds fail while any unconverted site is still in the tree, so its build only goes green once #1276, #1277, #1278, #1279, #1280, and #1281 are in. The commit of this change alone: ai-pipestream@25bc925
Keeps the policy of OPENNLP-1926 in place once the other parts are merged. This PR has to merge last: its build fails until the other parts are in.
NoRegexImport:IllegalImportofjava.util.regex.NoRegexStringCall:RegexpSinglelineJavaforString.replaceAll,replaceFirst,matcheswith a string argument, andsplitwith a backslash-escaped argument, since each compiles a Pattern per call.Test sources are excluded. The documented exceptions are in a new
checkstyle-suppressions.xml, wired throughsuppressionsLocationin the parent pom, each with its reason:RegexNameFinderand its factory,AncoraSpanishHeadRules,Parse,NameSample,SpellCheckingCharSequenceNormalizer, and the tokenizer alphanumeric classes from OPENNLP-1934.Verification: run on each other part alone, the checks report exactly the sites that part fixes; with all of them applied the full reactor validates and tests clean (23 modules, BUILD SUCCESS, offline,
-Dopennlp.forkCount=1).OPENNLP-1935