OPENNLP-1933: Replace per-call String regex splits and replacements with scans - #1279

Draft

krickert wants to merge 26 commits into

apache:mainfrom

ai-pipestream:OPENNLP-1933-string-regex-calls

Commits on Sep 7, 2026

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultLemmatizerContextGenerator
krickert
committed
0144431
OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultPOSContextGenerator
krickert
committed
71a79d9
OPENNLP-1928: Replace capPeriod pattern with explicit length and char checks in FeatureGeneratorUtil
krickert
committed
b4c44c0
OPENNLP-1928: Replace noLetters pattern with a char range scan in TokenPatternFeatureGenerator
krickert
committed
c0a731b
OPENNLP-1928: Replace typed outcome pattern with lastIndexOf and word char scan
krickert
committed
ad7142a
OPENNLP-1928: Replace tab pattern split with explicit tab splitting in BrownCluster
krickert
committed
44d6b34
OPENNLP-1928: Replace whitespace split and alphanumeric pattern with explicit char scans in TokenSampleStream
krickert
committed
1e466bd
OPENNLP-1928: Replace emoji pattern with a code point range scan in EmojiCharSequenceNormalizer
krickert
committed
e2c1d70
OPENNLP-1928: Replace token id split and text language pattern with explicit scans in ConlluStream
krickert
committed
32b9bee
OPENNLP-1928: Replace untokenized paren patterns with explicit two-pass scans in ParserTool
krickert
committed
99891dc
OPENNLP-1928: Replace checksum split and link pattern with explicit scans in DownloadUtil
krickert
committed
426e49a
OPENNLP-1928: Replace tag, whitespace, underline, hyphen, and alphanumeric patterns with explicit scans in ADNameSampleStream
krickert
committed
09e531c
OPENNLP-1928: Replace whitespace pattern with an explicit scan in ADPOSSampleStream
krickert
committed
80e448f
OPENNLP-1928: Replace metadata pattern with an explicit scan in ADSentenceSampleStream
krickert
committed
26950fd
OPENNLP-1928: Replace punctuation patterns with explicit scans in ADSentenceStream
krickert
committed
ca15512
OPENNLP-1928: Document the scan helpers that replace the patterns
krickert
committed
2b8db9e
OPENNLP-1928: Share the ASCII scan helpers and test each helper
krickert
committed
85dceea
OPENNLP-1933: Scan event context fields in the ml-commons streams
krickert
committed
8d0a412
OPENNLP-1933: Scan event context fields in RealBasicEventStream
krickert
committed
166d180
OPENNLP-1933: Scan Leipzig file names for the language code
krickert
committed
ddebc2c
OPENNLP-1933: Strip MASC identifier prefixes with a text search
krickert
committed
7be411d
OPENNLP-1933: Derive the dictionary file name with an endsWith check
krickert
committed
cb7e193
OPENNLP-1933: Scan dictionary columns on TAB and space runs
krickert
committed
b9e9c4b
OPENNLP-1933: Close the line-counting stream in LeipzigSentencesStream
krickert
committed
d7284ae
OPENNLP-1933: Test that closing a MarkableFileInputStream closes the file
krickert
committed
a7ff43f
OPENNLP-1933: Close the underlying file in MarkableFileInputStream
krickert
committed
60c0eed

Navigation Menu

Uh oh!

OPENNLP-1933: Replace per-call String regex splits and replacements with scans - #1279

OPENNLP-1933: Replace per-call String regex splits and replacements with scans#1279
krickert wants to merge 26 commits into
apache:mainfrom
ai-pipestream:OPENNLP-1933-string-regex-calls

Commits on Sep 7, 2026

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultLemmatizerContextGenerator

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultPOSContextGenerator

OPENNLP-1928: Replace capPeriod pattern with explicit length and char checks in FeatureGeneratorUtil

OPENNLP-1928: Replace noLetters pattern with a char range scan in TokenPatternFeatureGenerator

OPENNLP-1928: Replace typed outcome pattern with lastIndexOf and word char scan

OPENNLP-1928: Replace tab pattern split with explicit tab splitting in BrownCluster

OPENNLP-1928: Replace whitespace split and alphanumeric pattern with explicit char scans in TokenSampleStream

OPENNLP-1928: Replace emoji pattern with a code point range scan in EmojiCharSequenceNormalizer

OPENNLP-1928: Replace token id split and text language pattern with explicit scans in ConlluStream

OPENNLP-1928: Replace untokenized paren patterns with explicit two-pass scans in ParserTool

OPENNLP-1928: Replace checksum split and link pattern with explicit scans in DownloadUtil

OPENNLP-1928: Replace tag, whitespace, underline, hyphen, and alphanumeric patterns with explicit scans in ADNameSampleStream

OPENNLP-1928: Replace whitespace pattern with an explicit scan in ADPOSSampleStream

OPENNLP-1928: Replace metadata pattern with an explicit scan in ADSentenceSampleStream

OPENNLP-1928: Replace punctuation patterns with explicit scans in ADSentenceStream

OPENNLP-1928: Document the scan helpers that replace the patterns

OPENNLP-1928: Share the ASCII scan helpers and test each helper

OPENNLP-1933: Scan event context fields in the ml-commons streams

OPENNLP-1933: Scan event context fields in RealBasicEventStream

OPENNLP-1933: Scan Leipzig file names for the language code

OPENNLP-1933: Strip MASC identifier prefixes with a text search

OPENNLP-1933: Derive the dictionary file name with an endsWith check

OPENNLP-1933: Scan dictionary columns on TAB and space runs

OPENNLP-1933: Close the line-counting stream in LeipzigSentencesStream

OPENNLP-1933: Test that closing a MarkableFileInputStream closes the file

OPENNLP-1933: Close the underlying file in MarkableFileInputStream