OPENNLP-1934: Evaluate the tokenizer alphanumeric pattern as a character set - #1281

Draft

krickert wants to merge 18 commits into

apache:mainfrom

ai-pipestream:OPENNLP-1934-tokenizer-alphanumeric

Commits on Sep 7, 2026

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultLemmatizerContextGenerator
krickert
committed
0144431
OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultPOSContextGenerator
krickert
committed
71a79d9
OPENNLP-1928: Replace capPeriod pattern with explicit length and char checks in FeatureGeneratorUtil
krickert
committed
b4c44c0
OPENNLP-1928: Replace noLetters pattern with a char range scan in TokenPatternFeatureGenerator
krickert
committed
c0a731b
OPENNLP-1928: Replace typed outcome pattern with lastIndexOf and word char scan
krickert
committed
ad7142a
OPENNLP-1928: Replace tab pattern split with explicit tab splitting in BrownCluster
krickert
committed
44d6b34
OPENNLP-1928: Replace whitespace split and alphanumeric pattern with explicit char scans in TokenSampleStream
krickert
committed
1e466bd
OPENNLP-1928: Replace emoji pattern with a code point range scan in EmojiCharSequenceNormalizer
krickert
committed
e2c1d70
OPENNLP-1928: Replace token id split and text language pattern with explicit scans in ConlluStream
krickert
committed
32b9bee
OPENNLP-1928: Replace untokenized paren patterns with explicit two-pass scans in ParserTool
krickert
committed
99891dc
OPENNLP-1928: Replace checksum split and link pattern with explicit scans in DownloadUtil
krickert
committed
426e49a
OPENNLP-1928: Replace tag, whitespace, underline, hyphen, and alphanumeric patterns with explicit scans in ADNameSampleStream
krickert
committed
09e531c
OPENNLP-1928: Replace whitespace pattern with an explicit scan in ADPOSSampleStream
krickert
committed
80e448f
OPENNLP-1928: Replace metadata pattern with an explicit scan in ADSentenceSampleStream
krickert
committed
26950fd
OPENNLP-1928: Replace punctuation patterns with explicit scans in ADSentenceStream
krickert
committed
ca15512
OPENNLP-1928: Document the scan helpers that replace the patterns
krickert
committed
2b8db9e
OPENNLP-1928: Share the ASCII scan helpers and test each helper
krickert
committed
85dceea
OPENNLP-1934: Evaluate the tokenizer alphanumeric pattern as a character set
krickert
committed
1fc8ebe

Navigation Menu

Uh oh!

OPENNLP-1934: Evaluate the tokenizer alphanumeric pattern as a character set - #1281

OPENNLP-1934: Evaluate the tokenizer alphanumeric pattern as a character set#1281
krickert wants to merge 18 commits into
apache:mainfrom
ai-pipestream:OPENNLP-1934-tokenizer-alphanumeric

Commits on Sep 7, 2026

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultLemmatizerContextGenerator

OPENNLP-1928: Replace cap and digit patterns with ASCII range checks in DefaultPOSContextGenerator

OPENNLP-1928: Replace capPeriod pattern with explicit length and char checks in FeatureGeneratorUtil

OPENNLP-1928: Replace noLetters pattern with a char range scan in TokenPatternFeatureGenerator

OPENNLP-1928: Replace typed outcome pattern with lastIndexOf and word char scan

OPENNLP-1928: Replace tab pattern split with explicit tab splitting in BrownCluster

OPENNLP-1928: Replace whitespace split and alphanumeric pattern with explicit char scans in TokenSampleStream

OPENNLP-1928: Replace emoji pattern with a code point range scan in EmojiCharSequenceNormalizer

OPENNLP-1928: Replace token id split and text language pattern with explicit scans in ConlluStream

OPENNLP-1928: Replace untokenized paren patterns with explicit two-pass scans in ParserTool

OPENNLP-1928: Replace checksum split and link pattern with explicit scans in DownloadUtil

OPENNLP-1928: Replace tag, whitespace, underline, hyphen, and alphanumeric patterns with explicit scans in ADNameSampleStream

OPENNLP-1928: Replace whitespace pattern with an explicit scan in ADPOSSampleStream

OPENNLP-1928: Replace metadata pattern with an explicit scan in ADSentenceSampleStream

OPENNLP-1928: Replace punctuation patterns with explicit scans in ADSentenceStream

OPENNLP-1928: Document the scan helpers that replace the patterns

OPENNLP-1928: Share the ASCII scan helpers and test each helper

OPENNLP-1934: Evaluate the tokenizer alphanumeric pattern as a character set