Skip to content

OPENNLP-1885: Add SentencePiece subword add-on - #178

Draft
krickert wants to merge 8 commits into
apache:OPENNLP-1924-canary-addonfrom
ai-pipestream:OPENNLP-1885-subword-addon
Draft

OPENNLP-1885: Add SentencePiece subword add-on#178
krickert wants to merge 8 commits into
apache:OPENNLP-1924-canary-addonfrom
ai-pipestream:OPENNLP-1885-subword-addon

Conversation

@krickert

@krickert krickert commented Sep 3, 2026

Copy link
Copy Markdown

Summary

  • implements the OpenNLP SubwordTokenizer API with pure Java SentencePiece inference
  • loads SentencePiece .model files and supports unigram and byte-pair models
  • preserves exact UTF-16 spans into the caller's original text after model normalization
  • exposes vocabulary lookup, piece scores, algorithm type, and aligned normalization
  • adds generated test models, independent parity fixtures, usage documentation, and 66 tests

No runtime model, native library, or third-party training data is bundled.

This is the example add-on contribution described by #177.

Dependencies

The PR targets OPENNLP-1924-canary-addon. It remains a draft until the
core API dependency is available to the add-ons build.

Validation

  • mvn clean verify -Dopennlp.forkCount=1: all seven reactor projects passed
  • SentencePiece: 66 tests passed, with one opt-in real-model evaluation skipped
  • mvn -N clean site: 9 HTML pages generated
  • local links and fragments passed
  • RAT reported zero unapproved files
  • all six serializable implementation classes match their computed serialver values

Publish add-on artifacts under org.apache.opennlp.addons while keeping core dependencies under org.apache.opennlp. Replace the discussion ledger with a concise proposal and DEV thread references.

Red evidence: the canary effective groupId was org.apache.opennlp for the root and each module.
# Conflicts:
#	src/site/markdown/addon-guidelines.md
@krickert

krickert commented Sep 3, 2026

Copy link
Copy Markdown
Author

So this is our "canary" PR that would go into our "addons" - of course we'll need to setup the infra for the whole addons, but this would be the impl for the OPENNLP-1885. This will unlock a lot of the embedding add-ons that are being proposed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant