OPENNLP-1877: Static embedding engine for modern distilled embedding tables (opennlp-embeddings) - #1152

Draft

krickert wants to merge 168 commits into

apache:OPENNLP-1885-sentencepiecefrom

ai-pipestream:static-embeddings

Commits on Aug 9, 2026

Add opennlp-embeddings module with a safetensors reader
krickert
committed
2d95887
Add StaticEmbeddingModel: tokenize, gather, weight, mean-pool, normalize
krickert
committed
dc3fa77
Add word similarity and analogy convenience API
krickert
committed
cffed1f
Add a JMH benchmark for StaticEmbeddingModel, matching the perf-1850-followup jmh profile
krickert
committed
b13f849
Review pass on opennlp-embeddings: analogy exclusion fixes, faster neighbor scan, thread-safety hardening
krickert
committed
c9381b8
Stream safetensors tensor data with positional reads
krickert
committed
a919ceb
Unit-test SafetensorsHeaderParser directly, reject trailing header garbage
krickert
committed
795734b
Add StaticEmbeddingModel.load(modelDirectory) resolving switches from the model configs
krickert
committed
dcd15e9
Document opennlp-embeddings: Dev Manual chapter and module README
krickert
committed
e25ff87
OPENNLP-1877: Address review: value-safe TensorInfo, checked IO, typed load switches, distribution wiring
krickert
committed
c7610de
OPENNLP-1877: Register opennlp-embeddings in the root dependencyManagement
krickert
committed
78414d9
OPENNLP-1877: Rename WordPieceVocabulary to WordpieceVocabulary
krickert
committed
4cb03e8
OPENNLP-1877: Add the TextEmbedder seam; both engines implement it
krickert
committed
e76555e
OPENNLP-1877: Give the embeddings module its own tokenization pipeline
krickert
committed
d43f6e1
OPENNLP-1877: Load the tiny-vectors test model from the classpath stream
krickert
committed
4aea42b
OPENNLP-1877: Trim commentary and tighten javadoc per review conventions
krickert
committed
b7ca1b6
OPENNLP-1877: Rewrite the opennlp-embeddings README with diagrams and align the Dev Manual chapter
krickert
committed
3f9bead
OPENNLP-1877: Document implementation-defined empty-input behavior on TextEmbedder and both impls
krickert
committed
99538d3
OPENNLP-1877: Read F16 and BF16 safetensors tensors, not just F32
krickert
committed
110613c
OPENNLP-1877: Support SentencePiece models through the SubwordTokenizer seam
krickert
committed
78a3fd9
OPENNLP-1877: Document helpers and validate record and parser boundaries
krickert
committed
d423eb2
OPENNLP-1877: Let the embeddings JMH benchmark load real model directories
krickert
committed
9f7ea98
OPENNLP-1877: Add a distillation tutorial and real two-model benchmark numbers
krickert
committed
48e81e5
OPENNLP-1877: Load WordPiece tables that dropped the frame tokens
krickert
committed
6c2a938
OPENNLP-1877: Add the AssembleModel CLI to complete and verify distilled directories
krickert
committed
2ae4912
OPENNLP-1877: Ship the distillation script and the Python parity harness
krickert
committed
8b2c075
OPENNLP-1877: Trim residual commentary per review conventions
krickert
committed
dfe5834
OPENNLP-1877: Name format constants, document throws, and source or replace unsupported doc claims
krickert
committed
0349b1f
OPENNLP-1877: Cite the static embedding usage example test in the manual
krickert
committed
1fd2f68
OPENNLP-1877: Align dev helper placement and test literals with the review conventions
krickert
committed
9aa534c
OPENNLP-1877: Add a Java model distiller and address the review comments
krickert
committed
d3cf3ff
OPENNLP-1877: Fix distiller correctness bugs and cover the untested surface
krickert
committed
2501af1
OPENNLP-1877: Pin and verify teacher downloads
krickert
committed
1bc1928
OPENNLP-1877: Batch same-length inputs into one ONNX run in embedAll
krickert
committed
7694191
OPENNLP-1877: Mark public API experimental and reject bare minus headers
krickert
committed
742148c
OPENNLP-1877: Address review: complete experimental markers, hrefs, and edge tests
krickert
committed
595959d
OPENNLP-1877: Throw InvalidFormatException for malformed model content
krickert
committed
71b16b3
OPENNLP-1877: Demote test-only SafetensorsFile accessors to package-private
krickert
committed
7212e40
OPENNLP-1877: Document the DistillModel and AssembleModel tools in the manual
krickert
committed
ddb1992

Commits on Aug 10, 2026

Commits on Aug 13, 2026

Commits on Aug 16, 2026

Commits on Aug 17, 2026

Commits on Aug 18, 2026

Commits on Aug 21, 2026

OPENNLP-1877: load self-contained Model2Vec Unigram tokenizers
krickert
committed
f6f1bef

Commits on Aug 24, 2026

Bump io.github.classgraph:classgraph from 4.8.192 to 4.8.193 (#1241)
dependabot[bot]
authored
8881065

Commits on Aug 25, 2026

Commits on Aug 26, 2026

OPENNLP-1903: Replace per-candidate Sequence copies with chain nodes in BeamSearch (#1205)
krickert
authored
b46b957

Commits on Aug 30, 2026

Commits on Aug 31, 2026

Bump io.github.classgraph:classgraph from 4.8.193 to 4.8.194 (#1255)
dependabot[bot]
authored
7235395

Commits on Sep 1, 2026

Commits on Sep 2, 2026

Commits on Sep 4, 2026

Commits on Sep 5, 2026

Commits on Sep 6, 2026