Skip to content

Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625

Merged
ericstj merged 16 commits into
mainfrom
copilot/add-sentencepiece-tokenizer-api
Jul 16, 2026
Merged

Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625
ericstj merged 16 commits into
mainfrom
copilot/add-sentencepiece-tokenizer-api

Address Tarek review: support null unk_id and reject unknown pre-toke…

512eb13
Select commit
Loading
Failed to load commit list.
Dotnet Policy Service / license/cla succeeded Jul 16, 2026 in 0s

All CLA requirements met.

This check verifies that the author has agreed to a CLA with Microsoft.

View more details on Dotnet Policy Service