Skip to content

Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625

Open
ericstj with Copilot wants to merge 16 commits into
mainfrom
copilot/add-sentencepiece-tokenizer-api
Open

Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625
ericstj with Copilot wants to merge 16 commits into
mainfrom
copilot/add-sentencepiece-tokenizer-api

Address Tarek review: support null unk_id and reject unknown pre-toke…

512eb13
Select commit
Loading
Failed to load commit list.
Azure Pipelines / MachineLearning-CI (Ubuntu_x64 Release_Build) failed Jul 16, 2026 in 21m 51s

Ubuntu_x64 Release_Build failed

1 errors / 11 warnings

Annotations

Check failure on line 31 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CI (Ubuntu_x64 Release_Build)

Build log #L31 

Bash exited with code '1'.
View more details on Azure Pipelines