Skip to content

Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625

Merged
ericstj merged 16 commits into
mainfrom
copilot/add-sentencepiece-tokenizer-api
Jul 16, 2026
Merged

Add public SentencePieceTokenizer factory methods for Unigram from vocab list and tokenizer.json#7625
ericstj merged 16 commits into
mainfrom
copilot/add-sentencepiece-tokenizer-api

Address Tarek review: support null unk_id and reject unknown pre-toke…

512eb13
Select commit
Loading
Failed to load commit list.
Azure Pipelines / MachineLearning-CodeCoverage failed Jul 16, 2026 in 1h 18m 43s

Build #20260715.1 had test failures

1 errors / 5 warnings

Details

Tests

  • Failed: 35 (1.66%)
  • Passed: 1,954 (92.74%)
  • Other: 118 (5.60%)
  • Total: 2,107

Annotations

Check failure on line 156 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Build log #L156 

Cmd.exe exited with code '1'.

Check failure on line 1 in Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test 

System.IO.InvalidDataException : End of Central Directory record could not be found.
Raw output 
   at System.IO.Compression.ZipArchive.ReadEndOfCentralDirectory()
   at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.Open(String archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles)
   at Microsoft.ML.AutoML.Test.DatasetUtil.UnZip(String gzArchiveName, String destFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 231
   at Microsoft.ML.AutoML.Test.DatasetUtil.DownloadImageSet(String imagesDownloadFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 191
   at Microsoft.ML.AutoML.Test.DatasetUtil.GetFlowersDataset() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 136
   at Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\AutoFeaturizerTests.cs:line 107
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Check failure on line 1 in Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_TrainTest

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_TrainTest 

System.IO.InvalidDataException : End of Central Directory record could not be found.
Raw output 
   at System.IO.Compression.ZipArchive.ReadEndOfCentralDirectory()
   at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.Open(String archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles)
   at Microsoft.ML.AutoML.Test.DatasetUtil.UnZip(String gzArchiveName, String destFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 231
   at Microsoft.ML.AutoML.Test.DatasetUtil.DownloadImageSet(String imagesDownloadFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 191
   at Microsoft.ML.AutoML.Test.DatasetUtil.GetFlowersDataset() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 136
   at Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_TrainTest() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\AutoFitTests.cs:line 333
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Check failure on line 1 in Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image 

System.IO.InvalidDataException : End of Central Directory record could not be found.
Raw output 
   at System.IO.Compression.ZipArchive.ReadEndOfCentralDirectory()
   at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.Open(String archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles)
   at Microsoft.ML.AutoML.Test.DatasetUtil.UnZip(String gzArchiveName, String destFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 231
   at Microsoft.ML.AutoML.Test.DatasetUtil.DownloadImageSet(String imagesDownloadFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 191
   at Microsoft.ML.AutoML.Test.DatasetUtil.GetFlowersDataset() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 136
   at Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\AutoFitTests.cs:line 390
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Check failure on line 1 in Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_CV

See this annotation in the file changed.

@azure-pipelines azure-pipelines / MachineLearning-CodeCoverage

Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_CV 

System.IO.InvalidDataException : End of Central Directory record could not be found.
Raw output 
   at System.IO.Compression.ZipArchive.ReadEndOfCentralDirectory()
   at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.Open(String archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles)
   at Microsoft.ML.AutoML.Test.DatasetUtil.UnZip(String gzArchiveName, String destFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 231
   at Microsoft.ML.AutoML.Test.DatasetUtil.DownloadImageSet(String imagesDownloadFolder) in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 191
   at Microsoft.ML.AutoML.Test.DatasetUtil.GetFlowersDataset() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\DatasetUtil.cs:line 136
   at Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_CV() in D:\a\_work\1\s\test\Microsoft.ML.AutoML.Tests\AutoFitTests.cs:line 360
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
View more details on Azure Pipelines