Skip to content

Collapse invalid UTF-8 runs into a single U+FFFD on decode - #28

Open
mertyildiran wants to merge 1 commit into
tiktoken-go:mainfrom
mertyildiran:main
Open

Collapse invalid UTF-8 runs into a single U+FFFD on decode#28
mertyildiran wants to merge 1 commit into
tiktoken-go:mainfrom
mertyildiran:main

Commits

  1. Commits on Oct 9, 2026