Collapse invalid UTF-8 runs into a single U+FFFD on decode

[mertyildiran]

Closes #27

Problem

Decode can return a string that is not valid UTF-8, and feeding that string back to Encode does not round-trip — the text comes back changed.

Byte-level BPE tokens are allowed to split a multi-byte character, so the concatenation of decoded token bytes can end mid-sequence. In o200k_base, token 35895 is "\xe8\x96" — the first two bytes of a 3-byte character.

When that string was passed to Encode, regexp2 converted it to UTF-16 internally and substituted U+FFFD for each invalid byte, so a 2-byte truncated sequence produced two replacement characters instead of one.

Reference behavior

Python's tiktoken decodes with decode_bytes(...).decode("utf-8", errors="replace"), which collapses each run of invalid bytes into a single U+FFFD:

>>> ids = [53373, 41426, 35950, 53348, 35895, 41791]
>>> text = enc.decode(ids)
>>> print(text)
immt Arena spelen()*� ред
>>> print(enc.encode(text))
[53373, 41426, 35950, 53348, 3251, 41791]

Fix

Codec.Decode now returns strings.ToValidUTF8(out, "\uFFFD") — one replacement character per invalid run, matching errors="replace".

	before	after	tiktoken
Decode(ids)	"...()*\xe8\x96 ред"	"...()*� ред"	"...()*� ред"
Encode of that	[..., 10123, ...] ("��")	[..., 3251, ...] ("�")	[..., 3251, ...]
Decode again	"...()*�� ред" — changed	"...()*� ред" — stable	"...()*� ред"

The ids themselves are not preserved (35895 → 3251). That is inherent to re-encoding sanitized text and matches tiktoken exactly; only the text round-trips.

Behavior change to note

Decode will no longer return invalid UTF-8. Anyone relying on the raw bytes for byte-level work would now get replacement characters. I believe this is the right default since it matches tiktoken, but it is observable — happy to discuss alternatives (e.g. an opt-in DecodeBytes alongside the current behavior) if you prefer.

Test

Added TestO200kBaseDecodeInvalidUTF8, which decodes token 35895, encodes the result, and decodes again. It fails on the current code with 10123 and the doubled replacement characters.