Repository navigation

Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI, exact on 18 models - #19

Merged

bong-water-water-bong merged 2 commits into

mainfrom

Sep 23, 2026

Merged

Tokenizers: Hugging Face tokenizers v0.23.2 behind our C ABI, exact on 18 models#19
bong-water-water-bong merged 2 commits into
mainfrom
hf-tokenizers

Commits on Sep 23, 2026
- Tokenizers: Hugging Face tokenizers v0.23.2 behind our own C ABI, byte-exact on 18 models
  
  authored and
  bong-water-water-bong
  committed
  374b3ac
- copyright.py: Rust sources use // comments; Cargo.lock is generated (exempt)
  
  authored and
  bong-water-water-bong
  committed
  1ef449d