Skip to content

Document tokenizer goroutine safety and make Decode concurrency-safe - #26

Open
disintegrator wants to merge 3 commits into
tiktoken-go:mainfrom
disintegrator:docs/count-encode-goroutine-safety
Open

disintegrator wants to merge 3 commits into
tiktoken-go:mainfrom
disintegrator:docs/count-encode-goroutine-safety

Conversation

@disintegrator

@disintegrator disintegrator commented Sep 8, 2026 •

Copy link
Copy Markdown

Summary

  • Document that Count, Encode, and Decode may be called concurrently on the same instance, on both tokenizer.Codec and the concrete codec.Codec methods.
  • Make Decode safe from the first call by protecting lazy reverse-vocabulary initialization with sync.Once. Concurrent callers wait for the map to be fully populated; no warm-up is required.
  • Extend shared-instance concurrency tests across all five supported encodings. Twenty-four workers exercise Count, Encode, and Decode on an initially unused codec, checking counts, token IDs, token strings, and decoded text. Inputs include empty text, multilingual text, whitespace, numbers, and byte-pair merges.
  • Enable the race detector in CI.

Commits

  1. 8c47213 — Document and test the existing Count and Encode concurrency guarantee; enable race detection in CI.
  2. 0e2bfda — Make Decode concurrency-safe and extend the documentation and tests. The original commit is unchanged.

Problem and fix

Concurrent first calls to Decode previously assigned and populated reverseVocabulary without synchronization. An eight-goroutine reproduction produced a race report and a concurrent map read/write fatal error. sync.Once now ensures initialization completes before any caller reads the map. Codec must not be copied after first use, as documented.

Verification

  • The original eight-goroutine, first-use Decode reproduction now passes with go run -race, without pre-initialization.
  • go test -race -run '^TestConcurrentCountEncodeDecode$' -count=10 -parallel=24 ./...
  • go test -race ./...
  • go build ./...

All passed locally.

@disintegrator disintegrator changed the title Document and test Count and Encode goroutine safety Document tokenizer goroutine safety and make Decode concurrency-safe Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant