Tester for the Build your own Tokenizer challenge.
make testmake test creates a Python virtualenv, installs regex and tiktoken, and runs the Go tests. The reference CLI is internal/test_helpers/scenarios/base/tokenizer.py. Regenerate frozen outputs with:
.venv/bin/python scripts/generate_fixtures.py