Skip to content

About

Tester for the tokenizer challenge.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

tokenizer-tester

Tester for the Build your own Tokenizer challenge.

make test

make test creates a Python virtualenv, installs regex and tiktoken, and runs the Go tests. The reference CLI is internal/test_helpers/scenarios/base/tokenizer.py. Regenerate frozen outputs with:

.venv/bin/python scripts/generate_fixtures.py

About

Tester for the tokenizer challenge.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages