feat(qwen3): bundle Qwen3 tokenizer for offline single-file/GGUF encoders

[Pfannkuchensack]

Summary

Single-file (safetensors) and GGUF Qwen3 encoder checkpoints used by Anima
(0.6B) and Z-Image (4B/8B) ship weights only — no tokenizer files. The loader
pulled the tokenizer from Qwen/Qwen3-4B on HuggingFace, which fails offline /
airgapped and whenever the HF cache is not persisted (e.g. Docker without a
cache volume).

Vendor the self-contained Qwen3 fast tokenizer (Apache-2.0, from Qwen/Qwen3-4B)
in the package and load it locally, mirroring the bundled T5-XXL tokenizer
(#9244). The Qwen3 BPE tokenizer is identical across the 0.6B/4B/8B variants, so
a single copy serves every Qwen3 encoder. Removes the HuggingFace download path
from both the checkpoint and GGUF loaders.

Changes

• Add vendored tokenizer under invokeai/backend/qwen3/tokenizer/ (tokenizer.json, tokenizer_config.json, special_tokens_map.json).
• Add load_bundled_qwen3_tokenizer() helper (invokeai/backend/qwen3/qwen3_tokenizer.py), mirroring anima/t5_tokenizer.py.
• Point Qwen3EncoderCheckpointLoader and Qwen3EncoderGGUFLoader at the bundled tokenizer; drop the dead DEFAULT_TOKENIZER_SOURCE HF-download fallback.
• Register the tokenizer files as package data in pyproject.toml so they ship in the wheel.

Related Issues / Discussions

Follows the same approach as #9244 (bundled T5-XXL tokenizer for Anima). No linked issue.

A follow-up will apply the same fix to the Qwen2.5-VL encoder (Qwen-Image / Qwen-Image-Edit), which has the identical single-file problem for its tokenizer, config, and image processor.

QA Instructions

1. Automated: run the new offline test — fully passes without network:

   HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 pytest tests/backend/qwen3/test_qwen3_tokenizer.py
   

2. Manual (offline single-file encoder):

   • Remove/rename the HF cache for the repo to force the offline path: ~/.cache/huggingface/hub/models--Qwen--Qwen3-4B.
   • Start Invoke with no network (or HF_HUB_OFFLINE=1).
   • Install a single-file Qwen3 encoder (Anima 0.6B, or a Z-Image 4B/8B .safetensors) and run a generation.
   • Expected: prompt encoding succeeds with no network request; previously this raised an OSError / HF download error.

3. Regression: confirm a diffusers-folder Qwen3 encoder (with its own tokenizer/) and the GGUF encoder still load and generate correctly.

Merge Plan

Standard merge — no DB/schema changes. Note: adds ~11 MB of vendored tokenizer JSON to the package (same rationale/precedent as the bundled T5 tokenizer in #9244).

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — N/A (no frontend/redux changes)
• Documentation added / updated (if applicable) — N/A
• Updated What's New copy (if doing a release after this PR) — N/A