Are 4B failures only an issue on quantization (Tool Calling + DFlash)?

[TomLucidor]

It is noted previously that 4B has parsing issues, but usually as FP8 or even FP4. I wonder if for different models, there are ways to check if the quantizations need to be layer-specific. If there are a oneliners to test these combos, that would be good https://huggingface.co/nota-ai/Qwen3.5-4B-QAD-W4A16 https://huggingface.co/nota-ai/Qwen3.5-4B-DFlash-GPTQ-W4A16

[raullenchai]

I tested this on the current Rapid-MLX release using the same tool schema and prompt. Both qwen3.5-4b-4bit and qwen3.5-4b-8bit returned valid structured tool calls in 5/5 runs. Streaming also functioned correctly, returning proper delta.tool_calls without leaking raw XML.

At this time, we cannot reproduce a tool-call parsing failure caused specifically by 4-bit quantization. While quantization may still impact accuracy on more complex tasks, evaluating that would require a broader task-level benchmark rather than a basic parser test.

Additionally, the linked GPTQ/W4A16 models are not MLX models, so Rapid-MLX cannot load them directly. DFlash is also not currently enabled for the 4B variants; our validated DFlash alias is qwen3.5-27b-8bit.

[TomLucidor]

To be clear the naive 4-bit quants are the ones causing issues so GPTQ/W4A16 is more about if those can be "smart quants" to prevent issues, but leakage is a persistent issue. Also DFlash methods out there for GGUF and others have leakage issues still.