Are there bugs with E4B/E2B or with Gemma 4 at the moment?

[TomLucidor]

Tool calling seemed to be an issue with Gemma 4 models (including the smaller ones) for multiple agents and maybe they come from parsers/templates NousResearch/hermes-agent#6626 anomalyco/opencode#20995

Update: Hermes is partially busted and need some patches NousResearch/hermes-agent#6626 (comment)

[TomLucidor]

Self-research: There seems to be fixes like:

• @lucianommartins from google DeepMind https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/47 + https://huggingface.co/google/gemma-4-E4B-it/discussions/36
• @mann1x or ManniX-ITA https://huggingface.co/google/gemma-4-31B-it/discussions/119 + https://huggingface.co/google/gemma-4-E4B-it/discussions/37
• @RyanMullins https://huggingface.co/google/gemma-4-E4B-it/discussions/13 + https://huggingface.co/google/gemma-4-31B-it/discussions/55 + https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/20

Side discovery: even dLLMs have similar quality issues needing to be patched https://huggingface.co/google/diffusiongemma-26B-A4B-it/discussions/23

[TomLucidor]

Self-sidenote: PLE quants are okay https://github.com/FakeRocket543/mlx-gemma4

[lucianommartins]

Hi @TomLucidor - yes we identified some improvements to be done on the chat_templates. I'm proposing few PRs to the official ones... could you please test with them?

https://huggingface.co/google/gemma-4-E2B-it/discussions/35
https://huggingface.co/google/gemma-4-E4B-it/discussions/36
https://huggingface.co/google/gemma-4-12B-it/discussions/35
https://huggingface.co/google/gemma-4-26B-A4B-it/discussions/47
https://huggingface.co/google/gemma-4-31B-it/discussions/118

[TomLucidor]

Will do next week, and also would like @raullenchai to also vouch for this initiative to get Gemma 4 working at all costs. Are there ideal testing protocols that we can try with MLX?

[raullenchai]

Thanks for the detailed links — I reproduced this against the current Rapid-MLX code and a real gemma-4-e2b-4bit server.

There are two separate issues here:

1. Tool-call parsing / OpenAI API shape: I could not reproduce the reported empty-tool_calls failure in current Rapid-MLX. Required and auto tool choice both returned structured message.tool_calls; streaming emitted delta.tool_calls and ended with finish_reason="tool_calls"; a tool-result follow-up also completed correctly. Five different auto-tool prompts all produced valid calls. The current gemma4 parser, output router, streaming postprocessor, and grammar tests also pass. The linked OpenCode report itself identifies its failure on the OpenCode/AI SDK side after Ollama had already returned valid tool-call deltas.

2. Checkpoint chat templates: this is a real issue and we have started fixing it. Most Gemma 4 aliases in Rapid-MLX point to mlx-community checkpoints whose templates lag behind Google's current official E2B/E4B/12B/26B/31B templates. I reproduced stale-template behavior including JSON null rendering as Python None, permissive string function.arguments producing malformed wire output, and missing updated multi-turn continuation/thinking handling after tool responses. Rapid-MLX already normalizes OpenAI string arguments before template application, which masks one of those defects, but the checkpoint templates should still be brought onto a consistent fixed contract.

We are now comparing the current Google templates across all Gemma 4 sizes and implementing one offline Rapid-MLX fix with multi-turn golden tests, rather than relying on each converted checkpoint to refresh independently.

[TomLucidor]

BTW who can update the MLX parser under the community repo? Maybe things can be corrected at the source (ideally)

[raullenchai]

Answer for current releases: the stale-template part of this was fixed in v0.12.18 (#2179).

• Rapid-MLX now bundles fixed Gemma 4 chat templates (gemma4_full for 12B/26B/31B, gemma4_compact for E2B/E4B). The Gemma 4 aliases select them, so a converted checkpoint with an older template no longer changes tool-call or multi-turn rendering.
• The fix covers the issues reproduced above: JSON null rendering, string function.arguments, and continuation after tool responses. Multi-turn golden tests cover it.
• Tool-call parsing (gemma4 parser) returned structured tool_calls in both streaming and non-streaming modes in our reproduction, and grammar-constrained native tool calls are available (feat(tools): grammar-constrain Gemma-4 native tool-calls (#558 E4) #1171).

On the checkpoint repos: the templates inside mlx-community conversions are maintained by whoever uploaded each one, so fixes there have to go through those Hugging Face repos. Rapid-MLX applies its bundled template either way. If you still see a Gemma 4 tool-calling failure on a current release, please open an issue with the alias, the request and the response.