[EXPERIMENT][WIP] Fix Eagle3 VLM draft model export/load (qwen3_vl_eagle3) - #1901
Draft
mlukasze wants to merge 1 commit into
Draft
[EXPERIMENT][WIP] Fix Eagle3 VLM draft model export/load (qwen3_vl_eagle3)#1901mlukasze wants to merge 1 commit into
mlukasze wants to merge 1 commit into
Conversation
Two distinct bugs affected VLM-flavored Eagle3 speculative-decoding draft models (e.g. AngelSlim/Qwen3-VL-4B-Instruct_eagle3), reported in openvinotoolkit/omega#78: 1. Export (`optimum-cli export openvino`, any weight format) failed with `KeyError: 'input_ids'` in `OpenVINOConfigWithPast.generate_dummy_inputs`. VLM Eagle3 configs replace `input_ids` with `inputs_embeds` in dummy inputs, but the attention-mask padding branch only looked up `input_ids` and only triggered for `task == "text-generation"`, missing `image-text-to-text` (the task under which `qwen3_vl_eagle3` registers). Restored the `.get()` fallback and task condition that existed prior to the base.py export-config refactor. 2. Loading an exported VLM Eagle3 model via `OVModelForVisualCausalLM` failed with a cryptic `KeyError: 'llama'` in `_from_pretrained`, because these checkpoints self-report `model_type="llama"` (not a key in `MODEL_TYPE_TO_CLS_MAPPING`) while carrying VLM-oriented `modal_type`/`target_model_type` fields. These models are standalone draft causal LMs and must be loaded with `OVModelForCausalLM` instead. Added a pre-check that raises a clear, actionable `ValueError` pointing at the correct class. Added `test_exporters_cli_eagle3_vlm_quantization` covering fp16/int8/ int4 export + load for `qwen3_vl_eagle3`, using the existing tiny CI fixture, and asserting the improved error message from bug #2.
rkazants
reviewed
Jul 30, 2026
Comment on lines
+898
to
+910
| if config.model_type not in MODEL_TYPE_TO_CLS_MAPPING: | ||
| archs = getattr(config, "architectures", None) or [] | ||
| if archs and "eagle3" in archs[0].lower(): | ||
| raise ValueError( | ||
| f"Model with architecture '{archs[0]}' (model_type='{config.model_type}') is a standalone " | ||
| "Eagle3 speculative-decoding draft model, not a multi-component VLM, even though its " | ||
| "config declares a VLM-oriented `modal_type`/`target_model_type`. Please load it with " | ||
| "`OVModelForCausalLM` instead of `OVModelForVisualCausalLM`." | ||
| ) | ||
| raise ValueError( | ||
| f"Unsupported model_type '{config.model_type}' for `OVModelForVisualCausalLM`. Supported " | ||
| f"model types are: {sorted(MODEL_TYPE_TO_CLS_MAPPING)}." | ||
| ) |
rkazants
reviewed
Jul 30, 2026
| del expected_int8["decoder_with_past"] | ||
| check_compression_state_per_model(self, model.ov_models, expected_int8) | ||
|
|
||
| @parameterized.expand(["fp16", "int8", "int4"]) |
rkazants
reviewed
Jul 30, 2026
| and self.use_cache_branch is not False | ||
| and "attention_mask" in dummy_inputs | ||
| and self.task == "text-generation" | ||
| and self.task in ("text-generation", "image-text-to-text") |
Collaborator
There was a problem hiding this comment.
risky change because we have other vlm models. It can affect them
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes two regressions that block exporting/loading the Eagle3 speculative-decoding draft
model
AngelSlim/Qwen3-VL-4B-Instruct_eagle3for
Qwen/Qwen3-VL-4B-Instruct, as reported inopenvinotoolkit/omega#78.
Bug 1 —
KeyError: 'input_ids'at export timeOpenVINOConfigWithPast.generate_dummy_inputs(inoptimum/exporters/openvino/base.py)assumed the dummy input dict always contains
input_ids, and only ran its attention-maskpadding logic for
task == "text-generation". The Eagle3 draft model exports as a VLMcomponent with
task == "image-text-to-text"and producesinputs_embedsinstead ofinput_ids. Fix: fall back toinputs_embedswheninput_idsis absent, and extend thetask check to include
"image-text-to-text".Bug 2 —
KeyError: 'llama'at load timeOVModelForVisualCausalLM._from_pretrained(inoptimum/intel/openvino/modeling_visual_language.py)does a direct
MODEL_TYPE_TO_CLS_MAPPING[config.model_type]lookup. The Eagle3 draft model'sconfig.jsonsetsmodel_typeto the target model's type ("llama"/"qwen3_vl"family)rather than a VLM-registered type, causing an opaque
KeyError. Fix: add a pre-check thatraises a clear
ValueErrorexplaining that Eagle3 draft models should be loaded withOVModelForCausalLM, notOVModelForVisualCausalLM(matching the exact error reported inthe ticket, now actionable instead of an opaque
KeyError).Testing
Added
test_exporters_cli_eagle3_vlm_quantizationtotests/openvino/test_exporters_cli.py,covering fp16/int8/int4 export + load + the new error message, using the existing tiny CI
fixture
optimum-intel-internal-testing/tiny-random-qwen3-vl-eagle3. All 3 variants pass.Verified no regression on the existing non-VLM eagle3 export/load test.
Full end-to-end validation performed downstream (VLM + Eagle3 speculative decoding via
openvino_genai.VLMPipeline, on CPU/iGPU/dGPU, fp16/int8/int4, WWB accuracy + TTFT/TPOTperf) — see openvinotoolkit/omega#78 for the full report.
Scope
Minimal, surgical fix — no unrelated changes. Both hunks are guarded by narrow conditions
(
inputs_embedsfallback only wheninput_idsmissing; VLMmodel_typepre-check onlytriggers before the existing dict lookup).
Fixes reported issue in openvinotoolkit/omega#78
Before submitting