[OpenVINO] Support MiniCPM-V-4.6 with task image-text-to-text

[popovaan]

Description

Fixed the MiniCPM-V-4.6 (minicpmv4_6) end-to-end HF-vs-OpenVINO greedy divergence that the prior session mis-attributed to an unfixable qwen3_5 OV-conversion decode defect. Root cause was a Python-side patcher bug (reproducible in pure PyTorch, not OpenVINO): Qwen3_5DynamicCacheWrap.get_mask_sizes returned kv_offset=past_length instead of 0, so transformers>=5.6 create_causal_mask shifted the full-attention key positions past the query and masked every cached token during decode (mask [0,-inf,-inf,...]); prefill matched, first decode token onward diverged. Per-layer hooks isolated it to the single full-attention layer (linear layers exact ~1e-6). Fix: kv_offset=0 (canonical DynamicLayer contract). Also fixed a test bug: minicpmv4_6 was missing from test_seq2seq get_transformer_model_class AutoModelForImageTextToText branch, causing an AutoModelForCausalLM load error before inference. After fixes, patched-eager and OV decode match HF exactly, CLI export exit 0, real image-text generate() produces diverse tokens, and all targeted repo tests pass.

Conversion

optimum-cli export openvino --model openbmb/MiniCPM-V-4.6 output_dir --task image-text-to-text --trust-remote-code

Reproduce generation

from PIL import Image
from transformers import AutoProcessor
from optimum.intel.openvino import OVModelForVisualCausalLM

model_dir = "output_dir"
processor = AutoProcessor.from_pretrained(model_dir, trust_remote_code=True)
model = OVModelForVisualCausalLM.from_pretrained(model_dir, device="CPU")
image = Image.new("RGB", (64, 64), "white")
inputs = processor(images=image, text="Describe this image.", return_tensors="pt")
output_ids = model.generate(**inputs, max_new_tokens=10)
print(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

Validation

• Validation model: Tiny model (/home/openvino_bot/.local/share/openvino-model-agent/requests/issue-36/repository/workspace/tiny-minicpmv4_6)
• WWB similarity: 1.0

Related model-support PRs

• OpenVINO GenAI: [OpenVINO] Support MiniCPM-V-4.6 with task image-text-to-text openvinotoolkit/openvino.genai#4249

Before submitting

• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[popovaan]

@popovaan add a comment with the link to the original pytorch code of this method.

[Mohamed-Ashraf273]

@popovaan add a comment with the link to the original pytorch code of this method.

[popovaan]

👀 @Mohamed-Ashraf273, review request received.

[popovaan]

🛠️ This request needs source changes and has been added to the sequential modification queue.

[popovaan]

Addressed @popovaan's request to add a comment linking to the original PyTorch code. Since no inline anchor was provided for "this method", I added upstream-source reference comments to every method in the PR that reimplements original MiniCPM-V-4.6 PyTorch logic:

• optimum/intel/openvino/modeling_visual_language.py (_OVMiniCPMV4_6ForCausalLM):
  • _patch_position_ids -> links to MiniCPMV4_6VisionEmbeddings.forward / get_vision_nearest_position_ids (vision_utils.py)
  • _window_index -> links to MiniCPMV4_6ViTWindowAttentionMerger.get_window_index / get_vision_window_index
  • get_vision_embeddings -> links to the upstream get_image_features -> MiniCPMV4_6VisionModel.forward -> merger chain
  • merge_vision_text_embeddings -> links to the masked_scatter in MiniCPMV4_6Model.forward
• optimum/exporters/openvino/model_patcher.py:
  • _minicpmv4_6_vision_attention -> links to MiniCPMV4_6VisionAttention.forward
  • MiniCPMV4_6VisionEmbeddingsModelPatcher docstring -> links to the traced upstream vision forward + mergers

Links point to transformers/.../models/minicpmv4_6/modeling_minicpmv4_6.py and transformers/vision_utils.py on main. Changes are comment-only; both files pass python -m py_compile. If a specific single method was intended, point me at it and I can trim to just that one.

Pushed commit bd6bd41f1920.

Tests:

• python -m py_compile optimum/intel/openvino/modeling_visual_language.py optimum/exporters/openvino/model_patcher.py: pass