[OpenVINO] Support jina-vlm with task image-text-to-text by popovaan · Pull Request #1909 · huggingface/optimum-intel

popovaan · 2026-08-02T09:41:15Z

Description

Added native OpenVINO export + inference support for model_type 'jvlm' (JinaVLMForConditionalGeneration) in optimum-intel. Registered a VLM export config with custom vision-embeddings, language, and text-embeddings behaviors plus OV-traceable patchers/dummy generators; added the _OVJinaVLMForCausalLM runtime class with index-based image-feature merge. optimum-cli export produces 5 IR submodels and OpenVINO greedy generation exactly matches HF token IDs on a real image+text prompt. Added a cached _create_tiny_jvlm_model() fixture and registered jvlm across test_export/test_seq2seq/test_exporters_cli/utils_tests; all targeted tests pass. Docs updated.

Conversion

optimum-cli export openvino --model jinaai/jina-vlm output_dir --task image-text-to-text --trust-remote-code

Reproduce generation

from PIL import Image
from transformers import AutoProcessor
from optimum.intel.openvino import OVModelForVisualCausalLM

model_dir = "output_dir"
processor = AutoProcessor.from_pretrained(model_dir, trust_remote_code=True)
model = OVModelForVisualCausalLM.from_pretrained(model_dir, device="CPU")
image = Image.new("RGB", (64, 64), "white")
inputs = processor(images=image, text="Describe this image.", return_tensors="pt")
output_ids = model.generate(**inputs, max_new_tokens=10)
print(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

Validation

Validation model: Tiny model (/home/openvino_bot/.local/share/openvino-model-agent/requests/issue-40/repository/workspace/tiny_jina_vlm)
WWB similarity: 1.0

Final real-model validation

Estimated FP16 size: 4.62 GiB
Available RAM at validation: 122.60 GiB
70% RAM limit: 85.82 GiB
Model: Real model (jinaai/jina-vlm)
WWB Optimum similarity: 1.0
WWB GenAI similarity: 0.95056

Related model-support PRs

OpenVINO GenAI: [OpenVINO] Support jina-vlm with task image-text-to-text openvinotoolkit/openvino.genai#4252

Before submitting

Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2026-08-02T09:42:48Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Mohamed-Ashraf273 · 2026-08-03T09:02:08Z

@popovaan can you please review this PR

ma0358412 · 2026-08-03T11:04:56Z

@Mohamed-Ashraf273 Please summarize the validation performed in this PR. Do not modify any files.

Mohamed-Ashraf273 · 2026-08-03T11:05:20Z

👀 @ma0358412, I received this review request. A read-only response will be prepared first; source changes, if needed and permitted, will enter the sequential work queue.

Mohamed-Ashraf273 · 2026-08-03T11:05:38Z

❌ Review request 5165520245 failed: RuntimeError: Review agent exited with 1; see /home/mohamed-ashraf/.local/state/ov-agent/logs/review-5165520245.log.

Retry with ov-agent retry-comment 'huggingface/optimum-intel:issue_comment:5165520245'.

ma0358412 · 2026-08-03T11:12:59Z

@Mohamed-Ashraf273 Please summarize the validation performed in this PR. Do not modify any files.

Mohamed-Ashraf273 · 2026-08-03T11:13:32Z

👀 @ma0358412, I received this review request. A read-only response will be prepared first; source changes, if needed and permitted, will enter the sequential work queue.

Mohamed-Ashraf273 · 2026-08-03T11:13:52Z

❌ Review request 5165596560 failed: RuntimeError: Review agent exited with 1; see /home/mohamed-ashraf/.local/state/ov-agent/logs/review-5165596560.log.

Retry with ov-agent retry-comment 'huggingface/optimum-intel:issue_comment:5165596560'.

ma0358412 · 2026-08-03T14:42:39Z

@Mohamed-Ashraf273 Please summarize the validation performed in this PR. Do not modify any files.

Mohamed-Ashraf273 · 2026-08-03T14:43:05Z

👀 @ma0358412, review request received.

Mohamed-Ashraf273 · 2026-08-03T14:43:30Z

❌ Review request 5167833948 failed: RuntimeError: Review agent exited with 1; see /home/mohamed-ashraf/.local/state/ov-agent/logs/review-5167833948.log.

Retry with ov-agent retry-comment 'huggingface/optimum-intel:issue_comment:5167833948'.

Mohamed-Ashraf273 · 2026-08-04T14:46:37Z

@popovaan Please summarize the validation performed in this PR. Do not modify any files.

popovaan · 2026-08-04T14:46:42Z

👀 @Mohamed-Ashraf273, review request received.

Mohamed-Ashraf273 · 2026-08-04T14:47:02Z

👀 @popovaan, review request received.

popovaan · 2026-08-04T14:47:18Z

Validation summary

Based on the PR description and diff, the validation for jina-vlm (model_type='jvlm', task image-text-to-text) consists of:

Functional / export validation

optimum-cli export openvino produces 5 IR submodels for the model.
OpenVINO greedy generation exactly matches HF token IDs on a real image+text prompt.

Accuracy validation (WWB — who-what-benchmark similarity)

Tiny model (tiny_jina_vlm): WWB similarity 1.0.
Real model (jinaai/jina-vlm):
- WWB Optimum similarity: 1.0
- WWB GenAI similarity: 0.95056
Real-model resource context: estimated FP16 size 4.62 GiB; available RAM 122.60 GiB; 70% RAM limit 85.82 GiB (model fits within limit).

Automated tests (added/registered for jvlm)

tests/openvino/test_export.py — added to model mapping, GENERATIVE_MODELS, and remote-code / custom-gen-config handling.
tests/openvino/test_exporters_cli.py — added ("image-text-to-text", "jvlm") case and expected-submodel count.
tests/openvino/test_seq2seq.py — added jvlm to OVModelForVisualCausalLMIntegrationTest supported and remote-code lists.
tests/openvino/utils_tests.py — added cached _create_tiny_jvlm_model() fixture (synthesized locally from config + remote code, no original weights downloaded) plus expected submodel node counts (lm_model 18, text_embeddings_model 1, vision_embeddings_model 38).
The PR states all targeted tests pass.

Notes / caveats (not blocking, just observations)

The GenAI similarity (0.95056) is below the Optimum similarity (1.0); this reflects the companion OpenVINO GenAI support PR ([OpenVINO] Support jina-vlm with task image-text-to-text openvinotoolkit/openvino.genai#4252) and may be worth confirming it meets the accepted threshold.
Test results ("all targeted tests pass") are self-reported in the PR description; they were not re-executed as part of this read-only summary.

No files were modified and no tests were run (read-only triage).

popovaan · 2026-08-04T14:47:21Z

👀 @Mohamed-Ashraf273, review request received.

Mohamed-Ashraf273 · 2026-08-04T14:47:32Z

❌ Review request 5180673544 failed: RuntimeError: Review agent exited with 1; see /home/mohamed-ashraf/.local/state/ov-agent/logs/review-5180673544.log.

Retry with ov-agent retry-comment 'huggingface/optimum-intel:issue_comment:5180673544'.

popovaan · 2026-08-04T14:47:51Z

Validation summary for PR #1909 (jina-vlm, `model_type='jvlm'`, task `image-text-to-text`)

Based solely on the PR description and diff (read-only triage; no files modified, no tests run):

Functional / export

optimum-cli export openvino produces 5 IR submodels.
OpenVINO greedy generation exactly matches HF token IDs on a real image+text prompt.

Accuracy (WWB — who-what-benchmark similarity)

Tiny model (tiny_jina_vlm): 1.0
Real model (jinaai/jina-vlm):
- WWB Optimum similarity: 1.0
- WWB GenAI similarity: 0.95056
Resource context for the real-model run: estimated FP16 size 4.62 GiB, available RAM 122.60 GiB, 70% RAM limit 85.82 GiB (fits within limit).

Automated tests registered for `jvlm` (confirmed in diff)

tests/openvino/test_export.py: added to model mapping, GENERATIVE_MODELS, and remote-code / custom-gen-config handling.
tests/openvino/test_exporters_cli.py: added ("image-text-to-text", "jvlm") case with expected submodel count (2).
tests/openvino/test_seq2seq.py: added jvlm to OVModelForVisualCausalLMIntegrationTest supported and remote-code lists.
tests/openvino/utils_tests.py: added cached _create_tiny_jvlm_model() fixture (synthesized locally from config + remote code, no original weights downloaded) with expected submodel node counts (lm_model 18, text_embeddings_model 1, vision_embeddings_model 38).
PR states all targeted tests pass (self-reported; not re-executed here).

Observations (non-blocking)

GenAI similarity (0.95056) is below the Optimum similarity (1.0); worth confirming it meets the accepted acceptance threshold. This ties to the companion GenAI support PR ([OpenVINO] Support jina-vlm with task image-text-to-text openvinotoolkit/openvino.genai#4252).
"All tests pass" is self-reported in the PR description and was not verified in this read-only summary.

Mohamed-Ashraf273 · 2026-08-04T14:49:44Z

@popovaan Please summarize the validation performed in this PR. Do not modify any files.

popovaan · 2026-08-04T14:50:17Z

👀 @Mohamed-Ashraf273, review request received.

popovaan · 2026-08-04T14:50:51Z

Validation summary — PR #1909 (jina-vlm, `model_type='jvlm'`, task `image-text-to-text`)

Summarized from the PR description and diff only (read-only triage; no files modified, no tests run).

Functional / export validation

optimum-cli export openvino --model jinaai/jina-vlm --task image-text-to-text --trust-remote-code produces 5 IR submodels.
OpenVINO greedy generation exactly matches HF token IDs on a real image+text prompt.

Accuracy validation (WWB — who-what-benchmark similarity)

Tiny model (tiny_jina_vlm): WWB similarity 1.0.
Real model (jinaai/jina-vlm):
- WWB Optimum similarity: 1.0
- WWB GenAI similarity: 0.95056
Real-model resource context: estimated FP16 size 4.62 GiB; available RAM 122.60 GiB; 70% RAM limit 85.82 GiB (model fits within the limit).

Automated tests registered for `jvlm` (confirmed in the diff)

tests/openvino/test_export.py: added jvlm to the model-class mapping and to GENERATIVE_MODELS; extended remote-code and custom-gen-config handling (trust_remote_code=True on load/reload).
tests/openvino/test_exporters_cli.py: added the ("image-text-to-text", "jvlm") case with an expected submodel count of 2.
tests/openvino/test_seq2seq.py: added jvlm to OVModelForVisualCausalLMIntegrationTest supported and remote-code lists.
tests/openvino/utils_tests.py: added a cached _create_tiny_jvlm_model() fixture (synthesized locally from config + remote code, no original weights downloaded) and expected submodel node counts (lm_model 18, text_embeddings_model 1, vision_embeddings_model 38).
The PR states all targeted tests pass (self-reported).

Observations (non-blocking)

GenAI similarity (0.95056) is below the Optimum similarity (1.0). Worth confirming it meets the project's accepted acceptance threshold; this ties to the companion GenAI support PR ([OpenVINO] Support jina-vlm with task image-text-to-text openvinotoolkit/openvino.genai#4252).
"All tests pass" is self-reported in the PR description and was not re-executed as part of this read-only summary.

No files were modified and no commands were run.

Mohamed-Ashraf273 · 2026-08-04T16:01:15Z

@popovaan Please summarize the validation performed in this PR. Do not modify any files.

popovaan · 2026-08-04T17:14:32Z

👀 @Mohamed-Ashraf273, review request received.

popovaan · 2026-08-04T17:17:48Z

Validation summary — PR #1909 (jina-vlm, `model_type='jvlm'`, task `image-text-to-text`)

Summarized from the PR description and diff only (read-only triage; no files modified, no commands run).

Functional / export validation

optimum-cli export openvino --model jinaai/jina-vlm --task image-text-to-text --trust-remote-code produces 5 IR submodels.
OpenVINO greedy generation exactly matches HF token IDs on a real image+text prompt.

Accuracy validation (WWB — who-what-benchmark similarity)

Tiny model (tiny_jina_vlm): WWB similarity 1.0.
Real model (jinaai/jina-vlm):
- WWB Optimum similarity: 1.0
- WWB GenAI similarity: 0.95056
Real-model resource context: estimated FP16 size 4.62 GiB; available RAM 122.60 GiB; 70% RAM limit 85.82 GiB (model fits within the limit).

Automated tests registered for `jvlm` (confirmed in the diff)

tests/openvino/test_export.py: added jvlm to the model-class mapping and to GENERATIVE_MODELS; extended remote-code / custom-gen-config handling (trust_remote_code=True on load and reload).
tests/openvino/test_exporters_cli.py: added the ("image-text-to-text", "jvlm") case with expected submodel count 2.
tests/openvino/test_seq2seq.py: added jvlm to OVModelForVisualCausalLMIntegrationTest supported and remote-code lists.
tests/openvino/utils_tests.py: added cached _create_tiny_jvlm_model() fixture (synthesized locally from config + remote code, no original weights downloaded) with expected submodel node counts — lm_model 18, text_embeddings_model 1, vision_embeddings_model 38; also registered jvlm in the remote-code and OVModelForVisualCausalLM class maps.
The PR states all targeted tests pass (self-reported).

Observations (non-blocking)

GenAI similarity (0.95056) is below the Optimum similarity (1.0); worth confirming it meets the project's accepted acceptance threshold. This ties to the companion GenAI support PR ([OpenVINO] Support jina-vlm with task image-text-to-text openvinotoolkit/openvino.genai#4252).
"All tests pass" is self-reported in the PR description and was not re-executed as part of this read-only summary.

No files were modified and no commands were run.

Mohamed-Ashraf273 · 2026-08-05T16:00:07Z



+def jina_vlm_vision_embed_forward(self, image_patches, image_masks=None):
+    # Runs the JinaVLM vision tower + vision-language connector and returns the flattened image


@popovaan add a comment with the link to the original pytorch code of this method.

popovaan · 2026-08-05T16:00:10Z

👀 @Mohamed-Ashraf273, review request received.

popovaan · 2026-08-05T16:00:52Z

🛠️ This request needs source changes and has been added to the sequential modification queue.

popovaan · 2026-08-05T16:02:48Z

Addressed the review comment at optimum/exporters/openvino/model_patcher.py:3067. Added links to the original PyTorch remote code that jina_vlm_vision_embed_forward mirrors:

_encode_images: https://huggingface.co/jinaai/jina-vlm/blob/ddfa80b180f87f59873fd1cea352dec51183ab88/modeling_jvlm.py#L525
JinaVLMVisionModel.forward: https://huggingface.co/jinaai/jina-vlm/blob/ddfa80b180f87f59873fd1cea352dec51183ab88/modeling_jvlm.py#L223

The links are pinned to the commit ddfa80b1 of jinaai/jina-vlm for stability. This is a comment-only change; python -m py_compile confirms the file still parses.

Pushed commit d6eb700344b0.

Tests:

python3 -m py_compile optimum/exporters/openvino/model_patcher.py: passed

Mohamed-Ashraf273 · 2026-08-05T17:22:36Z

+
+
+def jina_vlm_language_model_forward(
+    self,


@popovaan add a comment with the link to the original pytorch code of this method.

👀 @Mohamed-Ashraf273, review request received.

This request needs source changes and has been added to the sequential modification queue.

Addressed the review comment on jina_vlm_language_model_forward (optimum/exporters/openvino/model_patcher.py:3140). Added a link to the original PyTorch remote code that this method mirrors:

JinaVLMForConditionalGeneration.forward: https://huggingface.co/jinaai/jina-vlm/blob/ddfa80b180f87f59873fd1cea352dec51183ab88/modeling_jvlm.py#L749

The link is pinned to commit ddfa80b1 of jinaai/jina-vlm for stability, matching the existing links already present on jina_vlm_vision_embed_forward (_encode_images L525, JinaVLMVisionModel.forward L223). The line number (L749) was verified by downloading modeling_jvlm.py at that pinned commit, not guessed. This is a comment-only change.

Validation:

python -m py_compile optimum/exporters/openvino/model_patcher.py: passed

Pushed commit b151e705813f.

Tests:

python -m py_compile optimum/exporters/openvino/model_patcher.py: passed

[OpenVINO] Support jina-vlm with task image-text-to-text

fb10e30

popovaan mentioned this pull request Aug 2, 2026

[OpenVINO] Support jina-vlm with task image-text-to-text openvinotoolkit/openvino.genai#4252

Draft

4 tasks

Mohamed-Ashraf273 reviewed Aug 5, 2026

View reviewed changes

Address review feedback from @Mohamed-Ashraf273

d6eb700

Mohamed-Ashraf273 reviewed Aug 5, 2026

View reviewed changes

Address review feedback from @Mohamed-Ashraf273

b151e70



		def jina_vlm_vision_embed_forward(self, image_patches, image_masks=None):
		# Runs the JinaVLM vision tower + vision-language connector and returns the flattened image

Uh oh!

Conversation

popovaan commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Conversion

Reproduce generation

Validation

Final real-model validation

Related model-support PRs

Before submitting

Uh oh!

HuggingFaceDocBuilderDev commented Aug 2, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

ma0358412 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

ma0358412 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

ma0358412 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 3, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Validation summary

Uh oh!

popovaan commented Aug 4, 2026

Uh oh!

Mohamed-Ashraf273 commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Validation summary for PR #1909 (jina-vlm, model_type='jvlm', task image-text-to-text)

Functional / export

Accuracy (WWB — who-what-benchmark similarity)

Automated tests registered for jvlm (confirmed in diff)

Observations (non-blocking)

Uh oh!

Mohamed-Ashraf273 commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Validation summary — PR #1909 (jina-vlm, model_type='jvlm', task image-text-to-text)

Functional / export validation

Accuracy validation (WWB — who-what-benchmark similarity)

Automated tests registered for jvlm (confirmed in the diff)

Observations (non-blocking)

Uh oh!

Mohamed-Ashraf273 commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Uh oh!

popovaan commented Aug 4, 2026

Validation summary — PR #1909 (jina-vlm, model_type='jvlm', task image-text-to-text)

Functional / export validation

Accuracy validation (WWB — who-what-benchmark similarity)

Automated tests registered for jvlm (confirmed in the diff)

Observations (non-blocking)

Uh oh!

Mohamed-Ashraf273 Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

popovaan commented Aug 5, 2026

popovaan commented Aug 2, 2026 •

edited

Loading

Validation summary for PR #1909 (jina-vlm, `model_type='jvlm'`, task `image-text-to-text`)

Automated tests registered for `jvlm` (confirmed in diff)

Validation summary — PR #1909 (jina-vlm, `model_type='jvlm'`, task `image-text-to-text`)

Automated tests registered for `jvlm` (confirmed in the diff)

Validation summary — PR #1909 (jina-vlm, `model_type='jvlm'`, task `image-text-to-text`)

Automated tests registered for `jvlm` (confirmed in the diff)