[OpenVINO] Support jina-vlm with task image-text-to-text

[popovaan]

Description

Fixed GenAI JinaVLM prefill tokenization: (1) prepend BOS <|endoftext|> id 151643 (OV tokenizer never adds it), (2) preserve the chat-template leading space that OV apply_chat_template dropped (' User:' token 2657 vs 'User' 1474), and (3) correct image_masks crop ordering (thumbnail -1 row last, matching the reference). Diagnosed by dumping actual C++ image_patches/image_masks/input_ids and comparing byte-for-byte to the HF processor on the failing WWB image: after the fix input_ids and image_masks are byte-identical to reference; image_patches differ only <=2/255 from inherent torchvision uint8 resize rounding. Rebuilt local checkout (openvino_genai.file = openvino.genai/build/openvino_genai). model-checker on the REAL model jinaai/jina-vlm reusing FP16 IR and HF ground truth (--skip-export --skip-llm-bench --skip-wwb-ground-truth --num-samples 4): Optimum similarity 1.0000 (PASS), GenAI similarity 0.9506 >= 0.95 (PASS, was 0.9345). GenAI vs optimum verified token-identical on the local tiny model for text-only and image 100x77/64x64 (the exact test_vlm_pipeline_match_optimum_with_resolutions assertion). Repo test wired for optimum-intel-internal-testing/tiny-random-jvlm but that Hub id is not published (404), a tiny_model_creator fixture blocker; enablement source+docs are complete and WWB passes.

Reproduce generation

from PIL import Image
import openvino_genai

pipe = openvino_genai.VLMPipeline("output_dir", "CPU")
image = Image.new("RGB", (64, 64), "white")
result = pipe.generate("Describe this image.", image=image, max_new_tokens=10)
print(result.texts[0])

Validation

• Validation model: Tiny model (/home/openvino_bot/.local/share/openvino-model-agent/requests/issue-40/repository/workspace/tiny_jina_vlm)
• WWB Optimum similarity: 1.0
• WWB GenAI similarity: 0.9506
• First-token latency: Not run
• Throughput: Not run

Final real-model validation

• Estimated FP16 size: 4.62 GiB
• Available RAM at validation: 122.60 GiB
• 70% RAM limit: 85.82 GiB
• Model: Real model (jinaai/jina-vlm)
• WWB Optimum similarity: 1.0
• WWB GenAI similarity: 0.95056

Related model-support PRs

• Optimum Intel: [OpenVINO] Support jina-vlm with task image-text-to-text huggingface/optimum-intel#1909

Checklist

• This PR follows the GenAI contributing guidelines.
• Tests have been updated or added to cover the new code.
• This PR fully addresses the model-support work.
• Documentation was updated, or its absence is explained in the validation handoff.

[Copilot]

Pull request overview

Adds OpenVINO GenAI enablement for JinaVLM (task: image-text-to-text) by integrating a new JVLM model type end-to-end (config, vision encoder preprocessing, prompt normalization, and embedding merge), plus associated WWB and test harness updates.

Changes:

• Add JVLM model support in C++ VLMPipeline (new VLMModelType::JVLM, JVLM-specific VisionEncoder + InputsEmbedder implementations).
• Extend Who-What-Benchmark to support JVLM model loading and allow using a local CSV as an offline dataset source.
• Update Python tests/utilities and documentation tables to include the new architecture/model entry.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/who_what_benchmark/whowhatbench/wwb.py	Allow --dataset to be a local CSV (including optional images/videos columns).
tools/who_what_benchmark/whowhatbench/model_loaders.py	Ensure JVLM loads via AutoModelForCausalLM to get a generate()-capable class.
tools/who_what_benchmark/whowhatbench/inputs_preprocessors/jvlm.py	Add HF-side JVLM input preprocessing using the processor chat template + multimodal content.
tools/who_what_benchmark/whowhatbench/inputs_preprocessors/init.py	Register the new JVLM input preprocessor mapping.
tests/python_tests/utils/hugging_face.py	Add JVLM tiny model to TRUST_REMOTE_CODE_MODELS.
tests/python_tests/test_vlm_pipeline.py	Add JVLM tiny model ID to the VLM test matrix and model-specific settings.
src/cpp/src/visual_language/vlm_config.hpp	Add VLMModelType::JVLM and JVLM token string fields in VLMConfig.
src/cpp/src/visual_language/vlm_config.cpp	Map "jvlm" model type and read JVLM-specific hidden_size.
src/cpp/src/visual_language/vision_encoder.cpp	Register VisionEncoderJVLM in the factory.
src/cpp/src/visual_language/jvlm/classes.hpp	Declare JVLM vision encoder + inputs embedder classes and preprocessing params.
src/cpp/src/visual_language/jvlm/classes.cpp	Implement Molmo-style overlap-and-resize preprocessing + <im_patch> scatter merge + tokenization fixes.
src/cpp/src/visual_language/inputs_embedder.hpp	Add JVLM embedder friend declaration.
src/cpp/src/visual_language/inputs_embedder.cpp	Register InputsEmbedderJVLM in the factory.
site/docs/supported-models/_components/vlm-models-table/models.ts	Document JinaVLM architecture in the supported VLM models table.
.model_analysis/jvlm_analysis.md	Add model analysis/design notes for JVLM enablement.
.model_analysis/inspect_ir.py	Add an IR inspection helper script.

[Mohamed-Ashraf273]

@popovaan Please summarize the validation performed in this PR. Do not modify any files.

[popovaan]

👀 @Mohamed-Ashraf273, review request received.

[popovaan]

Validation summary for PR #4252 (jina-vlm, image-text-to-text)

Based on the PR description and the accompanying .model_analysis/jvlm_analysis.md, the validation performed was:

WWB (who_what_benchmark) accuracy

Target	Optimum similarity	GenAI similarity
Tiny model (tiny_jina_vlm)	1.0	0.9506
Real model (jinaai/jina-vlm, FP16 IR)	1.0	0.95056

• Real-model run used model-checker reusing the FP16 IR + HF ground truth with --skip-export --skip-llm-bench --skip-wwb-ground-truth --num-samples 4. Optimum similarity 1.0000 (PASS); GenAI 0.9506 ≥ 0.95 threshold (PASS; improved from 0.9345 before the fix).
• Resource footprint recorded: estimated FP16 size 4.62 GiB, 122.60 GiB RAM available, 70% limit 85.82 GiB.

Byte-level tokenization / preprocessing verification

• After the three fixes (prepend BOS <|endoftext|> id 151643; preserve chat-template leading space so ' User:'=2657 rather than 'User'=1474; correct image_masks crop ordering with the thumbnail -1 row last), the C++ input_ids and image_masks are reported byte-identical to the HF processor on the failing WWB image.
• image_patches differ by only ≤2/255, attributed to inherent torchvision uint8 resize rounding.
• On the local tiny model, GenAI vs optimum were token-identical for text-only and for images at 100x77 and 64x64 (the test_vlm_pipeline_match_optimum_with_resolutions assertion).

Not run / gaps

• First-token latency and throughput: not measured.
• The repo test is wired to optimum-intel-internal-testing/tiny-random-jvlm, but that Hub id is not published (404), so the automated test cannot currently run in CI without the tiny model / tiny_model_creator fixture. This is called out as a blocker.
• Depends on Optimum Intel PR [OpenVINO] Support jina-vlm with task image-text-to-text huggingface/optimum-intel#1909.

Reviewer notes

• The performance numbers (latency/throughput) are absent, and the byte-identical / similarity claims are self-reported in the PR text; they were not independently reproduced in this triage.
• The blocking test-fixture 404 should be resolved (published tiny model) before merge so CI actually exercises the new path.
• The diff also adds analysis artifacts under .model_analysis/ (e.g. inspect_ir.py, jvlm_analysis.md); confirm whether these are intended to be committed.

[Mohamed-Ashraf273]

@popovaan Please summarize the validation performed in this PR. Do not modify any files.

[popovaan]

👀 @Mohamed-Ashraf273, review request received.

[popovaan]

Validation summary — PR #4252 (jina-vlm, image-text-to-text)

WWB (who_what_benchmark) accuracy

Target	Optimum similarity	GenAI similarity
Tiny model (tiny_jina_vlm)	1.0	0.9506
Real model (jinaai/jina-vlm, FP16 IR)	1.0	0.95056

• Real-model run used model-checker reusing the FP16 IR + HF ground truth with --skip-export --skip-llm-bench --skip-wwb-ground-truth --num-samples 4. Optimum similarity 1.0000 (PASS); GenAI 0.9506 ≥ 0.95 threshold (PASS; improved from 0.9345 before the fix).
• Resource footprint recorded: estimated FP16 size 4.62 GiB, 122.60 GiB RAM available, 70% limit 85.82 GiB.

Byte-level tokenization / preprocessing verification

Three fixes were validated against the HF processor on the failing WWB image:

1. Prepend BOS <|endoftext|> id 151643 (OV tokenizer never adds it).
2. Preserve the chat-template leading space (' User:'=2657 vs 'User'=1474) that OV apply_chat_template dropped.
3. Correct image_masks crop ordering (thumbnail -1 row last).

After the fixes: C++ input_ids and image_masks are reported byte-identical to the HF processor; image_patches differ by only ≤2/255 (attributed to inherent torchvision uint8 resize rounding). On the local tiny model, GenAI vs optimum were token-identical for text-only and for images at 100×77 and 64×64 (the test_vlm_pipeline_match_optimum_with_resolutions assertion).

Not run / gaps

• First-token latency and throughput: not measured.
• The repo test is wired to optimum-intel-internal-testing/tiny-random-jvlm, but that Hub id is not published (404), so CI cannot currently exercise the new path (a tiny_model_creator fixture blocker called out in the PR).
• Depends on Optimum Intel PR [OpenVINO] Support jina-vlm with task image-text-to-text huggingface/optimum-intel#1909.

Reviewer notes (not independently reproduced in triage)

• The similarity and byte-identical claims are self-reported in the PR text; they were not re-run here.
• Performance numbers (latency/throughput) are absent.
• The blocking test-fixture 404 should be resolved (published tiny model) before merge so CI actually runs the new path.
• The diff adds analysis artifacts under .model_analysis/ (inspect_ir.py, jvlm_analysis.md) — confirm whether these are intended to be committed.

[Mohamed-Ashraf273]

@popovaan Please summarize the validation performed in this PR. Do not modify any files.

[popovaan]

👀 @Mohamed-Ashraf273, review request received.

[popovaan]

Validation summary — PR #4252 (jina-vlm, image-text-to-text)

WWB (who_what_benchmark) accuracy

Target	Optimum similarity	GenAI similarity
Tiny model (tiny_jina_vlm)	1.0	0.9506
Real model (jinaai/jina-vlm, FP16 IR)	1.0	0.95056

• Real-model run used model-checker reusing the FP16 IR + HF ground truth with --skip-export --skip-llm-bench --skip-wwb-ground-truth --num-samples 4. Optimum similarity 1.0000 (PASS); GenAI 0.9506 ≥ 0.95 threshold (PASS; reported improvement from 0.9345 before the fix).
• Resource footprint recorded: estimated FP16 size 4.62 GiB; 122.60 GiB RAM available; 70% limit 85.82 GiB.

Byte-level tokenization / preprocessing verification

Three fixes were validated against the HF processor on the failing WWB image:

1. Prepend BOS <|endoftext|> id 151643 (OV tokenizer never adds it).
2. Preserve the chat-template leading space (' User:'=2657 vs 'User'=1474) that OV apply_chat_template dropped.
3. Correct image_masks crop ordering (thumbnail -1 row last).

After the fixes: C++ input_ids and image_masks are reported byte-identical to the HF processor; image_patches differ only ≤2/255 (attributed to inherent torchvision uint8 resize rounding). On the local tiny model, GenAI vs optimum were reported token-identical for text-only and for images at 100×77 and 64×64 (the test_vlm_pipeline_match_optimum_with_resolutions assertion).

Not run / gaps

• First-token latency and throughput: not measured.
• The added repo test is wired to optimum-intel-internal-testing/tiny-random-jvlm, but that Hub id is not published (404), so CI cannot currently exercise the new path (a tiny_model_creator fixture blocker called out in the PR).
• Depends on Optimum Intel PR [OpenVINO] Support jina-vlm with task image-text-to-text huggingface/optimum-intel#1909.

Reviewer notes (not independently reproduced in this triage)

• The similarity and byte-identical claims are self-reported in the PR text; they were not re-run here.
• The diff adds analysis artifacts under .model_analysis/ (inspect_ir.py, jvlm_analysis.md) — confirm whether these are intended to be committed.
• Resolve the test-fixture 404 (publish the tiny model) before merge so CI actually runs the new path.