Add multi components recipes

Qwen-Qwen3-VL-2B-Instruct/multi_comp/.gitignore

@@ -0,0 +1,12 @@
+# Exported ONNX packages
+exported_vlm_pkg/
+exported_vlm_gptq_pkg/
+
+# Quantized HF checkpoint
+vlm_decoder_gptq_hf/
+
+# Optimized components
+out/
+
+# Olive cache
+cache/

Qwen-Qwen3-VL-2B-Instruct/multi_comp/README.md

@@ -0,0 +1,178 @@
+# Qwen3-VL-2B-Instruct — Multi-Component Optimization
+
+These recipes demonstrate two multi-component flows for
+[Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct):
+
+- **Flow A — export first, then per-component optimization**
+  (`vlm_optimize_components.json`): export the VLM to ONNX once with the Mobius builder, then run a
+  single Olive config whose `builds` apply a **different pipeline to each component**.
+- **Flow B — optimize a Torch component first, then export**
+  (`vlm_quantize_then_export.json`): run a Torch-stage GPTQ pass on the decoder component while
+  saving a complete HF directory, then export that directory with
+  `olive capture-onnx-graph --use_mobius_builder`.
+
+Olive loads an exported directory as a `CompositeModel` whose **component names are the subfolder
+names**, so there is no need to memorize component names.
+
+## Prerequisites
+
+```
+pip install olive-ai
+pip install mobius-ai
+```
+
+Exporting also needs `transformers` and access to the model on Hugging Face.
+
+---
+
+## Recipe 1 — Export then per-component optimize (`vlm_optimize_components.json`)
+
+### Step 1 — Export
+
+```
+olive capture-onnx-graph --model_name_or_path Qwen/Qwen3-VL-2B-Instruct --use_mobius_builder --output_path exported_vlm_pkg
+```
+
+Mobius exports this model as three components, each in its own subfolder:
+
+```
+exported_vlm_pkg/
+  decoder/model.onnx
+  vision_encoder/model.onnx
+  embedding/model.onnx
+```
+
+### Step 2 — Optimize
+
+```
+olive run --config vlm_optimize_components.json
+```
+
+| component        | pipeline        | intent                              |
+|------------------|-----------------|-------------------------------------|
+| `decoder`        | `dynamic_quant` | INT8-quantize the language decoder  |
+| `vision_encoder` | `to_fp16`       | keep the vision tower in FP16       |
+| `embedding`      | `to_fp16`       | keep the embedding in FP16          |
+
+> The three component names (`decoder`, `vision_encoder`, `embedding`) are exactly what Mobius
+> produces for `Qwen/Qwen3-VL-2B-Instruct`. For a different VLM, adjust the component names in the
+> config to match the subfolder names your export actually produced.
+
+### Step 3 — Inference with ORT GenAI
+
+Run text generation with the exported ONNX models using **onnxruntime-genai**:
+
+```bash
+# Text-only
+python vlm_inference.py --prompt "The capital of France is"
+
+# With image input
+python vlm_inference.py --prompt "Describe this image." --image photo.jpg
+
+# Custom settings
+python vlm_inference.py --model_dir exported_vlm_pkg --max_new_tokens 256
+```
+
+The inference script (`vlm_inference.py`) uses ORT GenAI which handles:
+- **Tokenization**: built-in tokenizer from saved HF tokenizer files
+- **Embedding**: ONNX `embedding/model.onnx` (token embed + image feature mixing)
+- **Vision encoding**: ONNX `vision_encoder/model.onnx` (when `--image` is provided)
+- **Decoding**: ONNX `decoder/model.onnx` with KV cache (autoregressive generation)
+
+Options:
+```
+--prompt TEXT           Text prompt
+--image PATH            Optional image file for multimodal input
+--max_new_tokens N      Maximum tokens to generate (default: 128)
+--model_dir DIR         Path to exported model directory (default: exported_vlm_pkg)
+```
+
+#### Setup requirements
+
+The export directory needs these files alongside the ONNX models:
+
+```
+exported_vlm_pkg/
+  genai_config.json          # Model type, I/O mappings, search config
+  tokenizer.json             # HF tokenizer
+  tokenizer_config.json
+  vision_processor.json      # Vision preprocessing config
+  decoder/model.onnx
+  vision_encoder/model.onnx
+  embedding/model.onnx
+```
+
+To create the tokenizer files after export:
+
+```python
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-VL-2B-Instruct", trust_remote_code=True)
+tokenizer.save_pretrained("exported_vlm_pkg")
+```
+
+For the `genai_config.json` structure, see the
+[Mobius ORT GenAI examples](https://github.com/microsoft/mobius/tree/main/examples) which write the
+config automatically.
+
+> **Note.** Install `onnxruntime-genai` (`pip install onnxruntime-genai`) to use this script.
+
+---
+
+## Recipe 2 — Torch decoder quantization, then Mobius export (`vlm_quantize_then_export.json`)
+
+This is **Flow B**: quantize only the Torch decoder component first, then export the resulting
+complete HF directory with the Olive capture CLI using the Mobius builder.
+
+### Step 1 — Quantize the decoder component
+
+```
+olive run --config vlm_quantize_then_export.json
+```
+
+The config uses `builds.components: ["decoder"]`, so Olive asks Mobius for the VLM component plan,
+scopes the Torch GPTQ pass to the decoder submodule, and saves the original HF folder layout with
+the decoder quantized in place. This output is **not** a standalone decoder checkpoint; it is a
+complete HF model directory:
+
+```
+out/vlm_decoder_gptq_hf/
+```
+
+The recipe uses the GPTQ pass defaults for calibration data (`Salesforce/wikitext`). For production,
+add a `data_config` to `decoder_gptq` with your own text or multimodal calibration set.
+
+### Step 2 — Export the quantized HF directory with the Mobius builder
+
+```
+olive capture-onnx-graph \
+  --model_name_or_path vlm_decoder_gptq_hf \
+  --use_mobius_builder \
+  --trust_remote_code \
+  --precision fp16 \
+  --output_path exported_vlm_gptq_pkg
+```
+
+Output:
+
+```
+exported_vlm_gptq_pkg/
+  decoder/model.onnx
+  vision_encoder/model.onnx
+  embedding/model.onnx
+```
+
+> **Note.** The Torch GPTQ pass saves Olive-packed weights (`quant_method="olive"`). Use this export
+> step with a Mobius builder version that supports Olive-packed quantized HF checkpoints.
+
+---
+
+## Notes
+
+- The passes in Recipe 1 (`OnnxFloatToFloat16`, `OnnxDynamicQuantization`) are **illustrative** and
+  chosen to run without calibration data. Swap in `OrtTransformersOptimization`,
+  `OnnxStaticQuantization` (with a `data_config`), or other ONNX passes for production-quality
+  optimization.
+- The ONNX component recipe runs on the EP declared in its `systems` section. The Torch GPTQ recipe
+  targets CUDA because VLM decoder GPTQ is GPU-oriented.
+- `builds.components` selects which exported components to optimize. Only the components with a build
+  are touched; the rest remain as exported.

Qwen-Qwen3-VL-2B-Instruct/multi_comp/info.yml

@@ -0,0 +1,15 @@
+keywords:
+  - olive-ai
+recipes:
+  - name: qwen3vl-2B-Instruct
+    file: vlm_optimize_components.json
+    eps:
+      - CPUExecutionProvider
+    devices:
+      - cpu
+  - name: qwen3vl-2B-Instruct
+    file: vlm_quantize_then_export.json
+    eps:
+      - CUDAExecutionProvider
+    devices:
+      - gpu

Qwen-Qwen3-VL-2B-Instruct/multi_comp/vlm_inference.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+"""VLM (Qwen3-VL-2B-Instruct) inference using ORT GenAI with exported ONNX models.
+
+Usage:
+    # Text-only
+    python vlm_inference.py --prompt "The capital of France is"
+
+    # With image
+    python vlm_inference.py --prompt "Describe this image." --image photo.jpg
+
+    # Custom model directory
+    python vlm_inference.py --model_dir exported_vlm_pkg --prompt "What is 2+2?"
+"""
+
+import argparse
+import os
+
+import onnxruntime_genai as og
+
+
+def generate_text(model_dir: str, prompt: str, max_new_tokens: int = 128) -> str:
+    """Run text-only generation."""
+    model = og.Model(model_dir)
+    tokenizer = og.Tokenizer(model)
+
+    input_ids = tokenizer.encode(prompt)
+    params = og.GeneratorParams(model)
+    params.set_search_options(max_length=len(input_ids) + max_new_tokens)
+
+    generator = og.Generator(model, params)
+    generator.append_tokens(input_ids)
+
+    tokenizer_stream = tokenizer.create_stream()
+    generated = []
+    while not generator.is_done():
+        generator.generate_next_token()
+        token = generator.get_next_tokens()[0]
+        generated.append(token)
+        print(tokenizer_stream.decode(token), end="", flush=True)
+        if len(generated) >= max_new_tokens:
+            break
+
+    print()
+    del generator
+    return tokenizer.decode(generated)
+
+
+def generate_with_image(model_dir: str, prompt: str, image_path: str, max_new_tokens: int = 128) -> str:
+    """Run multimodal generation with image input."""
+    model = og.Model(model_dir)
+    tokenizer = og.Tokenizer(model)
+    processor = model.create_multimodal_processor()
+
+    images = og.Images.open(image_path)
+    inputs = processor(prompt, images=images)
+
+    params = og.GeneratorParams(model)
+    params.set_search_options(max_length=4096)
+
+    generator = og.Generator(model, params)
+    generator.set_inputs(inputs)
+
+    tokenizer_stream = tokenizer.create_stream()
+    generated = []
+    while not generator.is_done():
+        generator.generate_next_token()
+        token = generator.get_next_tokens()[0]
+        generated.append(token)
+        print(tokenizer_stream.decode(token), end="", flush=True)
+        if len(generated) >= max_new_tokens:
+            break
+
+    print()
+    del generator
+    return tokenizer.decode(generated)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="VLM inference with ORT GenAI")
+    parser.add_argument("--prompt", default="The capital of France is")
+    parser.add_argument("--image", default=None, help="Path to an image file for vision input")
+    parser.add_argument("--max_new_tokens", type=int, default=128)
+    parser.add_argument("--model_dir", default="exported_vlm_pkg")
+    args = parser.parse_args()
+
+    genai_config = os.path.join(args.model_dir, "genai_config.json")
+    if not os.path.exists(genai_config):
+        print(f"Error: genai_config.json not found in {args.model_dir}")
+        print("Run export first:")
+        print(
+            "  olive capture-onnx-graph --model_name_or_path Qwen/Qwen3-VL-2B-Instruct "
+            "--use_mobius_builder --output_path exported_vlm_pkg"
+        )
+        print("Then create genai_config.json and save tokenizer (see README.md).")
+        return
+
+    print(f"Model: {args.model_dir}")
+    print(f"Prompt: {args.prompt}")
+    if args.image:
+        print(f"Image: {args.image}")
+    print("-" * 50)
+
+    if args.image:
+        output = generate_with_image(args.model_dir, args.prompt, args.image, args.max_new_tokens)
+    else:
+        output = generate_text(args.model_dir, args.prompt, args.max_new_tokens)
+
+    print("-" * 50)
+    print(f"Output: {output}")
+
+
+if __name__ == "__main__":
+    main()

Qwen-Qwen3-VL-2B-Instruct/multi_comp/vlm_optimize_components.json

@@ -0,0 +1,20 @@
+{
+    "input_model": { "type": "CompositeModel", "config": { "model_path": "exported_vlm_pkg" } },
+    "systems": {
+        "local_system": {
+            "type": "LocalSystem",
+            "accelerators": [ { "device": "cpu", "execution_providers": [ "CPUExecutionProvider" ] } ]
+        }
+    },
+    "passes": { "to_fp16": { "type": "OnnxFloatToFloat16" }, "dynamic_quant": { "type": "OnnxDynamicQuantization" } },
+    "engine": { "host": "local_system", "target": "local_system", "evaluate_input_model": false, "cache_dir": "cache" },
+    "builds": {
+        "decoder": { "components": [ "decoder" ], "pipeline": [ "dynamic_quant" ], "output_dir": "out/decoder" },
+        "vision_encoder": {
+            "components": [ "vision_encoder" ],
+            "pipeline": [ "to_fp16" ],
+            "output_dir": "out/vision_encoder"
+        },
+        "embedding": { "components": [ "embedding" ], "pipeline": [ "to_fp16" ], "output_dir": "out/embedding" }
+    }
+}

Qwen-Qwen3-VL-2B-Instruct/multi_comp/vlm_quantize_then_export.json

@@ -0,0 +1,25 @@
+{
+    "input_model": {
+        "type": "HfModel",
+        "config": {
+            "model_path": "Qwen/Qwen3-VL-2B-Instruct",
+            "task": "image-text-to-text",
+            "load_kwargs": { "torch_dtype": "float16", "trust_remote_code": true }
+        }
+    },
+    "systems": {
+        "local_gpu": {
+            "type": "LocalSystem",
+            "accelerators": [ { "device": "gpu", "execution_providers": [ "CUDAExecutionProvider" ] } ]
+        }
+    },
+    "passes": { "decoder_gptq": { "type": "Gptq", "bits": 4, "group_size": 128, "sym": true, "lm_head": false } },
+    "engine": { "host": "local_gpu", "target": "local_gpu", "evaluate_input_model": false, "cache_dir": "cache" },
+    "builds": {
+        "decoder_gptq": {
+            "components": [ "decoder" ],
+            "pipeline": [ "decoder_gptq" ],
+            "output_dir": "vlm_decoder_gptq_hf"
+        }
+    }
+}

stabilityai-stable-diffusion-3-medium-diffusers/LICENSE

@@ -0,0 +1,14 @@
+Stable Diffusion 3 Medium is released under the Stability AI Community License.
+
+The model "stabilityai/stable-diffusion-3-medium-diffusers" is a gated model. You must
+review and accept its license on Hugging Face before downloading or using the weights:
+
+  https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers
+
+Full license text:
+
+  https://stability.ai/license
+
+The recipes in this directory are provided by the olive-recipes project under the
+repository's root LICENSE. The license referenced above governs use of the Stable
+Diffusion 3 model weights only.

stabilityai-stable-diffusion-3-medium-diffusers/multi_comp/.gitignore

@@ -0,0 +1,12 @@
+# Exported ONNX packages
+exported_pkg/
+exported_sd3_full2/
+
+# Optimized components
+out/
+
+# Olive cache
+cache/
+
+# Generated images
+*.png