Skip to content

[OpenVINO] Support Falcon-H1-0.5B-Instruct with task text-generation - #4251

Draft
popovaan wants to merge 1 commit into
openvinotoolkit:masterfrom
popovaan:model-support-Falcon-H1-0.5B-Instruct-openvino_genai
Draft

[OpenVINO] Support Falcon-H1-0.5B-Instruct with task text-generation#4251
popovaan wants to merge 1 commit into
openvinotoolkit:masterfrom
popovaan:model-support-Falcon-H1-0.5B-Instruct-openvino_genai

Conversation

@popovaan

@popovaan popovaan commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Description

Enabled Falcon-H1 (hybrid Mamba+attention, FalconH1ForCausalLM) for OpenVINO GenAI. Root cause: explicit PagedAttention request failed with 'Model references undeclared parameters: beam_idx' because SDPAToPagedAttention cannot rewrite the recurrent Mamba conv/ssm state. Fix: in all three LLMPipeline constructors, catch the specific PA model-construction failure (is_paged_attention_model_construction_failure) and fall back to the stateful backend, stripping scheduler_config. model-checker EXIT=0: GenAI llm_bench 1st=2.34ms 2nd=1.67ms/tok tput=597 tok/s; optimum similarity 1.0 and GenAI similarity 1.0 (threshold 0.95). Verified no regression: pytest test_llm_pipeline.py -k linear -> 32 passed, 6 skipped (qwen3-next STATEFUL+PAGED_ATTENTION). Docs updated with FalconH1ForCausalLM entry (npm build passed). Test wiring added to LINEAR_ATTENTION_MODELS_LIST but left commented: no non-degenerate tiny-random FalconH1 fixture is published; public community fixtures (tiny-random/falcon-h1, yujiepan/falcon-h1-tiny-random) segfault due to degenerate configs.

Reproduce generation

import openvino_genai

pipe = openvino_genai.LLMPipeline("output_dir", "CPU")
print(pipe.generate("What is the capital of France?", max_new_tokens=10))

Validation

  • Validation model: Tiny model (/home/openvino_bot/.local/share/openvino-model-agent/requests/issue-39/repository/workspace/tiny_falcon_h1)
  • WWB Optimum similarity: 1.0
  • WWB GenAI similarity: 1.0
  • First-token latency: 2.34
  • Throughput: 597.08

Final real-model validation

  • Estimated FP16 size: Not available
  • Available RAM at validation: 122.39 GiB
  • 70% RAM limit: 85.67 GiB
  • Status: Skipped
  • Reason: Could not estimate FP16 model size: No module named 'huggingface_hub'

Related model-support PRs

Checklist

  • This PR follows the GenAI contributing guidelines.
  • Tests have been updated or added to cover the new code.
  • This PR fully addresses the model-support work.
  • Documentation was updated, or its absence is explained in the validation handoff.

Copilot AI review requested due to automatic review settings August 1, 2026 21:11
@github-actions github-actions Bot added category: LLM LLM pipeline (stateful, static) no-match-files category: GH Pages Docs Github Pages documentation category: GGUF GGUF file reader labels Aug 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables Falcon-H1 (hybrid Mamba+attention) usage in OpenVINO GenAI text-generation by making LLMPipeline robust to PagedAttention/ContinuousBatching construction failures that occur when SDPAToPagedAttention cannot rewrite recurrent Mamba state. It also updates supported-model documentation and leaves test-model wiring notes for future regression coverage once a stable tiny-random fixture exists.

Changes:

  • Added utils::is_paged_attention_model_construction_failure() to detect the specific “undeclared/unregistered beam_idx” incompatibility during PagedAttention model construction.
  • Updated all three LLMPipeline constructors to catch that specific failure and fall back to the stateful backend (dropping scheduler_config).
  • Added FalconH1ForCausalLM to the supported LLM architectures list in site docs; updated test model list notes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/python_tests/data/models.py Adds an explanatory (commented) placeholder for a future Falcon-H1 tiny-random regression fixture.
src/cpp/src/utils.hpp Declares a new helper for detecting PA model-construction incompatibility from exceptions.
src/cpp/src/utils.cpp Implements the PA incompatibility detector using message substring matching.
src/cpp/src/llm/pipeline.cpp Adds explicit-PA constructor fallback logic to stateful backend when the model cannot be converted to PagedAttention.
site/docs/supported-models/_components/llm-models-table/models.ts Documents Falcon-H1 architectures/models as supported.
Suppressed comments (2)

src/cpp/src/llm/pipeline.cpp:319

  • The comment says the constructor will re-throw when Continuous Batching (PagedAttention) is invoked explicitly, but the new logic also falls back to the stateful backend for known model-construction incompatibilities. Updating the comment will keep it consistent with the actual behavior.
    } else if (utils::explicitly_requires_paged_attention(user_properties)) {
        // If CB is invoked explicitly, create CB adapter as is and re-throw in case if internal issues
        auto [device_properties, scheduler_config] = utils::extract_scheduler_config(properties, utils::get_latency_oriented_scheduler_config());

src/cpp/src/llm/pipeline.cpp:383

  • The comment says the constructor will re-throw when Continuous Batching (PagedAttention) is invoked explicitly, but the new logic also falls back to the stateful backend for known model-construction incompatibilities. Updating the comment will keep it consistent with the actual behavior.
    } else if (utils::explicitly_requires_paged_attention(user_properties)) {
        // If CB is invoked explicitly, create CB adapter as is and re-throw in case if internal issues
        auto [device_properties, scheduler_config] = utils::extract_scheduler_config(properties, utils::get_latency_oriented_scheduler_config());

Comment on lines 259 to 261
} else if (utils::explicitly_requires_paged_attention(user_properties)) {
// If CB is invoked explicitly, create CB adapter as is and re-throw in case if internal issues
auto [device_properties, scheduler_config] = utils::extract_scheduler_config(properties, utils::get_latency_oriented_scheduler_config());
Comment thread src/cpp/src/utils.cpp
Comment on lines +937 to +949
bool is_paged_attention_model_construction_failure(const ov::Exception& exception) {
const std::string message = exception.what();
// The SDPAToPagedAttention transformation fails to convert hybrid Mamba/attention models (e.g. Falcon-H1):
// the recurrent convolution/SSM state keeps referencing the beam_idx parameter after the parameter itself is
// removed, which ov::Model validation reports as an undeclared/unregistered beam_idx parameter. Match this
// specific model-construction failure so only genuinely PagedAttention-incompatible models fall back to the
// stateful backend, while models that merely hit a transient runtime error still surface it.
const bool references_beam_idx = message.find("beam_idx") != std::string::npos;
const bool undeclared_parameter =
message.find("undeclared parameters") != std::string::npos ||
message.find("unregistered_parameters") != std::string::npos;
return references_beam_idx && undeclared_parameter;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: GGUF GGUF file reader category: GH Pages Docs Github Pages documentation category: LLM LLM pipeline (stateful, static) no-match-files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants