[OpenVINO] Support Falcon-H1-0.5B-Instruct with task text-generation - #4251
Draft
popovaan wants to merge 1 commit into
Draft
Conversation
Draft
2 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enables Falcon-H1 (hybrid Mamba+attention) usage in OpenVINO GenAI text-generation by making LLMPipeline robust to PagedAttention/ContinuousBatching construction failures that occur when SDPAToPagedAttention cannot rewrite recurrent Mamba state. It also updates supported-model documentation and leaves test-model wiring notes for future regression coverage once a stable tiny-random fixture exists.
Changes:
- Added
utils::is_paged_attention_model_construction_failure()to detect the specific “undeclared/unregistered beam_idx” incompatibility during PagedAttention model construction. - Updated all three
LLMPipelineconstructors to catch that specific failure and fall back to the stateful backend (droppingscheduler_config). - Added
FalconH1ForCausalLMto the supported LLM architectures list in site docs; updated test model list notes.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/python_tests/data/models.py |
Adds an explanatory (commented) placeholder for a future Falcon-H1 tiny-random regression fixture. |
src/cpp/src/utils.hpp |
Declares a new helper for detecting PA model-construction incompatibility from exceptions. |
src/cpp/src/utils.cpp |
Implements the PA incompatibility detector using message substring matching. |
src/cpp/src/llm/pipeline.cpp |
Adds explicit-PA constructor fallback logic to stateful backend when the model cannot be converted to PagedAttention. |
site/docs/supported-models/_components/llm-models-table/models.ts |
Documents Falcon-H1 architectures/models as supported. |
Suppressed comments (2)
src/cpp/src/llm/pipeline.cpp:319
- The comment says the constructor will re-throw when Continuous Batching (PagedAttention) is invoked explicitly, but the new logic also falls back to the stateful backend for known model-construction incompatibilities. Updating the comment will keep it consistent with the actual behavior.
} else if (utils::explicitly_requires_paged_attention(user_properties)) {
// If CB is invoked explicitly, create CB adapter as is and re-throw in case if internal issues
auto [device_properties, scheduler_config] = utils::extract_scheduler_config(properties, utils::get_latency_oriented_scheduler_config());
src/cpp/src/llm/pipeline.cpp:383
- The comment says the constructor will re-throw when Continuous Batching (PagedAttention) is invoked explicitly, but the new logic also falls back to the stateful backend for known model-construction incompatibilities. Updating the comment will keep it consistent with the actual behavior.
} else if (utils::explicitly_requires_paged_attention(user_properties)) {
// If CB is invoked explicitly, create CB adapter as is and re-throw in case if internal issues
auto [device_properties, scheduler_config] = utils::extract_scheduler_config(properties, utils::get_latency_oriented_scheduler_config());
Comment on lines
259
to
261
| } else if (utils::explicitly_requires_paged_attention(user_properties)) { | ||
| // If CB is invoked explicitly, create CB adapter as is and re-throw in case if internal issues | ||
| auto [device_properties, scheduler_config] = utils::extract_scheduler_config(properties, utils::get_latency_oriented_scheduler_config()); |
Comment on lines
+937
to
+949
| bool is_paged_attention_model_construction_failure(const ov::Exception& exception) { | ||
| const std::string message = exception.what(); | ||
| // The SDPAToPagedAttention transformation fails to convert hybrid Mamba/attention models (e.g. Falcon-H1): | ||
| // the recurrent convolution/SSM state keeps referencing the beam_idx parameter after the parameter itself is | ||
| // removed, which ov::Model validation reports as an undeclared/unregistered beam_idx parameter. Match this | ||
| // specific model-construction failure so only genuinely PagedAttention-incompatible models fall back to the | ||
| // stateful backend, while models that merely hit a transient runtime error still surface it. | ||
| const bool references_beam_idx = message.find("beam_idx") != std::string::npos; | ||
| const bool undeclared_parameter = | ||
| message.find("undeclared parameters") != std::string::npos || | ||
| message.find("unregistered_parameters") != std::string::npos; | ||
| return references_beam_idx && undeclared_parameter; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enabled Falcon-H1 (hybrid Mamba+attention, FalconH1ForCausalLM) for OpenVINO GenAI. Root cause: explicit PagedAttention request failed with 'Model references undeclared parameters: beam_idx' because SDPAToPagedAttention cannot rewrite the recurrent Mamba conv/ssm state. Fix: in all three LLMPipeline constructors, catch the specific PA model-construction failure (is_paged_attention_model_construction_failure) and fall back to the stateful backend, stripping scheduler_config. model-checker EXIT=0: GenAI llm_bench 1st=2.34ms 2nd=1.67ms/tok tput=597 tok/s; optimum similarity 1.0 and GenAI similarity 1.0 (threshold 0.95). Verified no regression: pytest test_llm_pipeline.py -k linear -> 32 passed, 6 skipped (qwen3-next STATEFUL+PAGED_ATTENTION). Docs updated with FalconH1ForCausalLM entry (npm build passed). Test wiring added to LINEAR_ATTENTION_MODELS_LIST but left commented: no non-degenerate tiny-random FalconH1 fixture is published; public community fixtures (tiny-random/falcon-h1, yujiepan/falcon-h1-tiny-random) segfault due to degenerate configs.
Reproduce generation
Validation
/home/openvino_bot/.local/share/openvino-model-agent/requests/issue-39/repository/workspace/tiny_falcon_h1)Final real-model validation
Related model-support PRs
Checklist