Skip to content

Create test_irs.py to detect changes in tiny-model IRs - #1913

Open
anatyrova wants to merge 6 commits into
huggingface:mainfrom
anatyrova:ir-tests
Open

Create test_irs.py to detect changes in tiny-model IRs#1913
anatyrova wants to merge 6 commits into
huggingface:mainfrom
anatyrova:ir-tests

Conversation

@anatyrova

@anatyrova anatyrova commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

This PR implements test_irs.py to detect changes in tiny-model IRs against reference IRs.
It will allow us to accelerate moving to new transformers version by default and check that IRs generated for newer transformers versions stay performant. Git LFS is implemented to store the reference IRs, that are generated with latest for optimum-intel 5.5.4 transformers.

P.S. more tiny models will be added soon here

pytest tests/openvino/test_irs.py -v

eg. small part of failed traceback:

E             Not equal output element types of __module.encoder.block.4.layer.1.layer_norm/aten::mul/Multiply_1 and Constant_35780 on 0 index model_one: <Type: 'float32'>, model_two: <Type: 'int64_t'>.
E             Not equal op partial shapes of self.encoder.block.4.layer.1.DenseReluDense.wi.weight and __module.encoder.final_layer_norm/aten::mean/ReduceMean on 0 index model_one: [37,32], model_two: [?,?,1].
E             Not equal op partial shapes of __module.encoder.block.4.layer.1.DenseReluDense.wi/aten::linear/MatMul and Constant_39730 on 0 index model_one: [?,?,37], model_two: [1,1,1].
E             Not equal op partial shapes of __module.encoder.block.4.layer.1.DenseReluDense.act/aten::relu/Relu and __module.encoder.final_layer_norm/aten::add/Add on 0 index model_one: [?,?,37], model_two: [?,?,1].
E             Not equal op partial shapes of self.encoder.block.4.layer.1.DenseReluDense.wo.weight and __module.encoder.final_layer_norm/aten::rsqrt/Sqrt on 0 index model_one: [32,37], model_two: [?,?,1].
E             Not equal op partial shapes of __module.encoder.block.4.layer.1.DenseReluDense.wo/aten::linear/MatMul and __module.encoder.final_layer_norm/aten::rsqrt/Divide on 0 index model_one: [?,?,32], model_two: [?,?,1].
E             Not equal op partial shapes of Constant_40674 and __module.encoder.final_layer_norm/aten::mul/Multiply_1 on 0 index model_one: [1,1,1], model_two: [?,?,32].
E             Not equal op partial shapes of Constant_40672 and Result_35882 on 0 index model_one: [1,1,1], model_two: [?,?,32].
E           
E           Reference IR was generated with:
E             - transformers: 5.5.4
E             - optimum-intel: 2.1.0.dev0+fd94990
E             - openvino: 2026.2.1-21919-ede283a88e3-releases/2026/2
E             - date: 2026-08-03
E           
E           This may indicate a regression or improvement in IR generation. Review the changes and update reference IRs if the change is intentional.

tests/openvino/test_irs.py:222: Failed

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@anatyrova
anatyrova requested a review from echarlaix August 3, 2026 17:40
@anatyrova anatyrova changed the title Create test_irs.py Create test_irs.py to detect changes in tiny-model IRs Aug 3, 2026
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@anatyrova
anatyrova requested a review from rkazants August 3, 2026 17:43

@rkazants rkazants left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anatyrova
please revise changes, something wrong +133,707

Comment thread tests/openvino/test_irs.py Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove json files

@rkazants rkazants left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long will it take if we generate reference IRs during testing time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants