Properly isolated selftests - #6338
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
WalkthroughThe change adds pre-import filtering for disabled plugins and separates fully qualified name handling for extensions and entry points. The self-test runner now isolates source-tree execution, supports bounded parallelism, normalizes repeated selection options, and excludes unavailable or disabled optional plugins. Functional tests now support missing PyYAML, failed man-page commands, and JSON variant fixtures. A dispatcher test verifies that disabled plugins are not loaded. Estimated code review effort: 3 (Moderate) | ~25 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@selftests/check.py`:
- Around line 17-23: Add the Ansible plugin mapping to OPTIONAL_PLUGINS using
the key “ansible” and package “avocado-framework-plugin-ansible”, so
_external_plugins() preserves the Ansible entry points required by the
nrunner-interface avocado-runner-ansible-module variant.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f89d75a2-c9ae-4f80-ac6e-7e801a79a5a1
📒 Files selected for processing (8)
avocado/core/enabled_extension_manager.pyavocado/core/extension_manager.pyselftests/check.pyselftests/functional/basic.pyselftests/functional/plugin/jsonresult.pyselftests/functional/plugin/tmtresult.pyselftests/functional/plugin/xunit.pyselftests/unit/dispatcher.py
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6338 +/- ##
==========================================
- Coverage 71.74% 71.39% -0.36%
==========================================
Files 207 207
Lines 23624 23637 +13
==========================================
- Hits 16949 16875 -74
- Misses 6675 6762 +87 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Allow callers to cap parallel tasks while preserving suites that require serial execution. Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
Assert the TestInterrupt message emitted by the current runner instead of the obsolete SIGTERM RuntimeError text. Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
Avoid loading disabled plugin modules so missing optional dependencies and import side effects cannot pollute unrelated commands as well as the avocado (isolation) selftests. This is done with a separate enabled entry point switch than the one used for extensions (generalized now and default). Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
16df151 to
dccde94
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@selftests/check.py`:
- Around line 938-964: Update create_suites() so optional-plugin test references
are added only when the corresponding plugin distribution is available and not
listed in args.disable_plugin_checks, matching the TEST_SIZE adjustments for
golang, html, robot, varianter_cit, and varianter_yaml_to_mux. Alternatively,
derive TEST_SIZE from the references actually selected, while preserving
consistent suite counts.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 810d5f9d-12cc-4ac3-854d-a77212b19775
📒 Files selected for processing (1)
selftests/check.py
dccde94 to
79fdd53
Compare
Use checkout-local plugin paths and temporary configuration while excluding unrelated installations. Honor any plugin exclusions consistently and only test runner commands that exist. The main way to achieve proper test isolation here is by changing the system python paths and placing any avocado configs for the selftests in a temporary directory. Selftest-related plugins are allowed separately while test counts are kept consistent across the check script. Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
This is a better choice for testing the same functionality than the explicit sometimes disabled varianter_yaml_to_mux plugin which may not be available in all environments. Also improve command failure in the tests instead of a misleading "missing file" error if the command fails to make them faster to debug. Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
Treat an unavailable manual page as an inconclusive outcome instead of failing module import. Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
79fdd53 to
e462c56
Compare
|
@PraveenPenguin This branch mostly improves the test isolation of the avocado selftests so we can also run them in other environments outside of the CI (e.g. LXC or other containers). |
1 participant
Summary by CodeRabbit
New Features
--max-parallel-tasksto limit concurrent test execution while preserving serial test suites.Bug Fixes
Tests