Add AI-driven LISA test case selection pipeline for Azure DevOps#4574
Open
LiliDeng wants to merge 2 commits into
Open
Add AI-driven LISA test case selection pipeline for Azure DevOps#4574LiliDeng wants to merge 2 commits into
LiliDeng wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the AI-driven LISA “optional tests” selection flow from GitHub Actions to an Azure DevOps PR pipeline, while enhancing the selector script to support Azure OpenAI (in addition to GitHub Models) and to emit per-testcase runbook entries plus a selected Marketplace image.
Changes:
- Removed the existing GitHub Actions workflows for AI test selection, execution, and cancel-cleanup.
- Added an Azure DevOps PR pipeline that generates a PR diff, runs AI-based case selection, executes selected LISA tests on Azure, publishes artifacts/results, and comments back on the GitHub PR.
- Extended
.github/scripts/ai_select_cases.pywith Azure OpenAI support and updated selection/runbook output behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.github/workflows/ai-test-selection.yml |
Removed GitHub Actions-based AI selection + Azure test execution + PR reporting workflow. |
.github/workflows/ai-test-selection-cancel-cleanup.yml |
Removed GitHub Actions workflow_run-based cleanup for cancelled runs. |
.github/scripts/ai_select_cases.py |
Added Azure OpenAI support, refactored model prompting, added comment-only shortcut, and updated runbook output format (per-case + marketplace_image variable). |
.azuredevops/azure-pipelines.yml |
Added Azure DevOps PR pipeline implementing AI selection, Azure test execution, publishing, and GitHub PR commenting. |
Comment on lines
+1704
to
+1707
| if has_only_comment_changes(diff): | ||
| print("Only comment changes detected. No test cases will be selected.") | ||
| write_outputs([], [], DEFAULT_MARKETPLACE_IMAGE, output_file) | ||
| return |
Comment on lines
1709
to
+1713
| if not has_relevant_code_changes(changed_files): | ||
| print( | ||
| "No framework or testsuite Python changes detected. " | ||
| "Skipping test selection." | ||
| "Falling back to smoke_test." | ||
| ) |
Comment on lines
1722
to
+1725
| if has_only_unsupported_platform_changes(changed_files): | ||
| print( | ||
| "Detected non-Azure platform-only changes. " | ||
| "Skipping case selection for this workflow." | ||
| "Detected non-Azure platform-only changes. " "Falling back to smoke_test." | ||
| ) |
…I settings; drop hardcoded lisa_log_agent group and lisa_connection
Comment on lines
+152
to
+170
| def has_only_comment_changes(diff: str) -> bool: | ||
| """Return True if every added/removed line is a Python comment or blank. | ||
|
|
||
| Used to skip case selection for comment-only edits. Requires at least one | ||
| non-blank comment line and no changed code lines. | ||
| """ | ||
| saw_comment = False | ||
| for line in diff.splitlines(): | ||
| if line.startswith("+++") or line.startswith("---"): | ||
| continue | ||
| if not (line.startswith("+") or line.startswith("-")): | ||
| continue | ||
| stripped = line[1:].strip() | ||
| if not stripped: | ||
| continue | ||
| if not stripped.startswith("#"): | ||
| return False | ||
| saw_comment = True | ||
| return saw_comment |
Comment on lines
1709
to
+1713
| if not has_relevant_code_changes(changed_files): | ||
| print( | ||
| "No framework or testsuite Python changes detected. " | ||
| "Skipping test selection." | ||
| "Falling back to smoke_test." | ||
| ) |
Comment on lines
1722
to
+1726
| if has_only_unsupported_platform_changes(changed_files): | ||
| print( | ||
| "Detected non-Azure platform-only changes. " | ||
| "Skipping case selection for this workflow." | ||
| "Detected non-Azure platform-only changes. " "Falling back to smoke_test." | ||
| ) | ||
| write_outputs( |
| -v "location:$(location)" \ | ||
| -v "marketplace_image:$(MARKETPLACE_IMAGE)" \ | ||
| -v "test_case_name:$CASE_PATTERN" \ | ||
| -v "enable_tests:False" \ |
Comment on lines
+3
to
+7
| pr: | ||
| autoCancel: true | ||
| branches: | ||
| include: | ||
| - main |
Comment on lines
+60
to
+65
| def fetch(): | ||
| req = urllib.request.Request(api) | ||
| req.add_header("Accept", "application/vnd.github+json") | ||
| req.add_header("User-Agent", "ado-lisa-pipeline") | ||
| with urllib.request.urlopen(req) as resp: | ||
| return json.load(resp).get("check_runs", []) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Use azure pipeline instead.
Type of Change
Checklist
Test Validation
Key Test Cases:
Impacted LISA Features:
Tested Azure Marketplace Images:
Test Results