Enhance CI pipeline and unit tests for AI failure analysis#2029
Open
chensuyue wants to merge 17 commits into
Open
Enhance CI pipeline and unit tests for AI failure analysis#2029chensuyue wants to merge 17 commits into
chensuyue wants to merge 17 commits into
Conversation
- Updated run_ut.sh to support additional command-line arguments for failure context and failed test cases. - Implemented functions to handle rerunning failed test cases based on their categories (base, inc, llmc). - Improved environment setup functions for INC and LLMC unit tests. - Modified the unit test execution logic to accommodate reruns of failed tests. - Enhanced the Azure Pipelines template (ut-template.yml) to include parameters for failure log context and failed test cases. - Added a new AI analysis stage in the unit-test.yml pipeline to handle failure context merging and analysis. - Introduced new scripts for AI failure analysis, including analyze_and_suggest.py, merge_failure_context.py, and post_pr_comment.py. - Created a new template (ai-analysis-template.yml) for AI analysis steps in the CI pipeline. - Implemented logic to post analysis results as comments on pull requests. Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
for more information, see https://pre-commit.ci
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
…ailure log directory handling Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
for more information, see https://pre-commit.ci
…related methods Signed-off-by: chensuyue <suyue.chen@intel.com>
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new deterministic CI failure classification system, adding two new scripts:
classify.pyandevidence_collectors.py. The system analyzes CI failure logs, aggregates forensic signals, and classifies failures into actionable categories to improve automated triage and routing. The implementation includes robust evidence collection, group-level classification logic, and Azure DevOps pipeline integration.The most important changes are:
New CI Failure Classification Pipeline
.azure-pipelines/scripts/ai_failure_analysis/classify.py, which classifies CI failures into one of six categories (Known Issue, Environment, Dependency, Flaky Test, Code Regression, Other) using deterministic evidence and known-issue matching, and emits pipeline variables to control downstream handling.Deterministic Evidence Collection
Introduced
.azure-pipelines/scripts/ai_failure_analysis/evidence_collectors.py, which collects structured forensic signals from CI logs, including environment issues, PR relevance, dependency changes, and flaky test signals, to support accurate classification.Implemented robust environment signal detection using regex patterns to identify common infrastructure issues (e.g., network timeouts, disk full, out-of-memory) and aggregate supporting evidence.
Developed logic to correlate failed tests with PR-changed files, providing a strong signal for distinguishing code regressions from other failure types.
Pipeline Integration and Output
These changes lay the foundation for automated, evidence-driven CI failure triage and will improve classification accuracy and routing for CI failures.