Add agentic workflow for PerfStar MT regressions

[AlesProkop]

Related to #13743

Context

Adds a scheduled GitHub Agentic Workflow that detects possible PerfStar multithreaded (/mt) build-time regressions across Gold and Hosted data, investigates all candidates as one batch using evidence from the exact current and last-healthy runs plus matching scheduled binlog diagnostics, and proposes a single draft fix only when the evidence supports every actionable candidate. The OIDC exchange is based on #13743 by Jan Krivanek, which proved secretless GitHub Actions access to Microsoft Entra-protected services.

Changes Made

• Added Get-MtBuildTimeRegressions.kql to compare the latest paired production MT/non-MT run with a 21-day per-build-median baseline across Gold/Hosted and Windows/Linux.
• Added a last-healthy paired run to each candidate so the workflow can compare exact PerfStar and MSBuild revisions.
• Added Invoke-MtBuildTimeRegressionScan.ps1 to execute the Kusto query and produce bounded JSON and Markdown statistical evidence.
• Added Add-MtBuildTimeRegressionEvidence.ps1 to resolve component MSBuild source SHAs, download candidate-specific Hosted and Gold artifacts, extract allowlisted metrics and safe timing/completion lines, and delete all raw downloads.
• Added Add-MtBuildTimeDiagnosticEvidence.ps1 to match scheduled diagnostics pipeline 28394 by exact MSBuild source SHA and query task_wallclock(), target_wallclock(), eval_pass(), and task_inventory for bounded MT/non-MT deltas and migrated-task controls.
• Added mt-build-regression.agent.md with an OIDC-authenticated deterministic scan job and a secret-free Copilot investigation job.
• Restricted safe outputs to at most one aggregate issue and one draft PR per run, with PR edits limited to src/**.
• Reused the existing copilot-pat-pool environment so Copilot credentials remain isolated from Kusto and Azure DevOps access.
• Added mt-build-regression-agent.md with architecture, attribution, setup, detector thresholds, evidence handling, and current limitations.
• Generated mt-build-regression.agent.lock.yml with gh-aw v0.82.9.

Testing

• Compiled and strictly validated mt-build-regression.agent.md with gh aw v0.82.9.
• Ran actionlint through gh aw lint; all seven agentic workflows reported zero issues.
• Executed the detector against live perfstar-dev data; it returned five candidates spanning Gold and Hosted.
• Executed the full actual-run evidence path; all current/healthy PerfStar runs, MSBuild builds, source SHAs, and candidate artifacts resolved successfully.
• Matched diagnostics runs from definition 28394 by exact source SHA and produced current diagnostic task/target/evaluation evidence for all five live candidates, plus healthy-source evidence where available.
• Verified red diagnostic runs are accepted only when the affected scenario/OS has paired MT and non-MT task data.
• Verified an injected unsafe scenario identifier degrades to unavailable diagnostic evidence without failing the workflow.
• Verified derived evidence is bounded to approximately 265 KB and no raw artifact/binlog directory remains after processing.
• Executed the empty-result path; it emitted has_regressions=false, regression_count=0, and a valid zero-candidate report.
• Parsed the generated YAML and all PowerShell scripts successfully.
• Did not run the MSBuild product build because the change only adds workflow, query, script, and documentation files.

Notes

• The workflow is intentionally draft-only and does not queue experimental MSBuild or PerfStar validation runs yet.
• Before the first scheduled run, grant msbuild-azdo-reader Database Viewer access to perfstar-dev and Azure DevOps View builds access to definitions 25429, 28338, 28394, and 9434. Queue permission is not required.
• Raw Azure DevOps logs, artifacts, and binlogs are never uploaded to the public repository. Candidate artifacts are processed in an ephemeral trusted job and deleted; scheduled binlogs are consumed through already-ingested Kusto task/target/evaluation data. Only bounded derived JSON and Markdown evidence is passed to the agent.
• Scheduled binlog evidence is direct for Hosted candidates and explicitly labeled as Hosted corroboration for Gold candidates. Task/target totals can include nested work, so migrated task controls are included as the contention/noise floor.
• Security review: the workflow newly references existing secrets AZDO_READER_CLIENT_ID, AZDO_READER_TENANT_ID, and COPILOT_PAT_0 through COPILOT_PAT_9; no new secret values are introduced. Azure/Kusto/Azure DevOps credentials are restricted to mt_regression_scan, while the agent job receives only the generated evidence artifact. Generated actions are SHA-pinned by gh-aw; no redirect changes were added.

[AlesProkop]

Review — PR #14406 · 5 files, +2534/−0 · Deep

Methodology: portable core + dotnet/msbuild expert-reviewer · 13/13 dims (no product C# changed).

Verdict: 🟡 COMMENT — address findings — 0 blocking, 0 major, 2 moderate, 4 nit. Secret-isolation verifies end-to-end: Kusto/AzDO tokens stay job-local in mt_regression_scan, the agent gets only the evidence artifact, actions/containers are SHA-pinned.

Needs attention

• 🟡 Design — dedup marker is github.run_id (unique per run) → no real cross-run dedup; duplicate tracking issues possible. mt-build-regression.agent.md:266. Fix: content-derived key (hash of the sorted Backend/Os/ScenarioPair set) or rename it. (addressed in a later commit)
• 🟡 Deployment — mt_regression_scan has no environment:, so AZDO_READER_* must be repo/org secrets and the federated credential must trust repo:dotnet/msbuild:ref:refs/heads/main, else the first scheduled scan fails. Add to "Required setup".
• ⚪ Design — confirm allowed-files: ["src/**"] matches nested paths (reconcile with the flaky-test-fixer allowlist caveat); excluded-files: .github/** is redundant under it.
• ⚪ Docs — thresholds (21d/5%/250ms/4 runs) duplicated across the .kql, PS metadata, prompt, and docs; make the .kql the source of truth or add a sync comment.
• ⚪ Docs — frontmatter lacks the rationale comments the sibling workflows carry (network/timeout/tools).
• ⚪ Tests — no offline smoke test for the KQL/PS detector (matches sibling norms).

Clean (9): Correctness, Security, Concurrency, Backward-compat, Error handling, Performance, Portability, Observability, Simplicity — evaluated, no issues.

_{🤖 ultimate-reviewer (Deep). Read-only; nothing built or run.}

[AlesProkop]

Review — PR #14406 · re-review at 50d5e04 · 7 files, +3875/−0 · Deep

Methodology: portable core + dotnet/msbuild expert-reviewer · 13/13 dims (no product C# changed).

Verdict: 🟡 COMMENT — address findings — 0 blocking, 0 major, 1 moderate, 4 nit.

Since the earlier pass (+2534), the two Add-MtBuildTime*Evidence.ps1 scripts (~1,142 lines) are new — focus here. Prior findings look resolved (content-hash candidateSetKey, threshold-sync comment, frontmatter rationale, documented setup). Detector math, AzDO SHA/run-resolution, and migration_state (keys on marker_attribute, not the inherited marker_interface) re-verified correct. Secret-isolation re-verifies: tokens job-local + --exclude-env'd from the agent, agent network excludes AzDO/Kusto, safe-outputs bounded to max 1 draft issue/PR under src/** + protected files.

Needs attention

• 🟡 Design — set-hash dedup orphans prior tracking issues when the candidate set changes membership, and the agent has no close/comment path (safe-outputs = create/noop only) → issues accumulate. agent.md:283. Fix: dedup per Backend/Os/ScenarioPair, let the agent supersede stale issues, or document the behavior.
• ⚪ Perf — 100 MB artifact guard fires after the full download. Add-MtBuildTimeRegressionEvidence.ps1:189. Fix: check size from artifact metadata first.
• ⚪ Correctness — Add-MetricTable doesn't escape | in cell values, unlike Add-DeltaTable (Add-MtBuildTimeDiagnosticEvidence.ps1:466) — latent table break. Add-MtBuildTimeRegressionEvidence.ps1:450.
• ⚪ Naming — $matches shadows the automatic $Matches. Add-MtBuildTimeDiagnosticEvidence.ps1:175. Rename → $runsBySource.
• ⚪ Correctness — last-healthy diagnostic evidence is skipped for SHAs older than the 24 most-recent diagnostic runs. Add-MtBuildTimeDiagnosticEvidence.ps1:21. Tie MaximumRunsToInspect to the 21-day lookback.

Clean (9): Security, Concurrency, Backward-compat, Error handling, Test coverage, Portability, Observability, Documentation, Change hygiene.

_{🤖 ultimate-reviewer (Deep) at 50d5e04. Read-only; scripts not run (no creds).}

[JanKrivanek]

@copilot study and understand the workflow added here and then add a flow chart diagram to the doc to give reader quick overview of the datasources and sequences of processing

[Copilot]

@copilot study and understand the workflow added here and then add a flow chart diagram to the doc to give reader quick overview of the datasources and sequences of processing

Added a "Data flow at a glance" Mermaid flow chart to .github/docs/mt-build-regression-agent.md (commit will follow). It shows the three external data sources (Kusto perfstar-dev and the four Azure DevOps definitions 25429/28338/9434/28394), the trusted mt_regression_scan job's OIDC-authenticated 4-step processing pipeline, the derived-evidence artifact boundary, and the secret-free agent's investigate → issue/draft-PR/noop decision. I validated the diagram parses cleanly with the Mermaid parser.