Skip to content

[FIX]: Evaluate probe verdicts over final traces - #149

Draft
Spencer Schoenberg (spencrr) wants to merge 3 commits into
microsoft:mainfrom
spencrr:dev/spencrr/trace-probe-cadence
Draft

[FIX]: Evaluate probe verdicts over final traces#149
Spencer Schoenberg (spencrr) wants to merge 3 commits into
microsoft:mainfrom
spencrr:dev/spencrr/trace-probe-cadence

Conversation

@spencrr

@spencrr Spencer Schoenberg (spencrr) commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description

Changes behavioral probes to evaluate the verdict once over the completed trace. Probes no longer stop on a detected verdict by default; callers can provide stop_when when an online condition should define an earlier terminal trace.

This fixes sequence and prior-condition probes that cannot be resolved correctly from partial prefixes. Terminal verdict evidence is stored on Result.terminal_evaluation, online stop evidence remains on turns with EvaluationPurpose.STOP_CHECK, and trace completion is recorded with TraceEndReason. Terminal evaluation stays inside the active session.

This is the first layer that changes verdict cadence, so it bumps the private xdist envelope from v2 to v3. Controllers reject v2 envelopes rather than combining prefix-derived and terminal-trace statuses.

Depends on #148. Because the branches live on a fork, this PR temporarily includes lower-layer diffs and targets main; those diffs disappear as dependencies merge.

Breaking changes

Behavioral change: Result.eval_results remains a turn-only compatibility view and is normally empty for probes without stop_when; use Result.terminal_evaluation for terminal verdict evidence. Adaptive drivers without an online stop receive no evaluator feedback and may run to max_turns.

Checklist

  • pre-commit run --all-files passes
  • Tests cover one-call verdict cadence, sequence and prior-condition probes, scopes, stopping, feedback, budgets, session ordering, summaries, and error results
  • Documentation updated

Validation after reslicing: the full suite passes with 1,178 tests and 5 skips; all pre-commit checks pass.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant