Stop gating nightly-perf on fixture_build_ms wall-clock noise - #355
Merged
Merged
Conversation
fixture_build_ms times constructing the synthetic test fixture itself, not a real app code path, and isn't one of ADR-019's budgeted rows. It was the sole cause of 5 of the last 13 nightly-perf.yml failures (most recently 2026-08-24: +289%/+113% swings on large/stress in one run) while every genuinely budgeted metric in those same runs stayed within threshold. Still measured and printed for visibility, just no longer gated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
nightly-perf.yml has failed 5 of its last 13 runs. All 5 traced to the same cause:
fixture_build_ms(time to construct the synthetic test fixture itself viaworkload.build()) swinging wildly on the shared Windows runner — most recently (2026-08-24) +289% onlargeand +113% onstressin a single run, both well aboveMIN_TIMING_MS_FOR_PCT_CHECKso not caught by the existing noise filter. In that same run, every metric that's actually part of ADR-019's budgeted rows (payload_build_ms,serialize_ms,publish_total_ms, wire sizes) stayed within threshold.fixture_build_msisn't one of ADR-019 §2's budgeted rows — it measures test-harness overhead (building the synthetic scene fixture), not a real app code path, so gating on it was checking noise, not signal.Change
backend/tests/perf/check_baseline.py: movedfixture_build_msout of the gatedTIMING_METRICStuple into a newREPORTED_ONLY_METRICStuple — still measured and printed every run for visibility, just no longer able to fail the job.Test plan
python -m backend.tests.perf.check_baselinelocally —fixture_build_msno longer appears among reported failures,payload_build_ms/serialize_ms/publish_total_mscontinue to be gated normally.gh run list --workflow=nightly-perf.ymlthat this was the sole failure cause in the 5 recent failing runs before making the change.Note: that same local run also surfaced real, deterministic wire-size drift (
full_snapshot_kib/single_node_payload_kibup 13–19% across all workloads vs. the 2026-08-11 baseline) — unrelated to this fix and out of scope here; flagged separately, not touched by this PR.