[CRE] Mixed-env topology + automatic non-determinism detection - #23290
Merged
Conversation
product-security-plaid-production
Bot
requested review from
Tofel,
patrickhuie19,
tvc-robsondebraga and
vyzaldysanchez
July 31, 2026 16:53
prashantkumar1982
marked this pull request as draft
July 31, 2026 16:53
Contributor
|
✅ No conflicts with other open PRs targeting |
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
July 31, 2026 19:08
41c2e2f to
b4be2da
Compare
Contributor
|
I see you updated files related to
|
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
July 31, 2026 20:11
b4be2da to
8b25b56
Compare
Tofel
reviewed
Aug 3, 2026
Adds a "mixed-env" Local CRE topology that runs each multi-node DON with 2 nodes built from the PR image and 2 from the develop image, and fails the run if the two code versions ever disagree — surfacing non-determinism and cross-version incompatibilities before merge. - configs/mixed-env-don.toml.tmpl + render-mixed-env.sh: 2-2 image split per DON via override_mode = "each"; rendered with envsubst (gitignored output). - test-helpers/nondeterminism_scan.go + smoke/cre TestMain: scan all node logs after the suite (mixed-env only) and fail with "Non-Determinism introduced". Markers: the libocr OCR3 signature-mismatch warning plus the DON2DON request/response divergence warnings. - cre-system-tests.yaml: run mixed-env for Bucket_A/B; render step resolves PR + develop-nightly images; empty chainlink-image so per-node TOML images are used; optional mixed_env_baseline_image input; exclude TestMain from discovery. - docs/mixed-env.md: developer-facing overview. Reuses the already-built per-PR image and the nightly develop image, so no extra image builds are added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prashantkumar1982
force-pushed
the
cre/mixed-env-non-determinism
branch
from
August 3, 2026 18:20
8b25b56 to
63aada7
Compare
prashantkumar1982
marked this pull request as ready for review
August 3, 2026 18:48
Review feedback: keep cre-system-tests.yaml simple by extracting the mixed-env topology into a dedicated reusable workflow, mirroring how cre-regression-system-tests.yaml is structured. - New .github/workflows/cre-mixed-env-tests.yaml: owns the mixed-env matrix (Bucket_A, Bucket_B, EVM_Read_*), the render step, env start (empty CTF_CHAINLINK_IMAGE), and the non-determinism gate. No per-topology conditionals — every job is mixed-env, so it's simpler than the inline version. - integration-tests.yml: call it from a new run-core-cre-mixed-env-tests job, gated the same as the other CRE e2e workflows. - cre-system-tests.yaml: revert all mixed-env additions (matrix entries, render step, gate step, mixed_env_baseline_image input, chainlink-image conditional). Only the TestMain discovery exclusion remains (TestMain lives in the smoke pkg). - docs updated to point at the new workflow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tofel
reviewed
Aug 4, 2026
Tofel
reviewed
Aug 4, 2026
…r in gate Addresses review feedback on cre-mixed-env-tests.yaml: - Fail-fast image presence check: before starting the local CRE, verify both the PR and develop images exist in the registry (docker manifest inspect) and fail with an informative error if not (e.g. transient build/registry failure or the develop nightly not yet published) instead of failing deep inside env start. - Non-determinism gate now reuses the Go scanner instead of a duplicated bash grep: the marker list is centralized as helpers.NonDeterminismNeedles, and the gate runs cmd/mixed-env-nondeterminism-check (go run) which calls helpers.ScanContainersForNeedles. The smoke-suite TestMain uses the same list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alse positives Per-PR mixed-env now uses chainlink-integration-tests:<base-sha> — the exact develop commit the PR image is built on top of (the CI auto-merge base) — as the develop side of the 2-2 split, instead of the latest develop nightly. Because the PR image = that develop commit + the PR's changes, the two halves of each DON now differ ONLY by the PR's changes, so develop's own churn between the PR's branch point and HEAD no longer surfaces as a "Non-Determinism introduced" false positive. The base-sha image is the same build/repo as the PR image and is already published by the develop-push build, so no extra image build is added. Falls back to the develop nightly for non-PR runs or if the base-sha image has aged out of ECR (the existing image-presence check then guards the fallback). - cre-mixed-env-tests.yaml: new baseline_ref input; render step prefers chainlink-integration-tests:<baseline_ref> (existence-checked) over the nightly. - integration-tests.yml: pass baseline_ref: github.event.pull_request.base.sha. - docs/mixed-env.md: document the base-commit baseline and the nightly fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The baseline is now the exact commit of whatever branch the PR merges into (chainlink-integration-tests:<base.sha>), which already generalizes correctly to release branches — a release PR is compared against its release line, not develop. Fix the fallback for when that base image isn't published (release-branch commits aren't force-built like develop, so their branch-tip image is often absent): - develop base -> fall back to the develop nightly (a valid stand-in for develop). - non-develop base (e.g. release/2.57.1) -> SKIP mixed-env with a warning, instead of comparing a release-based PR against the develop nightly (which would flood the run with false positives from the entire release<->develop divergence). - cre-mixed-env-tests.yaml: split baseline resolution into a Resolve-baseline step (outputs image + skip); new base_ref_name input gates the fallback; all env-start, test, and gate steps are guarded on skip != 'true'. - integration-tests.yml: pass base_ref_name: github.event.pull_request.base.ref. - docs/mixed-env.md: document release-branch baseline + skip behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…escape hatch Make the mixed-env non-determinism check block merge, and give it a self-service release valve in the same change. Required: add run-core-cre-mixed-env-tests to the ETH Smoke Tests merge gate (check-e2e-test-results) alongside the other e2e suites, so a "Non-Determinism introduced" failure now fails the required gate. The gate treats a skipped job as a non-blocking warning, so non-CRE PRs and release-branch PRs with no baseline (which skip internally) don't block. Escape hatch: new `skip-mixed-env` PR label, mirroring skip-e2e-regression. The labels job detects it; the setup job turns it into a run-mixed-env output; the mixed-env job gates its `if` on it. Applying the label skips the job -> the gate sees a warning -> merge unblocked, no admin needed. Like the regression tests, mixed-env no longer runs in the merge queue, so the bypass carries through. docs/mixed-env.md: new "Required check & emergency bypass" section documenting the label, admin/ruleset bypass, and de-wiring, plus a link from the escape-hatch note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tofel
reviewed
Aug 5, 2026
…the gate) Addresses review feedback (Tofel/kalverra, r3720892981): mixed-env runs a pre-compiled cre-smoke.test binary, so restoring the Go build cache in Set up Go just wastes ~30-60s/run. Set no_cache: true. Mixed-env differs from cre-regression-system-tests (the cited precedent) in one way: the non-determinism gate used `go run ./cmd/mixed-env-nondeterminism-check`, which would compile the CTF framework cold once the cache is gone. Avoid that by having the gate reuse the same pre-compiled binary — TestMain runs the container scan after m.Run() and exits non-zero on any marker, so `cre-smoke.test -test.run '^$'` triggers the scan with zero tests and no Go build. Falls back to `go run` only if the binary is missing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… into cre/mixed-env-non-determinism
…ly release bases) Previously any non-develop base with no published base-sha image was skipped, which silently disabled mixed-env for PRs stacked on another feature branch (e.g. the nightly PR stacked on this one) — not just release branches. Narrow the skip to release/* bases only (where a develop comparison would be the wrong branch and flood false positives). Every other base with no base-sha image — develop, a non-PR dispatch/schedule run, or a stacked feature branch that ultimately targets develop — now falls back to the develop nightly, so stacked PRs get coverage. For a stacked PR this compares the whole stack (parent branch + this PR) against develop, so a runtime-changing parent can surface a marker; that's a real potential non-determinism (caught when the parent merges too), and skip-mixed-env unblocks it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Tofel
approved these changes
Aug 5, 2026
jmank88
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[CRE] Mixed-env topology + automatic non-determinism detection
What & why
Adds a
mixed-envLocal CRE topology that runs each multi-node DON with 2 nodes on the PR image and 2 on the develop image, and fails the run if the two code versions disagree. This surfaces non-determinism and cross-version incompatibilities (e.g. things that would break a rolling upgrade) in CI, before merge — instead of after.A DON only works if its nodes compute identical results. If a change makes nodes produce different consensus reports or different cross-DON request/response payloads, the DON can't agree. Putting old (develop) and new (PR) nodes in the same DON forces any such divergence to show up as a failed check.
What it catches
While the normal smoke tests run, every node's logs are scanned for markers that mean two nodes disagreed. A single hit fails the run with
Non-Determinism introduced:This is commonly caused by non-determinism(libocr)received messages with the same id and different payloadsreceived multiple unique responses for the same request/response quorum unreachableThese lines are silent when all nodes run the same code.
Changes
configs/mixed-env-don.toml.tmpl(workflow + capabilities DONs atoverride_mode = "each", 2 PR + 2 develop; bootstrap/gateway on the baseline),configs/render-mixed-env.sh(envsubst renderer),.gitignorefor the rendered file.test-helpers/nondeterminism_scan.go(ScanContainersForNeedles) +smoke/cre/nondeterminism_check_test.go(TestMainscans node logs after the suite, mixed-env only).cre-system-tests.yaml): mixed-env added toTest_CRE_V2_Suite_Bucket_Aand_BviaPER_TEST_TOPOLOGIES_JSON;TestMainexcluded from test discovery; render step resolves PR + develop-nightly images; emptychainlink-imagefor mixed-env so per-node TOML images win; optionalmixed_env_baseline_imageinput.core/scripts/cre/environment/docs/mixed-env.md.Reuses the already-built per-PR image and the nightly develop image — no extra image builds.
Design notes
CTF_CHAINLINK_IMAGEis required for mixed-env (a non-empty value forces one image on all nodes) — handled in the workflow.