-
Notifications
You must be signed in to change notification settings - Fork 150
fix(runner): conclude benign ErrNoValidDutiesToExecute as not_required, not failed #2988
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: stage
Are you sure you want to change the base?
Changes from 2 commits
54b0116
5cbe3bc
c92bad9
cba5938
b0ef6a9
82e1c78
e4be291
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -379,6 +379,10 @@ listener: | |
| ) | ||
|
|
||
| if totalAttestations == 0 && totalSyncCommittee == 0 { | ||
| // Benign terminal: the committee decided but this operator ended up with zero valid duties to | ||
| // sign. Conclude as not_required so the watcher doesn't report a false "stuck"; the sentinel | ||
| // still tells committee_queue to drop the message and terminate the runner. | ||
| r.markDutyNotRequired() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Finding 2 · [MINOR] Guard the consensus-phase not_required against context cancellation (shutdown now records a benign outcome) In This matters because the codebase deliberately treats cancellation as "not attempted", never as an outcome: On a rolling restart across a large fleet, this shows up as a burst of spurious
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in b0ef6a9. Added a cancelled-context regression test. |
||
| return ErrNoValidDutiesToExecute | ||
| } | ||
|
|
||
|
|
@@ -514,6 +518,9 @@ func (r *CommitteeRunner) ProcessPostConsensus(ctx context.Context, logger *zap. | |
| // are tagged recoverableReconstructError and must not be recorded as failed. | ||
| // Shutdown (context cancellation) needs no special-casing — markDutyFailed drops a context.Canceled | ||
| // reason, so a submission aborted by shutdown isn't recorded as a failure. | ||
| // The benign no-beacon-objects sentinel (ErrNoValidDutiesToExecute) pre-concludes the duty as | ||
| // not_required before returning, which makes this deferred markDutyFailed a no-op (concludeDuty | ||
| // is idempotent) — it must not be recorded as failed either. | ||
| defer func() { | ||
| if err != nil && !isRecoverableReconstructError(err) { | ||
| r.markDutyFailed(err) | ||
|
|
@@ -529,6 +536,12 @@ func (r *CommitteeRunner) ProcessPostConsensus(ctx context.Context, logger *zap. | |
| return fmt.Errorf("could not get expected post consensus roots and beacon objects: %w", err) | ||
| } | ||
| if len(beaconObjects) == 0 { | ||
| // Benign terminal: the committee reached consensus but this operator has no beacon objects to | ||
| // submit (e.g. divergent validator sets across the committee's operators). Conclude as | ||
| // not_required — not failed — before returning the sentinel; concludeDuty is idempotent, so | ||
| // the deferred markDutyFailed becomes a no-op. The sentinel still tells committee_queue to | ||
| // drop the message and terminate the runner. | ||
| r.markDutyNotRequired() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When every validator is skipped because duty validation, object construction, domain-data retrieval, or signing-root computation fails, this branch records the empty result as successful Knowledge Base Used: Protocol v2 Duty Runners and QBFT Consensus Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in c92bad9.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Building on the existing P1 by Greptile from above (empty objects masking construction failures) rather than repeating it — two clarifications that should help decide the fix: A template already exists in the sibling runner. Severity is likely below P1 in practice. Scope note: only this post-consensus branch is affected. The consensus-phase sibling change at L381-L387 is unambiguous —
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in c92bad9. A guard-invalid-only empty map stays not_required. Added a regression test with DomainData failing after consensus to pin the all-failure path.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Finding 3 · [MINOR] New not_required terminals skip EndDutyFlow/recordTotalDutyDuration and leave no operator-visible completion line Every pre-existing Two consequences. First, metric skew:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 82e1c78. The aggregator-committee site went back to failed in cba5938, so operator visibility there comes from the outcome watcher's warn. |
||
| return ErrNoValidDutiesToExecute | ||
| } | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Finding 1 · [IMPORTANT] Don't reclassify AggregatorCommitteeRunner's empty-objects branch as benign — there it is an invariant violation, not a per-operator no-op
CommitteeRunner's "nothing to submit" rationale doesn't transfer to AggregatorCommitteeRunner. CommitteeRunner builds beaconObjects from LOCAL state (local DutyGuard, local construction), so divergent validator sets across operators genuinely produce an empty map on one operator only — benign. AggregatorCommitteeRunner instead builds beaconObjects purely from the DECIDED consensus data, byte-identical for every operator, surfacing every error rather than skipping any. So
len(beaconObjects)==0here can only happen if the decided value had zero Aggregators and zero Contributors — a stateAggregatorCommitteeConsensusData.Validate()explicitly rejects, andvalidateDecidedConsensusDataenforces before storing it as decided.Reaching this branch therefore means the decided-value check was bypassed or regressed, QBFT decided on a value that never passed
CheckValue, or there's a spec-level encode/decode mismatch — a consensus-integrity signal, not routine idle time. Before this PR it surfaced as a loud failure; after, it's a silentnot_required+ debug log. The node goes quiet on a state that should be structurally impossible.Telling corroboration: the PR's own regression test for this branch can't reach it through the runner's normal flow — it hand-installs
State.DecidedValue, bypassingCheckValue/Validate()to force the condition.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in cba5938.
You're right, this one doesn't transfer: reverted the branch to failed with an explicit invariant-violation error (the sentinel stays wrapped so the queue still drops the message and terminates the runner), and the regression test now pins the failed outcome and documents why it has to hand-install the decided value.