consensus: after a block-sync handover while still behind, a validator proposes at historical heights and wedges itself permanently

[QuantumExplorer]

Expected Behavior

A node that hands over from block sync to consensus while it is still far behind the network should keep catching up. It should not act as a proposer at heights the network committed long ago, and nothing it does while behind should leave it unable to start.

Current Behavior

A mainnet evonode re-syncing from scratch stalled in block sync at height 24174 (5 peers, all seeds, none serving blocks), handed over to consensus, and as a member of that era's quorum was selected proposer at h=24175 r=1. It built a block for a 2024 height from its present-day Core state (chain lock at core height 2,523,755; the real block 24175 carries 2,157,967), prevoted it, and then — as soon as a peer started catching it up over consensus gossip — panicked, and kept panicking on every restart until the Tenderdash data (WAL + block store) was discarded. A second full re-sync with ~260 peers (block sync never stalled) went through.

Drive log at the moment of the stall:

2026-08-18T06:16:15.362578Z  PrepareProposal  h=24175 r=1  received ABCI request
2026-08-18T06:16:15.362708Z  PrepareProposal  h=24175 r=1  JSON-RPC request: getbestchainlock []
2026-08-18T06:16:15.364823Z  PrepareProposal  h=24175 r=1  propose chain lock update to height 2523755 at block 24175

Mainnet facts (platform explorer): block 24175 — app hash 402A8119…, core-locked height 2,157,967, proposer EE22F5BC…; block 24176 — proposer F1616A59…. Proposers at 24171..24180 are strictly increasing proTxHashes (round-robin, one step per height).

Mechanism

1. Handover while behind. In 1.6.0 WaitForSync returns after a 60 s stall regardless of distance to the tip (v1.6.0 synchronizer.go#L290). 1.7.0 (fix(blocksync): survive transient peer failures and stalls #1396) only hands over when no peer can serve the next block or after maxSyncStall (10 min) (synchronizer.go#L414-L470) — it still hands over, and SwitchToConsensus (reactor.go#L446) only uses the caughtUp flag to decide whether to replay the WAL. Nothing tells consensus "you are behind".

2. Proposing at a historical height. EnterProposeAction proposes whenever isProposer (state_enter_propose.go#L53-L114). The validator set at 24175 is the October-2024 quorum; a long-lived evonode is in it. With the consensus-version-0 selector, proposer(H, r) is proposer(H, 0) advanced by r, not persisted (height_proposer.go#L25, #L182-L195) — so proposer(H, r1) == proposer(H+1, r0). The node that proposed at (24175, r1) is therefore F1616A59…, the genuine round-0 proposer of 24176. That guarantees what follows.

3. The block it builds. The ABCI app (Drive) answers PrepareProposal with whatever Core's current best chain lock is, if it is above the last committed core height (prepare_proposal.rs#L85-L127); Tenderdash has no "chain lock is absurdly far ahead of this block's position" check either (validateCoreChainLock only requires it to be higher). The proposal is a 2024 block carrying a 2026 chain lock; the node prevotes it and rolls rounds alone.

4. Collision at the next height. A real peer eventually arrives. Once the stale round-1 proposal has expired, the genuine commit for 24175 (round 0) is accepted and applied (different round → the app re-executes → fine). The node then enters 24176 round 0 as proposer and calls PrepareProposal immediately. The genuine 24176 commit — also round 0, proposed in 2024 by this very node — arrives after that round expires → ApplyCommit → mustEnsureProcess → ProcessProposal(genuine 24176, round 0) (state_apply_commit.go#L49-L54). The app already holds an execution context for (24176, round 0) — its own block — and (until the Drive fix referenced below) either answered with the cached app hash of its own block (process_proposal.rs#L43 → mustValidate panics with wrong Block.Header.AppHash at state height 24175, block 24176. Expected <ours>, got <genuine> — execution.go#L439, block_executor.go#L111-L114), or refused with an ABCI error (process_proposal.rs#L74-L81 → mustEnsureProcess panics — block_executor.go#L77-L80).

5. Why restarts do not help.

   • catchupReplay re-dispatches the node's own proposal and block parts first (→ ProcessProposal rebuilds the stale round-0 context in the app), then the peer's commit and the genuine block parts → the same panic, deterministically, on every start (replay.go#L105, #L89). This survives app restarts and app upgrades; with 1.6.0 the 60 s block-sync window before the handover is usually too short for sparse peers, so the WAL is replayed every time.
   • If only Tenderdash restarts and block sync does get blocks, the applier saves the block before ApplyBlock (applier.go#L84-L91); the app (still holding the stale context) fails it, the panic leaves the store one ahead, and every later start fails in the handshake (syncStateAt → error on replay, replayer.go#L160-L178).

   Only discarding the WAL / block store gets the node out. While the stale proposal is live the node also rejects the real commit and (1.7.0) evicts the peer that sent it — filed separately as consensus: verifyCommit checks a peer's commit against our own proposal's BlockID — a lagging node rejects the real commit and (1.7.0) evicts the honest peer #1414.

The app-side half — answering a different block at the same height/round from a stale context — is a Drive defect, fixed in dashpay/platform#4462 (process_proposal now re-executes a different block instead of serving the cached hash or erroring; tests reproduce both behaviours). With that fix the collision becomes harmless: the genuine block executes and commits, and the node crawls forward over consensus gossip (or block-syncs on the next restart). But Tenderdash should not be putting a validator in that position in the first place, and on its own the Tenderdash side still turns the collision into a restart-proof panic loop.

Possible Solution

1. Do not propose after a handover that was not caughtUp. SwitchToConsensus already receives the flag. Suppress EnterPropose's proposal creation (voting can stay) while the node is catching up — e.g. until it has committed a block through consensus and no connected peer reports a height above its own, or until maxPeerHeight <= LastBlockHeight. Whatever signal is used has to be robust against a single peer advertising a bogus height to silence a validator; tying it to the block-sync handover (a state the node only enters while provably behind) keeps that window narrow.
2. Prefer staying in block sync while behind. The 10-minute stopStalledTooLong backstop hands over a node that is thousands of blocks behind; consensus catch-up from there is one block per gossip cycle (days), and every height where the node was the genuine round-0 proposer is a collision opportunity. Handing over only when maxPeerHeight <= height (and keeping the backstop only for the case where no peer reports a higher height) would avoid the whole class.
3. Defensive: do not SaveBlock in the block-sync applier before ApplyBlock succeeds (or make the handshake re-process with a fresh app context), so that an app-side failure does not leave the store ahead of the app.
4. Optional / app-side: the app could refuse to propose a chain-lock update that is absurdly far ahead of the last committed core height / block time — but a legitimate restart after a long halt also needs big jumps, so this is a policy decision rather than the fix.

Steps to Reproduce (for bugs)

1. Take a validator whose proTxHash is in historical quorums (i.e. one that has been a member for a long time) and re-sync it from genesis with peers that do not serve blocks (e.g. seed nodes only), so block sync stalls and hands over to consensus at some height H far below the tip.
2. Wait for the node to become proposer at H (it will, within at most one quorum's worth of rounds); observe PrepareProposal with a chain-lock update to the current Core tip.
3. Give it a peer that serves blocks. Once the stale round expires, H is caught up; at H+1 (or the next height where the node was the genuine round-0 proposer) it proposes again and the genuine round-0 commit arrives: panic wrong Block.Header.AppHash … (or ProcessProposal abci method: … received a process proposal request twice with different hash with an app that refuses the second block).
4. Restart: the WAL replay reproduces the same panic on every start (or, after a Tenderdash-only restart, error on replay in the handshake). Only a reset recovers.

Context

Operators see an app-hash mismatch that looks like a Drive/state problem ("did you reset Tendermint without resetting your application's data?") and reset everything. The actual cause — five seed-only peers at the moment block sync timed out — is invisible in the error. Any long-lived evonode re-syncing from scratch with a thin peer set can hit this.

Your Environment

• Version used: observed on 1.6.0 (platform 4.1.0); 1.7.0 changes when the handover happens but not what follows
• Operating System and version: Linux, dashmate
• ABCI app: Drive (dashpay/platform, drive-abci 4.1.x)

Have you tried the latest version: yes (1.7.0 analysed; the consensus paths involved are unchanged apart from the peer-eviction addition, see #1414)

Logs

PrepareProposal  h=24175 r=1  propose chain lock update to height 2523755 at block 24175

(mainnet, 2026-08-18 06:16 UTC; real block 24175: core height 2157967, app hash 402A8119…, proposer EE22F5BC…; block 24176 proposer F1616A59…)

[QuantumExplorer]

App-side fix (Drive re-executes a different block at the same height/round instead of serving the stale context or erroring, with tests for both behaviours): dashpay/platform#4462. With it the collision described in step 4 becomes harmless; the Tenderdash side of this issue (not proposing while behind, applier ordering) and #1414 remain.

[Claudius-Maginificent]

Follow-up investigation: different block hashes at the same ABCI height/round

Following the review of dashpay/platform#4462, we checked whether a different hash necessarily indicates a deeper Tenderdash issue rather than normal WAL/block-store replay. The hypothesis is partly confirmed: there is a deeper Tenderdash catch-up problem, and the original explanation omitted an essential state transition. Normal block-store replay does use the stored block, and WAL replay does not freshly propose. However, a node can first prepare/process its own candidate A, advance its local round, then receive the already-committed candidate B from peers. Processing B uses the commit's original round, so the application receives two different hashes at the same height/round even though local consensus advanced between calls.

I reproduced that sequence using unmodified Tenderdash v1.6.0 production code and a small test fixture. This establishes a concrete mechanism; it does not establish that the specific mainnet H24176 incident followed it. The original incident WAL, full logs, and signer configuration were not supplied.

Versions and scope

• Incident-version source: v1.6.0, 4cba43f529eee3e8a6f29b4ff61c369bb0894134, exported with git archive into an isolated snapshot.
• Reference checkout HEAD: 072f6512919158b989314901c0e0367de77571e8 (v1.7.0 merge). The reference checkout remained unchanged and read-only. The historical-proposal, round-reset, commit-round, and replay behavior discussed here exists at v1.6.0. Current HEAD has newer blocksync stall handling and peer-eviction classification.
• Only a test file was added to the disposable snapshot; no production source was changed. Tests linked existing BLS libraries read-only from the reference checkout. Go output was redirected to a separate build cache.

The upstream entry point

At v1.6.0, WaitForSync returns after 60 seconds without advancement, whether or not the node has reached peers' heights. poolRoutine then stops blocksync and switches to consensus using the last applied state. The boolean passed alongside it controls WAL skipping, not whether the node may propose while behind.

EnterProposeAction suppresses proposing during replay but has no historical-height/catch-up guard. If selected by the historical validator set, it calls the proposal creator. Proposaler.Create reuses an in-memory valid block when available; otherwise it builds a new block through CreateProposalBlock and ABCI PrepareProposal, then attempts to sign it.

The node's local block store need not contain the canonical block at this height: its committed state is still at H−1. The local WAL contains this node's observations, not a canonical archive of every historical network proposal. Peers supply the canonical candidate later: GossipCommit explicitly loads old commits for lagging peers from the sender's block store.

What the reproduction proves

The fixture constructs two candidates with the same proposer and H/R, with a transaction present only in B. The network's B has a valid threshold-signed commit made with the fixture validator set. Local A is built by the real blockExecutor.create → CreateProposalBlock → ABCI PrepareProposal path; its proposal and block are installed directly in the test, then the real ensureProcess sends A to ABCI. The fixture does not claim to reproduce historical signing or full P2P scheduling.

1. Process A at H1/R0. The application records A's hash.
2. Try B's commit while A's proposal remains installed. It is rejected with invalid commit -- wrong block ID. This is why the simplistic “own proposal, then a different same-round commit immediately replaces it” explanation is incomplete. StateData.verifyCommit verifies against the installed proposal's ID in that state.
3. Advance local consensus to R1. The test directly dispatches the real EnterNewRoundEvent, representing the timeout-driven transition. EnterNewRoundAction clears Proposal, ProposalBlock, and block parts. This transition makes no ABCI call; Drive can still hold A/R0. Automatic proposal creation is disabled in the fixture so the new round does not overwrite the context; this corresponds to a round in which the local node is not proposer or has not built a candidate.
4. Receive B's R0 commit, then its block parts. With no proposal installed, verifyCommit verifies against B's own committed ID and remembers the commit. The block-part handler explicitly tolerates a different message round and, once complete, dispatches AddCommit.
5. Process B at H1/R0 despite local R1. ApplyCommitAction explicitly takes round = commit.Round and calls mustEnsureProcess. BlockExecutor.ProcessProposal forwards that round and B's header hash to ABCI. Local round monotonicity therefore does not imply monotonic ABCI proposal rounds.

Captured request hashes, both H1/R0:

A = A54975AA1082D636C2A44C6435BC8D0A7DA7799FDF2463456C96AD61A437A25A
B = F940BB6EFF458B7048260176698B53011A3808C8F783DFD3870152FE5698E334

The stock kvstore test application initially panicked at mustEnsureProcess, because it has an explicit duplicate-ProcessProposal-per-round guard. That failing run independently demonstrated the duplicate call. With the existing WithDuplicateRequestDetection(false) application option, the same unmodified Tenderdash sequence passed, finalized B, and advanced to H2. This option changes only the test application's duplicate-call policy.

Verification:

• TestInvestigation4462DifferentHashAfterRoundReset: passed, finalized B and advanced to H2 (test runtime 0.09s).
• With the default kvstore duplicate guard: panicked at blockExecutor.mustEnsureProcess on the second request.
• Existing baseline TestStateTryAddCommitCallsProcessProposal: passed.
• Command: go test -mod=readonly ./internal/consensus -run '^TestInvestigation4462' -count=1 -v, in the isolated v1.6.0 snapshot with the project's BLS libraries configured. The full fixture is included below.
• No actual Drive node, historical mainnet state, blocksync network, signing service, or WAL file was used in this reproduction. The captured hashes are from one randomized fixture run and need not repeat.

Why WAL replay does not disprove the mechanism

The user is correct about fresh proposal creation: catchupReplay enables replay mode, which suppresses Create in EnterProposeAction. The normal stored-block replayer also loads the stored block and its commit.

But WAL replay re-dispatches recorded proposals, block parts, commits, and timeouts. If the WAL records local A, the subsequent round transition, and B's commit/parts before the fatal application call, it can replay the same two ProcessProposal calls without any new PrepareProposal. ApplyCommitAction calls process/validation before saving B and writing EndHeight, so that failure can leave the height unfinalized for the next replay. This persistent-loop explanation is supported statically but was not independently reproduced with a WAL file. A Prepare-only context resulting from a failed sign is a different case: that cache is not itself persisted in the WAL, so one cannot infer the same restart loop from that event alone.

Signer protections and a second, distinct window

PrepareProposal occurs before SignProposal. Consequently, a signer can reject a historical proposal while Drive still retains the freshly prepared, hashless context. A subsequent canonical B may then reach the pre-hash comparison branch fixed by Platform PR #4462. This does not require an actual second signature.

FilePV checks persisted height/round/step and rejects conflicting signing bytes at the same position. However, node setup can instead use Dash Core signing, whose SignProposal forwards quorum signing to Core rather than using FilePV's local state. The incident's signing behavior and any double-sign prevention therefore require its actual signer configuration and Core state. No double-sign claim is established here.

Conclusion and remaining evidence

The different hash is not evidence that Tenderdash randomly substitutes bytes when replaying one stored proposal. It can be evidence of two genuinely different proposal candidates: a locally generated stale candidate and the canonical committed block learned later. Premature blocksync handover plus historical proposal creation is an upstream trigger; Drive's height/round-only cache then turns the recovery attempt into a wrong result or fatal error. Both layers need attention. Handling the canonical block in Drive remains necessary defensive behavior even when historical proposing is prevented upstream.

For the exact reported H24175/R1 → H24176/R0 incident, the currently available issue text establishes only a reported PrepareProposal at H24175/R1 and an author's reconstructed later sequence. To distinguish the reproduced round-reset path from the hashless prepared-context path, replay behavior, or another ordering, inspect the original WAL and contiguous Drive/Tenderdash logs around H24174–24176, including local rounds, both request hashes, proposer identities, signing errors, and startup replay boundaries. Without those, the incident-specific root cause remains incomplete.

Related work: #1414 / #1415 cover commit identity validation, #1416 covers historical proposing after an early block-sync handover, and dashpay/platform#4462 covers the app-side stale context. These are distinct layers of the failure.

Full reproduction fixture for v1.6.0 (internal/consensus/investigation_4462_test.go)

This is an investigation fixture, not a proposed production change. With the project's test/build prerequisites installed, place it under internal/consensus/ in a disposable v1.6.0 checkout and run the command above. To observe the duplicate-request panic, change kvstore.NewMemoryApp(kvstore.WithDuplicateRequestDetection(false)) to kvstore.NewMemoryApp().

package consensus

import (
	"context"
	"testing"

	"github.com/dashpay/tenderdash/abci/example/kvstore"
	abci "github.com/dashpay/tenderdash/abci/types"
	"github.com/dashpay/tenderdash/dash"
	cstypes "github.com/dashpay/tenderdash/internal/consensus/types"
	"github.com/dashpay/tenderdash/internal/mempool"
	sm "github.com/dashpay/tenderdash/internal/state"
	"github.com/dashpay/tenderdash/internal/test/factory"
	"github.com/dashpay/tenderdash/types"
	"github.com/stretchr/testify/require"
)

type investigationApp struct {
	*kvstore.Application
	processed []*abci.RequestProcessProposal
}

func (a *investigationApp) ProcessProposal(ctx context.Context, req *abci.RequestProcessProposal) (*abci.ResponseProcessProposal, error) {
	a.processed = append(a.processed, req)
	return a.Application.ProcessProposal(ctx, req)
}

func TestInvestigation4462DifferentHashAfterRoundReset(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	genDoc, privVals := factory.RandGenesisDoc(4, factory.ConsensusParams())
	state, err := sm.MakeGenesisState(genDoc)
	require.NoError(t, err)
	build := func(record bool) (*State, *investigationApp) {
		base, err := kvstore.NewMemoryApp(kvstore.WithDuplicateRequestDetection(false))
		require.NoError(t, err)
		t.Cleanup(func() { _ = base.Close() })
		app := &investigationApp{Application: base}
		vals := types.TM2PB.ValidatorUpdates(state.Validators)
		_, err = app.InitChain(ctx, &abci.RequestInitChain{ValidatorSet: &vals})
		require.NoError(t, err)
		cfg := configSetup(t)
		cfg.Consensus.DontAutoPropose = true
		cs := newStateWithConfig(ctx, t, consensusLogger(t), cfg, state, privVals[0], app)
		return cs, app
	}
	local, app := build(true)
	network, _ := build(false)
	ctx = dash.ContextWithProTxHash(ctx, local.privValidator.ProTxHash)
	localState := local.GetStateData()
	networkState := network.GetStateData()
	height := localState.Height

	// The network's already-committed candidate contains a transaction the
	// resyncing node does not have. Both candidates use the same proposer and R0.
	require.NoError(t, network.txNotifier.(mempool.Mempool).CheckTx(ctx, types.Tx("key=network"), nil, mempool.TxInfo{}))
	b, err := network.blockExecutor.create(ctx, &networkState.RoundState, 0)
	require.NoError(t, err)
	bParts, err := b.MakePartSet(types.BlockPartSizeBytes)
	require.NoError(t, err)
	bID := b.BlockID(bParts)
	commit, err := factory.MakeCommit(ctx, bID, height, 0, networkState.Votes.Precommits(0), networkState.Validators, privVals)
	require.NoError(t, err)

	a, err := local.blockExecutor.create(ctx, &localState.RoundState, 0)
	require.NoError(t, err)
	aParts, err := a.MakePartSet(types.BlockPartSizeBytes)
	require.NoError(t, err)
	localState.Proposal = types.NewProposal(height, a.CoreChainLockedHeight, 0, -1, a.BlockID(aParts), a.Time)
	localState.ProposalBlock = a
	localState.ProposalBlockParts = aParts
	localState.updateRoundStep(0, cstypes.RoundStepPrevote)
	require.NoError(t, local.blockExecutor.ensureProcess(ctx, &localState.RoundState, 0))
	require.Len(t, app.processed, 1)
	require.NotEqual(t, a.Hash(), b.Hash())
	require.NotEqual(t, a.ResultsHash, b.ResultsHash)

	// While A is still installed, B's valid commit is rejected against A's ID.
	verified, err := localState.verifyCommit(commit, "peer", false)
	require.False(t, verified)
	require.ErrorContains(t, err, "block")
	t.Logf("R0 with own proposal: valid network commit rejected: %v", err)

	// A timeout can advance the local consensus round without an ABCI call.
	require.NoError(t, local.ctrl.Dispatch(ctx, &EnterNewRoundEvent{Height: height, Round: 1}, &localState))
	require.Nil(t, localState.Proposal)
	require.Nil(t, localState.ProposalBlock)
	require.Len(t, app.processed, 1)

	msg := msgInfo{Msg: &CommitMessage{commit}, PeerID: "peer"}
	ctx = msgInfoWithCtx(ctx, msg)
	require.NoError(t, local.ctrl.Dispatch(ctx, &TryAddCommitEvent{Commit: commit, PeerID: "peer"}, &localState))
	require.NotNil(t, localState.Commit)
	for i := 0; i < int(bParts.Total()); i++ {
		part := &BlockPartMessage{Height: height, Round: 0, Part: bParts.GetPart(i)}
		ctx = msgInfoWithCtx(ctx, msgInfo{Msg: part, PeerID: "peer"})
		require.NoError(t, local.ctrl.Dispatch(ctx, &AddProposalBlockPartEvent{Msg: part, PeerID: "peer"}, &localState))
	}
	require.Len(t, app.processed, 2)
	require.Equal(t, height, app.processed[0].Height)
	require.Equal(t, height, app.processed[1].Height)
	require.Equal(t, int32(0), app.processed[0].Round)
	require.Equal(t, int32(0), app.processed[1].Round)
	require.NotEqual(t, app.processed[0].Hash, app.processed[1].Hash)
	require.Equal(t, height+1, localState.Height)
	t.Logf("ABCI ProcessProposal H=%d R=0 A=%X, then local R=1, then ABCI ProcessProposal H=%d R=0 B=%X; committed B", height, app.processed[0].Hash, height, app.processed[1].Hash)
}

Investigation performed with a single Astra agent; source tracing and fixture results are separated above from incident-specific hypotheses.

_{🤖 Co-authored by Claudius the Magnificent AI Agent}