Coordinator can advertise a leader before the target tablet server admits the replica, causing NotLeaderOrFollowerException to recur for minutes after a large-table restart

[damokelis]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

The coordinator maintains two independently-updated views of replica state:

• TabletServerMetadataCache (pushed via UpdateMetadataRequest) — determines what leader info clients see via metadata().
• ReplicaManager.allReplicas (activated via NotifyLeaderAndIsrRequest, ReplicaManager#maybeCreateReplica) — determines whether a tablet server can actually serve fetchLog/putKv for a bucket.

These are sent as two separate RPCs (CoordinatorRequestBatch#sendRequestToTabletServers) with no atomicity guarantee between when the coordinator decides a new leader and when the target tablet server has actually admitted the replica.

Under normal conditions this window is milliseconds and effectively unobservable. For large tables, however, a restarting tablet server can take minutes to complete local log recovery (observed: a single 12-bucket table taking ~11 minutes) before it gets to processing NotifyLeaderAndIsrRequest, while UpdateMetadataRequest can be broadcast to clients well before that (e.g. as part of CoordinatorEventProcessor#processNewTabletServer when the server re-registers).

A client that picks up the new leader from metadata during this window sends requests to a server whose ReplicaManager still treats the bucket as NoneReplica. The server responds with NotLeaderOrFollowerException (ReplicaManager#getReplicaOrException), the client refreshes metadata (which still shows the same, still-not-actually-ready leader), and the cycle repeats — a tight retry loop reproducing hundreds to thousands of errors per minute — until the tablet server finally catches up. In our testing this recurred for anywhere from several minutes to over ten minutes after the corresponding tablet server pod already reported Running/ready at the Kubernetes level.

Symptom in client logs (verbatim):

ERROR org.apache.fluss.client.table.scanner.log.LogFetcher - Failed to fetch log from node 1 for bucket TableBucket{tableId=3, bucket=2}
org.apache.fluss.exception.NotLeaderOrFollowerException: Not leader or follower.
WARN  org.apache.fluss.client.table.scanner.log.LogFetcher - Invalid metadata error in fetch log request. Going to request metadata update.

The coordinator side is completely quiet during this window — no errors, no failed leader elections, ZooKeeper's leader_isr data is correct throughout. This is a pure client-observable symptom; the coordinator's authoritative state is never actually corrupted, it's just prematurely advertised.

Reproduction

1. Create a table with a large amount of data (large enough that a tablet server restart takes multiple minutes to complete local log recovery for its buckets).
2. Restart the tablet server hosting that table's leader/follower replicas (e.g. as part of a rolling update).
3. Have a client (e.g. a continuously-running scan/tiering job) keep reading from the affected buckets throughout the restart.
4. Observe NotLeaderOrFollowerException/LeaderNotAvailableException continuing to recur on the client well after the tablet server pod reports ready, until the coordinator happens to trigger another NotifyLeaderAndIsrRequest/UpdateMetadataRequest round for the affected buckets.

Small tables (log recovery completing in seconds) are unlikely to trigger this in practice — the window is too short to reliably observe.

Solution

I have a fix and will submit a PR shortly. Summary: the coordinator already has a pendingLeaderActivationBuckets mechanism in CoordinatorContext that tracks buckets awaiting NotifyLeaderAndIsrRequest acknowledgement, but it was previously only consulted by the read-only Cluster Health API. The fix wires this into CoordinatorRequestBatch#addUpdateMetadataRequestForTabletServers so that a bucket's leader is withheld from UpdateMetadataRequest for as long as it is marked pending, and the mark is only cleared once the corresponding NotifyLeaderAndIsrRequest is genuinely acknowledged by the target server (or the replica goes offline via re-election). This aligns "client-visible leader" with "server-admitted leader" without needing any client-side retry/backoff changes.

Are you willing to submit a PR?

• I'm willing to submit a PR!