[FIP-28] Refresh server partition status for historical write routing

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Description

Parent issue: #3631

Follow-up to #4119 and PR #4120.

Historical write routing currently uses the client-side auto-partition retention boundary as a precheck before refreshing metadata for an original partition. The client calculation is not authoritative: the client and Coordinator may evaluate time expressions with different default time zones, and the client metadata cache may remain stale while the server creates, retains, freezes, or removes a partition. Performing a synchronous metadata refresh on the per-record path is also too expensive.

Introduce a periodically refreshed, server-authoritative partition status for historical write routing. The client should fetch and cache partition existence or lifecycle information from the server in a batched and rate-limited manner, rather than repeatedly deriving the routing decision from its local clock.

The implementation should:

• Refresh relevant partition status periodically or when server metadata indicates that the cached status is stale, without synchronous RPCs on every record.
• Deduplicate concurrent refreshes per table or original partition path.
• Keep writes on the original partition while the server reports it as active or still present.
• Permit historical routing only after the server reports a state that deterministically allows it. This should align with the durable partition retirement protocol discussed in [server] add lake-aware partition drop when auto-partition retention is enabled #3820, where a generic missing or freezing state is not sufficient to reroute.
• Treat client-side time expressions only as an optional refresh hint, not as the source of truth for partition existence or retirement.
• Define cache invalidation and recovery behavior across metadata refreshes and client restarts.

This avoids early routing caused by client/server time-zone differences and reduces late detection caused by stale metadata, while keeping metadata I/O off the per-record hot path.

Willingness to contribute

• I am willing to submit a PR!