fix: recover from half-open postgres connections with bounded liveness ping - #479
Draft
TimoGlastra wants to merge 1 commit into
Draft
fix: recover from half-open postgres connections with bounded liveness ping#479TimoGlastra wants to merge 1 commit into
TimoGlastra wants to merge 1 commit into
Conversation
TimoGlastra
marked this pull request as draft
July 24, 2026 12:13
TimoGlastra
force-pushed
the
fix/postgres-half-open-connections
branch
from
July 24, 2026 12:14
fb6e12b to
71af9da
Compare
…s ping The postgres pool sets test_before_acquire(false), so a pooled connection that died while idle is handed out untested and the next operation fails. Worse, if the connection went half-open (silently dropped by a NAT or proxy, as documented for Cloud Run and similar infrastructure), the next operation hangs until the kernel gives up retransmitting (~15 minutes), with no timeout at any layer above it. Enabling test_before_acquire(true) only fixes the clean-drop case: for a half-open connection the test ping itself hangs, and since it shares the acquire_timeout deadline, the acquire still fails. Instead, add a before_acquire hook that pings idle connections under a separate short deadline. A hung ping counts as a failed test, so the pool discards the connection and tries another (or dials a fresh one) within the regular acquire timeout. Connections used within the last second skip the ping, keeping hot-path overhead at zero (the same heuristic used by pgxpool, see jackc/pgx#1838). Two new postgres URI options make this tunable: - ping_timeout (seconds, default 1): deadline for the liveness ping on an idle connection; 0 disables the check, restoring the old behavior - min_idle_for_ping (seconds, default 1): minimum idle time before a connection is pinged on acquire; 0 pings on every acquire Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Timo Glastra <timo@animo.id>
TimoGlastra
force-pushed
the
fix/postgres-half-open-connections
branch
from
July 24, 2026 12:17
71af9da to
c2bb0d3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The postgres pool is built with
.test_before_acquire(false), and sqlx sets no TCP keepalive. A pooled connection that died while idle is therefore handed out untested, and the first operation on it fails — or worse, hangs.Two distinct failure modes:
Why not just
test_before_acquire(true)That fixes the clean-drop case transparently (the ping fails instantly, sqlx discards the connection and redials within the same acquire). But for a half-open connection the test ping itself hangs, and since it shares the overall
acquire_timeoutdeadline, the whole acquire still fails afterconnect_timeout— even though the database is perfectly reachable.Fix
A
before_acquirehook that pings idle connections under its own short deadline. A hung ping counts as a failed test, so the pool discards the connection and moves on to another one or a fresh dial, all within the regular acquire timeout. Connections used within the last second skip the ping, keeping hot-path overhead at zero — the same heuristic pgxpool uses (see jackc/pgx#1838).Validation
Tested against a local postgres behind a TCP proxy that can (a) cleanly kill connections, (b) silently blackhole existing connections while new ones work, (c) blackhole everything. URI:
connect_timeout=4&idle_timeout=60&max_connections=30; each check =store.session().open()+ fetch + close.test_before_acquire(true)With this change, callers never see an error for a connection that died while idle as long as the database is actually reachable: each dead connection just adds ~
ping_timeoutto the first acquire that touches it.Additionally, the full postgres backend test suite passes with
min_idle_for_ping=0(ping forced on every acquire), confirming the hook does not discard healthy connections.Configuration
Two new postgres URI query parameters, following the existing option conventions (whole seconds):
ping_timeout(default1): deadline for the liveness ping on an idle connection.0disables the check entirely, restoring the previous behavior.min_idle_for_ping(default1): minimum time a connection must have been idle before it is pinged on acquire.0pings on every acquire.The sqlite backend also sets
test_before_acquire(false)but is left untouched, since a liveness ping is meaningless for a local file database.🤖 Generated with Claude Code