Skip to content

fix: recover from half-open postgres connections with bounded liveness ping - #479

Draft
TimoGlastra wants to merge 1 commit into
mainfrom
fix/postgres-half-open-connections
Draft

fix: recover from half-open postgres connections with bounded liveness ping#479
TimoGlastra wants to merge 1 commit into
mainfrom
fix/postgres-half-open-connections

Conversation

@TimoGlastra

Copy link
Copy Markdown
Contributor

Problem

The postgres pool is built with .test_before_acquire(false), and sqlx sets no TCP keepalive. A pooled connection that died while idle is therefore handed out untested, and the first operation on it fails — or worse, hangs.

Two distinct failure modes:

  • Clean drop (server closed the connection, FIN/RST received): the next operation errors immediately. Annoying but fast.
  • Half-open drop (connection silently discarded by intermediate infrastructure — NAT gateways, Cloud Run, load balancers all document doing this to idle TCP flows): the next operation just hangs. The kernel keeps retransmitting for ~15 minutes and there is no timeout at any layer above it.

Why not just test_before_acquire(true)

That fixes the clean-drop case transparently (the ping fails instantly, sqlx discards the connection and redials within the same acquire). But for a half-open connection the test ping itself hangs, and since it shares the overall acquire_timeout deadline, the whole acquire still fails after connect_timeout — even though the database is perfectly reachable.

Fix

A before_acquire hook that pings idle connections under its own short deadline. A hung ping counts as a failed test, so the pool discards the connection and moves on to another one or a fresh dial, all within the regular acquire timeout. Connections used within the last second skip the ping, keeping hot-path overhead at zero — the same heuristic pgxpool uses (see jackc/pgx#1838).

Validation

Tested against a local postgres behind a TCP proxy that can (a) cleanly kill connections, (b) silently blackhole existing connections while new ones work, (c) blackhole everything. URI: connect_timeout=4&idle_timeout=60&max_connections=30; each check = store.session().open() + fetch + close.

Scenario stock v0.5.0 test_before_acquire(true) bounded ping (this PR)
idle conns killed cleanly error per dead conn OK ~20ms OK ~20ms
idle conns half-open, DB fine hung 20s+ (unbounded) error at 4s (acquire timeout) OK ~1s, no error
DB fully unreachable hung (unbounded) error at 4s error at 4s (correct)

With this change, callers never see an error for a connection that died while idle as long as the database is actually reachable: each dead connection just adds ~ping_timeout to the first acquire that touches it.

Additionally, the full postgres backend test suite passes with min_idle_for_ping=0 (ping forced on every acquire), confirming the hook does not discard healthy connections.

Configuration

Two new postgres URI query parameters, following the existing option conventions (whole seconds):

  • ping_timeout (default 1): deadline for the liveness ping on an idle connection. 0 disables the check entirely, restoring the previous behavior.
  • min_idle_for_ping (default 1): minimum time a connection must have been idle before it is pinged on acquire. 0 pings on every acquire.

The sqlite backend also sets test_before_acquire(false) but is left untouched, since a liveness ping is meaningless for a local file database.

🤖 Generated with Claude Code

@TimoGlastra TimoGlastra linked an issue Jul 24, 2026 that may be closed by this pull request
@TimoGlastra
TimoGlastra marked this pull request as draft July 24, 2026 12:13
@TimoGlastra
TimoGlastra force-pushed the fix/postgres-half-open-connections branch from fb6e12b to 71af9da Compare July 24, 2026 12:14
…s ping

The postgres pool sets test_before_acquire(false), so a pooled connection
that died while idle is handed out untested and the next operation fails.
Worse, if the connection went half-open (silently dropped by a NAT or
proxy, as documented for Cloud Run and similar infrastructure), the next
operation hangs until the kernel gives up retransmitting (~15 minutes),
with no timeout at any layer above it.

Enabling test_before_acquire(true) only fixes the clean-drop case: for a
half-open connection the test ping itself hangs, and since it shares the
acquire_timeout deadline, the acquire still fails.

Instead, add a before_acquire hook that pings idle connections under a
separate short deadline. A hung ping counts as a failed test, so the pool
discards the connection and tries another (or dials a fresh one) within
the regular acquire timeout. Connections used within the last second skip
the ping, keeping hot-path overhead at zero (the same heuristic used by
pgxpool, see jackc/pgx#1838).

Two new postgres URI options make this tunable:

- ping_timeout (seconds, default 1): deadline for the liveness ping on an
  idle connection; 0 disables the check, restoring the old behavior
- min_idle_for_ping (seconds, default 1): minimum idle time before a
  connection is pinged on acquire; 0 pings on every acquire

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Timo Glastra <timo@animo.id>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reason for setting test_before_acquire(false)?

1 participant