Skip to content

feat: cache: redis: configurable client timeouts and idle connection recycling - #597

Open
rnjn wants to merge 1 commit into
ContentSquare:masterfrom
base-14:feat/redis-client-timeouts
Open

rnjn wants to merge 1 commit into
ContentSquare:masterfrom
base-14:feat/redis-client-timeouts

Conversation

@rnjn

@rnjn rnjn commented Oct 8, 2026 •

Copy link
Copy Markdown

Description

Fixes #596. Related: #337, #259.

The redis cache client had no configurable timeouts, and go-redis keeps idle pooled connections for 30 minutes. When a firewall or NAT silently drops idle connections (e.g. GCP VPC connection tracking after 10 minutes), commands stall on dead connections: each one waits for the 3s read timeout, up to 8 times with retries. That blocks request handling for a long time.

This PR adds optional redis client settings to the cache config:

redis:
  dial_timeout: 2s          # default 5s (go-redis)
  read_timeout: 1s          # default 3s (go-redis)
  write_timeout: 1s         # default read_timeout (go-redis)
  pool_timeout: 2s          # default read_timeout + 1s (go-redis)
  conn_max_idle_time: 4m    # default 5m (new chproxy default; go-redis default is 30m)
  max_retries: 2            # default 7 (unchanged); -1 disables retries
  • clients/redis.go: option building moved into a pure redisOptions(cfg) function so it can be unit-tested without a redis server.
  • When a field is omitted, the timeouts fall back to the go-redis defaults and max_retries stays at 7, so existing behaviour is preserved. The one change is that idle connections are now recycled after 5m instead of 30m.
  • config/README.md documents the new fields.

Pull request type

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

Checklist

  • Linter passes correctly (golangci-lint v1.64.8, go vet)
  • Add tests which fail without the change (if possible)
  • All tests passing. TestReverseProxy_ServeHTTP1/queue_overflow_for_user fails intermittently, and fails the same way on master; it is unrelated to redis.
  • Extended the README / documentation, if necessary

Does this introduce a breaking change?

  • Yes
  • No

Further comments

Not in scope here (possible follow-up): setting ContextTimeoutEnabled: true so the existing per-operation context.WithTimeout deadlines in cache/redis_cache.go (500ms–2s) also bound socket I/O, and giving the redis transaction registry real deadlines instead of context.Background(). That would change behaviour for existing users, so I kept it separate.

…recycling

Expose dial_timeout, read_timeout, write_timeout, pool_timeout,
conn_max_idle_time and max_retries in the redis cache config.

conn_max_idle_time defaults to 5m (go-redis default is 30m) so pooled
connections are recycled before firewall/NAT idle timeouts (e.g. 10m on
GCP) silently drop them, which made cache operations stall on dead
connections.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] chproxy stalls for minutes when the redis/valkey connection breaks: redis client timeouts are not configurable

1 participant