refactor: add support for postgres store(s) - #28
Merged
Conversation
frrist
marked this pull request as ready for review
April 23, 2026 18:37
alanshaw
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #27.
Adds a PostgreSQL backend alongside the existing DynamoDB and in-memory backends so sprue can run self-hosted as part of the Forge stack.
Summary
Thirteen new
postgres/store implementations (one per domain interface underpkg/store/), a goose-based migrations runner, an fx module that wires them in, astorage.typebackend selector in config, and a docker-compose dev stack built around Postgres + MinIO. The default backend is nowpostgres.The UCAN handlers, routing, service layer, and external clients are untouched — the
Storeinterfaces are the seam and no interface changed in this PR.Scope — in
Store implementations
New
pkg/store/<domain>/postgres/store.gofor each of the 13 domains, matching the existingaws/andmemory/layout:agent,blob_registry,consumer,customer,delegation,metrics(bothSpaceStoreand adminStore),replica,revocation,space_diff,storage_provider,subscription,upload.The three stores that also persist blob payloads (
agent,delegation,upload) keep their S3 side unchanged — only the DynamoDB metadata half moved to Postgres.Schema + migrations
internal/migrations/sql/— one goose migration per store (00001_init.sql…00013_agent_index.sql).internal/migrations/migrations.go— embedded FS + agoose.UpContextrunner, with a zap adapter for goose logging.storage.postgres.skip_migrations: truedisables them.fx wiring
internal/fx/store/postgres/provider.go— newpostgres.Modulethat provides the pool, the migrated pool, the S3 client (reused from the AWS module for the three stores that need it), and every Postgres-backed store.internal/fx/app.go—AppModulenow switches oncfg.Storage.Type(memory|postgres|aws) and wires the matching store module. An emptyTypeis treated aspostgresso tests constructingConfigliterals don't have to set it.Config
internal/config/config.go— replaces the oldInMemoryStoresboolean with aStorageConfig.Typeselector (memory|postgres|aws, defaultpostgres). AddsPostgresConfig(DSN, max/min conns, skip_migrations). DynamoDB and S3 settings moved underStorageConfigfor symmetry.SPRUE_*convention:SPRUE_STORAGE_TYPE,SPRUE_STORAGE_POSTGRES_DSN,SPRUE_STORAGE_POSTGRES_MAX_CONNS,SPRUE_STORAGE_POSTGRES_SKIP_MIGRATIONS, etc.Tests
internal/testutil/postgres.go— spins up a throwaway Postgres container viatestcontainers-go, applies migrations, and returns a*pgxpool.Pool.*_test.gostore test matrix gains apostgrescase alongsidememory/aws— same assertions run against all three backends.internal/fx/app_test.goTestWireAppexercises Postgres wiring end-to-end.Dev stack
docker-compose.yaml(top level) — Postgres + MinIO + sprue service, healthchecks, sprue env vars pre-wired to the Postgres backend.Dockerfile,Makefile,config.example.yaml,README.md,CLAUDE.mdupdated for the new backend + config surface.Design decisions
pgx + pgxpool, no ORM, no sqlc (yet)
Queries are hand-written SQL via
pgx.Pool.QueryRow/Exec. Reasons:blob_registrycoordinating withconsumer,space_diff,metrics) — explicit coordination is already the pattern, an opaque ORM would fight it.goose for migrations, embedded from the binary
Matches the convention already set by Guppy in the Forge stack. Migrations are embedded via
//go:embed sql/*.sqlso there's nothing to copy at deploy time. Running at startup (and skippable via config) keeps operator workflow simple for the early self-hosted deployments this unblocks.Single shared schema, one migration per store
One Postgres schema holds all 13 stores' tables. Per-store schemas would buy nothing (no access-control story here) and would force every cross-domain query to be schema-qualified. One migration per store keeps ownership boundaries clear within the shared schema.
MigratedPoolas a lifecycle ordering primitiveMigratedPoolis a typed wrapper around*pgxpool.Poolwhose sole job is to order fx lifecycle hooks.NewMigratedPoolregisters the migrationOnStarthook; every store constructor depends on*MigratedPool(not*pgxpool.Pool), so the fx dependency graph resolves migrations first and — sinceOnStarthooks fire in registration order — guarantees every store'sInitializehook sees a fully migrated schema. No explicit sequencing code, no ad-hoc "wait for migrations" checks.Postgres is the default backend
storage.typedefaults topostgres(with a sensible default DSN for the docker-compose stack). EmptyTypeis also treated aspostgresinAppModuleso test-constructed configs don't need to set it. AWS stays available viastorage.type: aws; memory stays available for tests and local experimentation.Agent store: removed the async job queue (commit
2e66d37)The Postgres agent store originally mirrored the AWS store by using
go-libstoracha/jobqueueto fan out writes (one S3 put + N index inserts perWrite). That design has durability holes independent of the Postgres backend — the job handler runs on a background context, so cancellation and shutdown can leak or drop in-flight work, and the separate jobs offer no atomicity across the fan-out. Full analysis and the AWS counterpart are tracked in #33.For the Postgres store specifically, the job queue has been replaced with:
ctx; cancellation propagates.INSERT ... VALUES (…), (…) ON CONFLICT (task, kind) DO UPDATE. All index rows commit atomically in one statement, preserving the AWS overwrite semantic for cross-call conflicts.agent.Indexcan yield the same(task, kind)pair more than once (e.g. a receipt re-yielding its invocation viaRan()); rows are deduped in Go before the INSERT. Forkind="in"this is provably safe (task CID is the invocation CID). Potential ambiguity forkind="out"across different receipts in one message is a known edge case — see discussion in Remove jobqueue from agent.Store — unneeded complexity with durability risks #33; not changed here because it exists identically in the AWS store and resolving it is orthogonal to the Postgres port.Scope — out
pkg/store/agent/aws/store.go; tracked in Remove jobqueue from agent.Store — unneeded complexity with durability risks #33. Not touched here so this PR stays scoped to the Postgres port.Gates before merge (still-draft items)
Open questions
sprue,forge, something else) — currently uses the defaultpublicschema.