Clean up old generations that are building up in active long-running workflows

Cargo.toml

@@ -36,8 +36,9 @@ uuid = { version = "1.0", features = ["v4", "serde"] }
 percent-encoding = "2.3"
 
 # duroxide integration
-duroxide = "=0.1.30"
-duroxide-pg = "=0.1.34"
+duroxide = "0.1.32"
+# DONOTLAND: requires the timestamp and batch-pruning fixes; replace with a crates.io release.
+duroxide-pg = { git = "https://github.com/microsoft/duroxide-pg", branch = "thomcc/continue_as_new_completed_at" }
 tokio = { version = "1", features = ["rt-multi-thread", "sync", "time"] }
 
 # For ExecuteSQL activity (connecting back to PostgreSQL)

USER_GUIDE.md

@@ -2531,19 +2531,25 @@ satellite `df._worker_epoch` is normal; satellites do not have separate workers.
 pg_durable tracks each workflow in its own `df` tables, while the durable engine
 keeps workflow state in a separate schema. To keep those two stores consistent —
 and to keep `df.instances`/`df.nodes` from growing without bound — the background
-worker runs a **best-effort reconciliation pass** that does two things:
+worker runs a **best-effort reconciliation pass** that:
 
 - **Removes expired terminal instances.** Old **terminal** instances (status
   `completed`, `failed`, or `cancelled`) and their `df.nodes` rows are deleted,
-    along with their engine records. Retention does not remove running or pending
-    instances from an existing installation, regardless of age.
+  along with their engine records. Retention does not remove running or pending
+  instances from an existing installation, regardless of age.
 - **Reclaims orphaned engine records.** `df.start()` writes the `df` rows in the
   caller's transaction but hands the workflow to the engine over a separate
   connection; if that transaction **rolls back**, the `df` rows vanish while the
   engine keeps an inert record (it can never load its rolled-back graph, so it
   ends up failed). Reconciliation deletes such df-less engine records once they
-  age past `retention_days`. Anything the engine is still tracking with a live
-  `df` row is left untouched.
+  age past `retention_days`. Instances with a live `df` row are not orphans.
+- **Prunes old execution generations.** Loops start a new execution generation
+  on each iteration. Reconciliation removes historical generations and their
+  history events once their completion is older than `retention_days`, including
+  those belonging to still-running workflows and nested loops in any origin.
+  The current execution and any running execution are always preserved. This
+  does not remove the workflow's `df` rows or its current results, but old
+  generations will no longer appear in execution-history inspection.
 
 Satellite maintenance visits `_origins` registrations made by activity routing for
 submitted work, in bounded batches under the shared origin connection budget.
@@ -2562,6 +2568,15 @@ for at most one 1,000-candidate page, then other origins get a turn before a bus
 origin resumes. Only age-expired or excess-over-10,000 rows consume the retention
 candidate budget. Legacy orphan candidates exclude satellite roots and children
 in SQL before their bounded page is loaded into memory.
+Execution pruning scans at most 1,000 execution keys per slice, including young
+and protected generations. Each batch deletes at most those 1,000 generations;
+this is a work limit, not a generation-count retention policy. Failed pages are
+logged, skipped for the rest of the scan, and retried on the next scan so they
+cannot block later pages. Pruning uses the maintenance connection, not the
+workflow runtime's connection pool, with server-side five-second statement and
+1.5-second lock deadlines. A timeout rolls back the batch. The number of history
+events in one generation is not capped; a batch that cannot finish within the
+deadline is deferred.
 
 Metadata queries have server-side deadlines and satellite operations have a
 15-second overall deadline. The candidate limit bounds materialized rows, not
@@ -2579,12 +2594,13 @@ Two Postmaster-context GUCs govern it (set in `postgresql.conf`, restart to appl
 pg_durable.reconcile_interval = 3600
 
 # Days a terminal instance is retained before reconciliation removes it (and its
-# engine record); also the age bound for reclaiming orphaned engine records.
-# 0 removes terminal instances as soon as the next pass runs.
+# engine record); also the age bound for orphaned engine records and completed
+# historical execution generations. The current execution is never pruned.
+# 0 makes cleanup eligible on the next pass.
 pg_durable.retention_days = 30
 ```
 
-Retention combines that window with a fixed hard cap:
+Terminal-instance retention combines that window with a fixed hard cap:
 
 - **Hard cap per origin — at most 10,000 terminal instances are retained, regardless of
   age.** The newest 10,000 terminal instances are kept; any beyond that are
@@ -2596,9 +2612,27 @@ Equivalently, a terminal instance is retained only while it is **both** among th
 newest 10,000 terminal instances **and** younger than `retention_days`; otherwise
 it is eligible for removal.
 
+Historical execution generations use only the age window, not a count cap.
+Pruning bounds how long completed history is retained, not its byte size: a
+busy workflow can still retain many generations within the window. An execution
+that has not continued into a new generation retains its current history.
+Completed child instances created by earlier loop iterations retain their current
+execution until the owning workflow is removed.
+Setting `reconcile_interval = 0` also disables execution pruning.
+
+On binary upgrade, the worker recovers missing completion timestamps for older
+continued generations from their recorded terminal history events. If an event
+is missing, it logs a migration warning and leaves that generation ineligible for
+age-based pruning rather than guessing its age. This one-time backfill runs
+before the worker becomes ready and can delay workflow processing on a large
+history store. Allow sufficient time for it during upgrade; a role or database
+`statement_timeout` shorter than the backfill will cause startup to retry rather
+than finish. Adjust the worker's migration-session timeout for the upgrade if
+necessary.
+
 Notes:
 
-- "Age" is measured from `completed_at` when set (instances that reached
+- Terminal-instance "age" is measured from `completed_at` when set (instances that reached
   `completed`), otherwise from `created_at`. `updated_at` is intentionally **not**
   used, because it is user-writable and would let a low-privilege user influence
   what is removed.

docs/upgrade-testing.md

@@ -412,6 +412,41 @@ the shipped 0.2.7 to 0.2.8 script must remain byte-identical.
 See [deferred lifetime guarantees](multi-database-installation.md#release-blockers)
 for the separate runtime shutdown and server-side cancellation limitations.
 
+#### Historical execution pruning
+
+- **B1:** background reconciliation prunes completed non-current execution
+  generations using `pg_durable.retention_days`, including running workflows and
+  nested loops in every origin. The query uses existing provider tables, so no
+  extension upgrade DDL or new `df` schema detection is needed.
+- **Provider migrations:** the worker's startup `ApplyAll` applies migration `0024`
+  before starting the runtime. It timestamps `ContinuedAsNew` executions and
+  backfills older NULL timestamps from terminal history events. Unrecoverable
+  timestamps remain NULL with a warning; those generations are not age-pruned.
+  Stop any other runtimes sharing the provider schema during this one-time
+  backfill. Existing histories may make startup slower; any worker-role/database
+  statement timeout must permit that backfill to finish, or startup retries it.
+  This dependency pair also includes `0023` (partial indexes), `0025` (fetch
+  leases), and `0026` (cursor-based batch pruning). Readiness schema version `3`
+  prevents backends from treating a pre-migration worker marker as ready.
+- **Bounded maintenance:** each slice scans at most 1,000 execution keys, including
+  protected rows. The provider owns deletion predicates and atomically prunes
+  that page on the maintenance connection under server-side deadlines. Failed
+  pages are revisited next scan; runtime provider connections are not borrowed.
+- **Inspection:** `df.instance_executions()` reads execution details and event
+  counts in one MVCC snapshot, so concurrent pruning does not produce spurious
+  per-generation not-found errors. Genuine query failures still raise errors.
+- **Replay:** no orchestration code, durable-operation inputs, or continuation
+  payloads change. Pruning preserves the current and running executions.
+- **Dependency pair:** duroxide `0.1.32` is required by the provider's current
+  branch. Its shutdown operation now cancels and joins owned tasks; zero means
+  immediate cancellation rather than a drain grace period. Validate the real
+  previous-binary lifecycle and worker shutdown when updating this pair.
+- **Release blocker:** the provider is temporarily pinned to the timestamp-fix
+  git branch in `Cargo.toml`. The local review fixes add migration `0026`; until
+  they are pushed, validation requires a local Cargo patch override. Refresh the
+  git lockfile after publishing those fixes and replace the `DONOTLAND` pin with
+  a compatible release before landing.
+
 #### Typed HTTP endpoints
 
 - Adds composite type `df.http_endpoint(server text, path text)`,

scripts/test-e2e-docker.sh

@@ -50,6 +50,7 @@ SKIP_TESTS=(
     "54_reconcile_orphans"
     "76_multi_database_reconcile"
     "88_multi_database_maintenance_backlog"
+    "89_execution_pruning"
     # Require the force-drop phase (one execution slot, reconciliation disabled).
     "78_multi_database_force_drop"
     "79_multi_database_remote_origin"

scripts/test-e2e-local.sh

@@ -221,7 +221,7 @@ phase_for_test() {
         46_connection_limit_startup_validation)
             echo "connlimit-startup"
             ;;
-        54_reconcile_orphans|76_multi_database_reconcile)
+        54_reconcile_orphans|76_multi_database_reconcile|89_execution_pruning)
             echo "reconcile"
             ;;
         88_multi_database_maintenance_backlog)

src/lib.rs

@@ -53,11 +53,13 @@ pub static LIST_INSTANCES_MAX_LIMIT: GucSetting<i32> = GucSetting::<i32>::new(10
 /// Days a terminal ('completed'/'failed'/'cancelled') instance is retained
 /// before reconciliation removes it and its engine record. The same age bound
 /// governs when orphaned engine records left by a rolled-back `df.start()` are
-/// reclaimed. `0` removes terminal instances as soon as the next pass runs.
+/// reclaimed and completed historical executions are pruned. The current
+/// execution is always preserved. `0` makes cleanup eligible on the next pass.
 pub static RETENTION_DAYS: GucSetting<i32> = GucSetting::<i32>::new(30);
 
 /// Seconds between background reconciliation passes. Each pass removes expired
-/// terminal instances and reclaims orphaned engine records. `0` disables it.
+/// terminal instances, reclaims orphaned engine records, and prunes historical
+/// executions. `0` disables it.
 pub static RECONCILE_INTERVAL: GucSetting<i32> = GucSetting::<i32>::new(3600);
 
 /// When `false`, the worker log omits the SQL text of executed workflow nodes.
@@ -93,7 +95,7 @@ pub use types::Durofut;
 /// by the background worker after successful initialization. Increment whenever
 /// a new binary introduces new duroxide-pg migration scripts or any other
 /// BGW-applied duroxide schema change.
-pub const WORKER_SCHEMA_VERSION: i32 = 2;
+pub const WORKER_SCHEMA_VERSION: i32 = 3;
 
 ::pgrx::pg_module_magic!(name, version);
 
@@ -290,8 +292,8 @@ pub extern "C-unwind" fn _PG_init() {
 
     GucRegistry::define_int_guc(
         c"pg_durable.retention_days",
-        c"Days a terminal instance is retained before reconciliation removes it",
-        c"Background reconciliation removes 'completed'/'failed'/'cancelled' df.instances rows (and their engine records) once they are older than this many days, subject to a fixed hard cap on the number retained. The same age bound governs when orphaned engine records left by a rolled-back df.start() are reclaimed. 0 removes terminal instances as soon as the next pass runs.",
+        c"Days to retain terminal instances and completed historical executions",
+        c"Background reconciliation removes 'completed'/'failed'/'cancelled' df.instances rows (and their engine records) once they are older than this many days, subject to a fixed hard cap on the number retained. The same age bound governs orphaned engine records and completed historical executions, including those of running workflows. Pruning always preserves the current execution. 0 makes cleanup eligible on the next pass.",
         &RETENTION_DAYS,
         0,
         36500,
@@ -302,7 +304,7 @@ pub extern "C-unwind" fn _PG_init() {
     GucRegistry::define_int_guc(
         c"pg_durable.reconcile_interval",
         c"Idle seconds between reconciliation scans (0 disables reconciliation)",
-        c"Unfinished bounded pages continue promptly between scans. Reconciliation removes expired terminal instances, orphaned engine records, and confirmed removed registrations with no engine work. Set to 0 to disable all reconciliation.",
+        c"Unfinished bounded pages continue promptly between scans. Reconciliation removes expired terminal instances, orphaned engine records, completed historical executions, and confirmed removed registrations with no engine work. Set to 0 to disable all reconciliation.",
         &RECONCILE_INTERVAL,
         0,
         86400,