Skip to content

Release delayed workflows against the database's clock, and stamp updated_at - #613

Open
devhawk wants to merge 1 commit into
workflow-timeout-sweepfrom
delay-release-db-clock
Open

devhawk wants to merge 1 commit into
workflow-timeout-sweepfrom
delay-release-db-clock

Conversation

@devhawk

@devhawk devhawk commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #612.

A DELAYED workflow was released when its delay_until_epoch_ms was at or before the clock of whichever JVM ran the release. That is often not the JVM that enqueued it. transitionDelayedWorkflows now compares against the database's now(), the one clock every executor sharing the system database shares, so whichever executor runs the release, a delay ends at the same instant.

  • updated_at: the release also stamps it from the database's clock. The statement used to leave it unchanged.
  • Not changed: the end of a relative delay is still resolved from the enqueuing JVM's clock, once, outside the write's retry loop. Moving that into the statement too would let a retried insert push a delay later by its backoff, which would have to be settled for every SDK first. So the enqueuer's clock still sets the delay; only the releasing executor's clock is gone. Go releases delays the same way.
  • Javadoc: the NOW_EPOCH_MS javadoc now lists the release among the times kept on the database's clock.

Tests

DelayReleaseClockTest (new, Postgres only):

  • shifts the database's clock an hour either way by shadowing now() and clock_timestamp() on the connections' search_path;
  • delays a workflow by a day, then moves its delay to end a second from the database's now();
  • checks that it is released a second later, with updated_at on the database's clock.

The executor in the test doesn't dequeue the workflow's queue, so the released row stays ENQUEUED and keeps the release's own updated_at. On the old statement the test fails both ways: with the database an hour ahead the workflow is never released, and with it an hour behind it is released at once.

Run locally on Postgres: spotlessCheck, compileTestJava and javadoc, plus:

  • DelayReleaseClockTest, DatabaseClockTest, WorkflowTimeoutSweepTest, TimeoutTest and DebounceDelayedWorkflowTest on this branch;
  • DebouncerTest, DebouncerClientTest, StartWorkflowTest, ClientTest and SystemDatabaseTest before it was stacked.

🤖 Generated with Claude Code

@devhawk
devhawk added this pull request to stack #614 October 8, 2026 21:08
…ated_at

A DELAYED workflow was released when its delay_until_epoch_ms was at or
before the clock of whichever JVM ran the release, which is often not the
JVM that enqueued it. Every executor sharing a system database shares its
clock, so the release now compares against the database's now(). Whichever
executor runs the release, a delay ends at the same instant.

The release also stamps updated_at, which it used to leave unchanged.

The end of a relative delay is still resolved from the enqueuing JVM's
clock, once, outside the write's retry loop.

DelayReleaseClockTest shifts the database's clock an hour either way by
shadowing now() and clock_timestamp() on the connections' search path,
sets a delay to end a second from the database's now, and checks the
workflow is released a second later with updated_at on the database's
clock. On the JVM's clock it was never released with the database an hour
ahead, and released at once with it an hour behind.
@devhawk
devhawk force-pushed the delay-release-db-clock branch from 25fe0f5 to 100bc77 Compare October 8, 2026 21:24
@devhawk
devhawk requested a balanced review from Copilot October 8, 2026 22:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused implementation is consistent with existing database-time handling and is adequately tested.

0 open findings

What changed in this PR

Uses the shared database clock to release delayed workflows consistently across executors and stamp release time.

Changes:

  • Compares delayed-workflow deadlines against database now().
  • Updates updated_at when releasing workflows.
  • Adds PostgreSQL clock-skew coverage and updates clock documentation.
File Description
DelayReleaseClockTest.java Tests release timing and timestamping under clock skew.
SystemDatabase.java Documents database-clock use for delayed releases.
WorkflowDAO.java Uses database time for delayed transitions and updated_at.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants