feat(rewrite): add durable jobs and worker foundation

greenfield/docs/architecture/greenfield-rewrite/application-architecture.md

@@ -196,8 +196,8 @@ paths, code/module loaders, and process-execution authorities outside their expl
 Source-tree symlinks and imports outside the future root are prohibited. Fast Oxlint restrictions
 provide earlier feedback for supported import and global patterns; the AST check is the path-aware
 policy gate for the source surfaces it explicitly scans, not a replacement for runtime sandboxing.
-The browser and worker roles are already classified even where their composition remains future
-work.
+The browser and worker roles are classified, and the worker composition owns the durable
+coordinator, database runtime, process signals, and ordered shutdown boundary.
 
 ## Application API
 
@@ -389,7 +389,7 @@ once. Reusable procedure builders are limited to:
 
 Expected errors use a small stable code set such as `UNAUTHORIZED`, `FORBIDDEN`, `CONFLICT`,
 `NOT_FOUND`, `PRECONDITION_FAILED`, `TOO_MANY_REQUESTS`, and `SERVICE_UNAVAILABLE` with safe
-structured details. The `ContractErrorCode` union, all 68 actual router paths, the server-owned
+structured details. The `ContractErrorCode` union, all 77 actual router paths, the server-owned
 runtime allowlist, and generated contract metadata must match exactly. The base procedure
 middleware enforces that allowlist for immediate and deferred subscription failures; an
 implemented procedure missing from the policy or an undeclared code becomes a redacted internal

greenfield/docs/architecture/greenfield-rewrite/data-and-security.md

@@ -24,8 +24,8 @@
   write-admission port: it may retry only before `BEGIN IMMEDIATE` admits the transaction and the
   synchronous callback starts. A callback is never replayed. Exhausted admission and post-admission
   contention remain typed failures; only mutation routes that declare temporary write unavailability
-  expose the fixed redacted `SERVICE_UNAVAILABLE` response. The future worker must use the same
-  explicit policy before that process starts.
+  expose the fixed redacted `SERVICE_UNAVAILABLE` response. The worker uses that same explicit
+  admission policy before entering any immediate transaction.
 - Use Drizzle's typed query builder for ordinary reads/writes and its parameterized `sql`
   tagged template for SQLite-specific queries, CTEs, queue claims, and expressions not
   represented cleanly by the builder.
@@ -111,7 +111,7 @@ tag.
 | Tasks/agents         | `tasks`, `task_labels`, `task_automation_profiles`, `task_updates`, `task_events`, `agent_task_runs`                                                                                                                                                                           |
 | Monitoring           | `reports`, `monitor_runs`, `incidents`, `incident_observations`, `notifications`                                                                                                                                                                                               |
 | Realtime             | `realtime_events`                                                                                                                                                                                                                                                              |
-| Scheduling/work      | `scheduled_jobs`, `job_disable_intents`, `job_runs`, `job_run_events`, `worker_instances`, `resource_leases`                                                                                                                                                                   |
+| Scheduling/work      | `scheduled_jobs`, `job_disable_intents`, `job_runs`, `job_run_events`, `worker_instances`, `resource_leases`, `job_worker_control`                                                                                                                                             |
 | Chat                 | `chat_runs`, `chat_run_events`, `chat_runtime_snapshots`                                                                                                                                                                                                                       |
 | Delivery             | `deployments`, `deployment_events`, `release_records`                                                                                                                                                                                                                          |
 | Docker               | `managed_docker_services`, `docker_update_events`                                                                                                                                                                                                                              |
@@ -199,9 +199,9 @@ queryable lifecycle.
 | Incident identity             | unique `incidents(monitor_key, fingerprint)`                                              |
 | Unread notifications          | partial `notifications(occurred_at_ms DESC) WHERE read_at_ms IS NULL`                     |
 | Incident notification         | unique `(incident_id, incident_generation, channel)` when incident is non-null            |
-| Queue claim                   | partial `job_runs(available_at_ms, priority DESC, queued_at_ms) WHERE state = 'queued'`   |
+| Queue claim                   | partial `job_runs(available_at, priority DESC, queued_at, id) WHERE state = 'queued'`     |
 | One active scheduled run      | unique partial `job_runs(scheduled_job_id) WHERE state IN ('queued', 'running')`          |
-| Worker expiry                 | `worker_instances(heartbeat_at_ms)`                                                       |
+| Worker expiry                 | `worker_instances(heartbeat_at, id)`                                                      |
 | Job timeline                  | `job_run_events(job_run_id, sequence)`                                                    |
 | Realtime catch-up             | `realtime_events(topic, id)`                                                              |
 | Chat replay                   | unique `chat_run_events(chat_run_id, sequence)`                                           |
@@ -271,9 +271,15 @@ the worker.
 
 Queue behavior is explicit:
 
-- a transaction claims one eligible run and assigns a lease;
+- a strict singleton `job_worker_control` row persists cross-process claim pause state and
+  versioned operator changes; its absence is an integrity failure, never an implicit resume;
+- one immediate transaction considers at most 32 totally ordered candidates, skips candidates
+  with occupied resources, and atomically assigns the first eligible run plus every required
+  resource lease;
 - each run has an idempotency key, resource class, priority, timeout, attempt limit, and
   cancellation policy;
+- manual-run idempotency is scoped to the requesting principal and hashes stable request intent,
+  while schedule ticks use a deterministic schedule-and-occurrence namespace;
 - the worker renews its lease and writes ordered progress events;
 - expired leases can be recovered only when the action is declared retry-safe;
 - resource leases prevent conflicting deploy, restore, Docker, or OpenClaw operations;
@@ -282,6 +288,16 @@ Queue behavior is explicit:
   necessary only beneath `<project-root>/production/state/job-output`; and
 - final structured output is validated before persistence or display.
 
+Run history is bounded to 1,000 events and 1 MiB of encoded payload per run. The first 967 slots
+may carry progress/stdout/stderr payloads; 33 structural slots remain reserved so every legal
+ten-attempt lifecycle can record claims, retry decisions, cancellation, truncation, and a terminal
+event. Schedule cursor advancement is separate from operator configuration versioning. A due
+schedule with an active queued or running occurrence keeps its original cursor; after completion
+the scheduler creates one coalesced run for that occurrence and advances directly to the first
+future occurrence, never replaying an unbounded backlog. Disabled schedules retain that cursor as
+an internal cadence anchor but do not become due until re-enabled; the public summary exposes a
+next run only while enabled. Manual runs do not move cadence.
+
 `Bun.spawn` receives argument arrays, a deliberate environment allowlist, an explicit working
 directory, a timeout, and an abort signal. It never receives interpolated shell text for user
 input. High-risk jobs run in dedicated transient systemd units or templates with their own

greenfield/docs/architecture/greenfield-rewrite/progress.md

@@ -7,15 +7,15 @@
 This matrix is the living phase status. Update it in the same change that materially advances or
 closes a phase; dated entries below provide the evidence, not a second status source.
 
-| Phase                               | Status                               | Current evidence and remaining gate                                                                                                                                                                                                                                                                                                                                          |
-| ----------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| 0 — Evidence and qualification      | Complete                             | All eight mandatory spikes pass on exact Bun revision `17d6843606d76620cb55d31424d7fb0aed51c367`: build, transport, cross-process SQLite/outbox, Drizzle/Bun SQLite, browser data, chat batching, shutdown, and capped resources. Source-derived parity and the OpenClaw source audit pass as additional evidence.                                                           |
-| 1 — Foundation                      | Complete                             | The self-contained future root builds immutable browser/web/worker artifacts, protects project-local production state, installs exact Bun and systemd artifacts, migrates a database copy, atomically promotes the release/database pair, serves readiness/browser assets, writes project-local logs, and proves crash-safe rollback and shutdown in a disposable lifecycle. |
-| 2 — Trust and transport             | Complete for the stated server scope | Authentication, MFA, WebAuthn, automation credentials, audit, authenticated renewable SSE, one-shot native Gateway bootstrap verification, and the consolidated [threat model](../../security/greenfield-phase-two-threat-model.md) have executable evidence. Browser UI and production cutover remain later gates.                                                          |
-| 3 — Core operator domains           | Started                              | Task and agent-directory parity are implemented with durable history, realtime invalidation, and browser workflows. Monitoring ingestion plus report, incident, and notification server parity are implemented; report, incident, and global notification browser state are also complete. Schedules/jobs, overview, cache/metrics, and the real worker remain open.         |
-| 4 — Gateway and chat                | Not started                          | The Phase 2 verifier is one-shot only. Persistent native Gateway lifecycle, current-protocol re-audit, sessions, chat journal/recovery, attachments, and frontend remain open.                                                                                                                                                                                               |
-| 5 — Privileged and external domains | Not started                          | Worker-owned file/media, Docker, database, OpenClaw, GitHub, deployment, backup, and other privileged adapters remain open.                                                                                                                                                                                                                                                  |
-| 6 — Parity, hardening, and cutover  | Not started                          | Full UI parity, generated `/docs`, load/resource/restore evidence, cutover rehearsal, fresh production database, and legacy removal remain open.                                                                                                                                                                                                                             |
+| Phase                               | Status                               | Current evidence and remaining gate                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| ----------------------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 0 — Evidence and qualification      | Complete                             | All eight mandatory spikes pass on exact Bun revision `17d6843606d76620cb55d31424d7fb0aed51c367`: build, transport, cross-process SQLite/outbox, Drizzle/Bun SQLite, browser data, chat batching, shutdown, and capped resources. Source-derived parity and the OpenClaw source audit pass as additional evidence.                                                                                                                                        |
+| 1 — Foundation                      | Complete                             | The self-contained future root builds immutable browser/web/worker artifacts, protects project-local production state, installs exact Bun and systemd artifacts, migrates a database copy, atomically promotes the release/database pair, serves readiness/browser assets, writes project-local logs, and proves crash-safe rollback and shutdown in a disposable lifecycle.                                                                              |
+| 2 — Trust and transport             | Complete for the stated server scope | Authentication, MFA, WebAuthn, automation credentials, audit, authenticated renewable SSE, one-shot native Gateway bootstrap verification, and the consolidated [threat model](../../security/greenfield-phase-two-threat-model.md) have executable evidence. Browser UI and production cutover remain later gates.                                                                                                                                       |
+| 3 — Core operator domains           | Started                              | Task and agent-directory parity are implemented with durable history, realtime invalidation, and browser workflows. Monitoring ingestion plus report, incident, and notification server parity are implemented; report, incident, and global notification browser state are also complete. Dashboard-local durable schedules/jobs and real worker execution are implemented. The `/jobs` browser, OpenClaw cron, overview, and cache/metrics remain open. |
+| 4 — Gateway and chat                | Not started                          | The Phase 2 verifier is one-shot only. Persistent native Gateway lifecycle, current-protocol re-audit, sessions, chat journal/recovery, attachments, and frontend remain open.                                                                                                                                                                                                                                                                            |
+| 5 — Privileged and external domains | Not started                          | Worker-owned file/media, Docker, database, OpenClaw, GitHub, deployment, backup, and other privileged adapters remain open.                                                                                                                                                                                                                                                                                                                               |
+| 6 — Parity, hardening, and cutover  | Not started                          | Full UI parity, generated `/docs`, load/resource/restore evidence, cutover rehearsal, fresh production database, and legacy removal remain open.                                                                                                                                                                                                                                                                                                          |
 
 ### 2026-08-03 — Phase 0 started
 
@@ -916,3 +916,32 @@ full-browser parity, production rehearsal, cutover, and legacy deletion remain o
   performs a new transport request instead of reviving an empty ready instance.
 - Notification server and browser parity are now complete. Schedules/jobs, overview,
   cache/metrics, and real worker execution remain open Phase 3 gates.
+
+### 2026-08-07 — Phase 3 durable schedules, jobs, and worker foundation
+
+- Seven strict scheduling tables now persist the reviewed schedule directory, explicit disable
+  intents, one durable run state machine, bounded ordered events, worker instances, fenced
+  resource leases, and the singleton cross-process claim-pause control. SQL checks and triggers
+  protect immutable execution snapshots, legal lifecycle transitions, append-only history,
+  canonical resource sets, caller-scoped idempotency, event/byte reservations, and optimistic
+  versions even when writes bypass the service layer.
+- Nine `jobs:read`/`jobs:write` procedures expose stable keyset-paginated run and schedule reads,
+  session-only cancel/pause/update operations, and a caller-scoped idempotent manual-run boundary.
+  Automation can invoke only registry actions explicitly marked for `jobs:write`; this slice
+  exposes only the harmless `system.worker-smoke` action. Durable audit rows and compact
+  `jobs.runs` / `schedules.records` invalidations commit with each externally visible mutation.
+- Schedule cadence is distinct from operator configuration versioning. A due schedule with active
+  work retains its cursor and later coalesces exactly one occurrence; manual runs never move it.
+  Disabled schedules retain a dormant internal cursor, so expiry or re-enable resumes interval
+  cadence without drift. Expired intents close under a system actor and re-enable the schedule in
+  one admitted transaction, while disabling cancels only queued schedule-triggered work.
+- The separate Bun worker now owns an Effect-coordinated single-capacity execution loop with
+  registration/heartbeat, bounded recovery and candidate scans, atomic resource claims, lease
+  renewal, persisted cooperative cancellation, retry-safe backoff, timeout, bounded progress and
+  output, fenced settlement, and ordered drain/stop. Unexpected heartbeat, scheduler, claim, or
+  coordinator completion fails the process rather than leaving a zombie worker.
+- A migrated-database system test enqueues the code-owned smoke action through the shared
+  repository, lets the worker claim it, and observes a durable successful result without shell,
+  Gateway, or host-mutation authority. The parity inventory marks the nine Dashboard jobs and
+  schedules operations implemented. The five `openClawCron.*` operations and `/jobs` browser
+  remain planned, along with overview and cache/metrics.