diff --git a/greenfield/docs/architecture/greenfield-rewrite/application-architecture.md b/greenfield/docs/architecture/greenfield-rewrite/application-architecture.md index e5427878a..c5eabe0fd 100644 --- a/greenfield/docs/architecture/greenfield-rewrite/application-architecture.md +++ b/greenfield/docs/architecture/greenfield-rewrite/application-architecture.md @@ -196,8 +196,8 @@ paths, code/module loaders, and process-execution authorities outside their expl Source-tree symlinks and imports outside the future root are prohibited. Fast Oxlint restrictions provide earlier feedback for supported import and global patterns; the AST check is the path-aware policy gate for the source surfaces it explicitly scans, not a replacement for runtime sandboxing. -The browser and worker roles are already classified even where their composition remains future -work. +The browser and worker roles are classified, and the worker composition owns the durable +coordinator, database runtime, process signals, and ordered shutdown boundary. ## Application API @@ -389,7 +389,7 @@ once. Reusable procedure builders are limited to: Expected errors use a small stable code set such as `UNAUTHORIZED`, `FORBIDDEN`, `CONFLICT`, `NOT_FOUND`, `PRECONDITION_FAILED`, `TOO_MANY_REQUESTS`, and `SERVICE_UNAVAILABLE` with safe -structured details. The `ContractErrorCode` union, all 68 actual router paths, the server-owned +structured details. The `ContractErrorCode` union, all 77 actual router paths, the server-owned runtime allowlist, and generated contract metadata must match exactly. The base procedure middleware enforces that allowlist for immediate and deferred subscription failures; an implemented procedure missing from the policy or an undeclared code becomes a redacted internal diff --git a/greenfield/docs/architecture/greenfield-rewrite/data-and-security.md b/greenfield/docs/architecture/greenfield-rewrite/data-and-security.md index 85a77d105..7f2b9c1eb 100644 --- a/greenfield/docs/architecture/greenfield-rewrite/data-and-security.md +++ b/greenfield/docs/architecture/greenfield-rewrite/data-and-security.md @@ -24,8 +24,8 @@ write-admission port: it may retry only before `BEGIN IMMEDIATE` admits the transaction and the synchronous callback starts. A callback is never replayed. Exhausted admission and post-admission contention remain typed failures; only mutation routes that declare temporary write unavailability - expose the fixed redacted `SERVICE_UNAVAILABLE` response. The future worker must use the same - explicit policy before that process starts. + expose the fixed redacted `SERVICE_UNAVAILABLE` response. The worker uses that same explicit + admission policy before entering any immediate transaction. - Use Drizzle's typed query builder for ordinary reads/writes and its parameterized `sql` tagged template for SQLite-specific queries, CTEs, queue claims, and expressions not represented cleanly by the builder. @@ -111,7 +111,7 @@ tag. | Tasks/agents | `tasks`, `task_labels`, `task_automation_profiles`, `task_updates`, `task_events`, `agent_task_runs` | | Monitoring | `reports`, `monitor_runs`, `incidents`, `incident_observations`, `notifications` | | Realtime | `realtime_events` | -| Scheduling/work | `scheduled_jobs`, `job_disable_intents`, `job_runs`, `job_run_events`, `worker_instances`, `resource_leases` | +| Scheduling/work | `scheduled_jobs`, `job_disable_intents`, `job_runs`, `job_run_events`, `worker_instances`, `resource_leases`, `job_worker_control` | | Chat | `chat_runs`, `chat_run_events`, `chat_runtime_snapshots` | | Delivery | `deployments`, `deployment_events`, `release_records` | | Docker | `managed_docker_services`, `docker_update_events` | @@ -199,9 +199,9 @@ queryable lifecycle. | Incident identity | unique `incidents(monitor_key, fingerprint)` | | Unread notifications | partial `notifications(occurred_at_ms DESC) WHERE read_at_ms IS NULL` | | Incident notification | unique `(incident_id, incident_generation, channel)` when incident is non-null | -| Queue claim | partial `job_runs(available_at_ms, priority DESC, queued_at_ms) WHERE state = 'queued'` | +| Queue claim | partial `job_runs(available_at, priority DESC, queued_at, id) WHERE state = 'queued'` | | One active scheduled run | unique partial `job_runs(scheduled_job_id) WHERE state IN ('queued', 'running')` | -| Worker expiry | `worker_instances(heartbeat_at_ms)` | +| Worker expiry | `worker_instances(heartbeat_at, id)` | | Job timeline | `job_run_events(job_run_id, sequence)` | | Realtime catch-up | `realtime_events(topic, id)` | | Chat replay | unique `chat_run_events(chat_run_id, sequence)` | @@ -271,9 +271,15 @@ the worker. Queue behavior is explicit: -- a transaction claims one eligible run and assigns a lease; +- a strict singleton `job_worker_control` row persists cross-process claim pause state and + versioned operator changes; its absence is an integrity failure, never an implicit resume; +- one immediate transaction considers at most 32 totally ordered candidates, skips candidates + with occupied resources, and atomically assigns the first eligible run plus every required + resource lease; - each run has an idempotency key, resource class, priority, timeout, attempt limit, and cancellation policy; +- manual-run idempotency is scoped to the requesting principal and hashes stable request intent, + while schedule ticks use a deterministic schedule-and-occurrence namespace; - the worker renews its lease and writes ordered progress events; - expired leases can be recovered only when the action is declared retry-safe; - resource leases prevent conflicting deploy, restore, Docker, or OpenClaw operations; @@ -282,6 +288,16 @@ Queue behavior is explicit: necessary only beneath `/production/state/job-output`; and - final structured output is validated before persistence or display. +Run history is bounded to 1,000 events and 1 MiB of encoded payload per run. The first 967 slots +may carry progress/stdout/stderr payloads; 33 structural slots remain reserved so every legal +ten-attempt lifecycle can record claims, retry decisions, cancellation, truncation, and a terminal +event. Schedule cursor advancement is separate from operator configuration versioning. A due +schedule with an active queued or running occurrence keeps its original cursor; after completion +the scheduler creates one coalesced run for that occurrence and advances directly to the first +future occurrence, never replaying an unbounded backlog. Disabled schedules retain that cursor as +an internal cadence anchor but do not become due until re-enabled; the public summary exposes a +next run only while enabled. Manual runs do not move cadence. + `Bun.spawn` receives argument arrays, a deliberate environment allowlist, an explicit working directory, a timeout, and an abort signal. It never receives interpolated shell text for user input. High-risk jobs run in dedicated transient systemd units or templates with their own diff --git a/greenfield/docs/architecture/greenfield-rewrite/progress.md b/greenfield/docs/architecture/greenfield-rewrite/progress.md index c8493e720..4c81a1915 100644 --- a/greenfield/docs/architecture/greenfield-rewrite/progress.md +++ b/greenfield/docs/architecture/greenfield-rewrite/progress.md @@ -7,15 +7,15 @@ This matrix is the living phase status. Update it in the same change that materially advances or closes a phase; dated entries below provide the evidence, not a second status source. -| Phase | Status | Current evidence and remaining gate | -| ----------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 0 — Evidence and qualification | Complete | All eight mandatory spikes pass on exact Bun revision `17d6843606d76620cb55d31424d7fb0aed51c367`: build, transport, cross-process SQLite/outbox, Drizzle/Bun SQLite, browser data, chat batching, shutdown, and capped resources. Source-derived parity and the OpenClaw source audit pass as additional evidence. | -| 1 — Foundation | Complete | The self-contained future root builds immutable browser/web/worker artifacts, protects project-local production state, installs exact Bun and systemd artifacts, migrates a database copy, atomically promotes the release/database pair, serves readiness/browser assets, writes project-local logs, and proves crash-safe rollback and shutdown in a disposable lifecycle. | -| 2 — Trust and transport | Complete for the stated server scope | Authentication, MFA, WebAuthn, automation credentials, audit, authenticated renewable SSE, one-shot native Gateway bootstrap verification, and the consolidated [threat model](../../security/greenfield-phase-two-threat-model.md) have executable evidence. Browser UI and production cutover remain later gates. | -| 3 — Core operator domains | Started | Task and agent-directory parity are implemented with durable history, realtime invalidation, and browser workflows. Monitoring ingestion plus report, incident, and notification server parity are implemented; report, incident, and global notification browser state are also complete. Schedules/jobs, overview, cache/metrics, and the real worker remain open. | -| 4 — Gateway and chat | Not started | The Phase 2 verifier is one-shot only. Persistent native Gateway lifecycle, current-protocol re-audit, sessions, chat journal/recovery, attachments, and frontend remain open. | -| 5 — Privileged and external domains | Not started | Worker-owned file/media, Docker, database, OpenClaw, GitHub, deployment, backup, and other privileged adapters remain open. | -| 6 — Parity, hardening, and cutover | Not started | Full UI parity, generated `/docs`, load/resource/restore evidence, cutover rehearsal, fresh production database, and legacy removal remain open. | +| Phase | Status | Current evidence and remaining gate | +| ----------------------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 0 — Evidence and qualification | Complete | All eight mandatory spikes pass on exact Bun revision `17d6843606d76620cb55d31424d7fb0aed51c367`: build, transport, cross-process SQLite/outbox, Drizzle/Bun SQLite, browser data, chat batching, shutdown, and capped resources. Source-derived parity and the OpenClaw source audit pass as additional evidence. | +| 1 — Foundation | Complete | The self-contained future root builds immutable browser/web/worker artifacts, protects project-local production state, installs exact Bun and systemd artifacts, migrates a database copy, atomically promotes the release/database pair, serves readiness/browser assets, writes project-local logs, and proves crash-safe rollback and shutdown in a disposable lifecycle. | +| 2 — Trust and transport | Complete for the stated server scope | Authentication, MFA, WebAuthn, automation credentials, audit, authenticated renewable SSE, one-shot native Gateway bootstrap verification, and the consolidated [threat model](../../security/greenfield-phase-two-threat-model.md) have executable evidence. Browser UI and production cutover remain later gates. | +| 3 — Core operator domains | Started | Task and agent-directory parity are implemented with durable history, realtime invalidation, and browser workflows. Monitoring ingestion plus report, incident, and notification server parity are implemented; report, incident, and global notification browser state are also complete. Dashboard-local durable schedules/jobs and real worker execution are implemented. The `/jobs` browser, OpenClaw cron, overview, and cache/metrics remain open. | +| 4 — Gateway and chat | Not started | The Phase 2 verifier is one-shot only. Persistent native Gateway lifecycle, current-protocol re-audit, sessions, chat journal/recovery, attachments, and frontend remain open. | +| 5 — Privileged and external domains | Not started | Worker-owned file/media, Docker, database, OpenClaw, GitHub, deployment, backup, and other privileged adapters remain open. | +| 6 — Parity, hardening, and cutover | Not started | Full UI parity, generated `/docs`, load/resource/restore evidence, cutover rehearsal, fresh production database, and legacy removal remain open. | ### 2026-08-03 — Phase 0 started @@ -916,3 +916,32 @@ full-browser parity, production rehearsal, cutover, and legacy deletion remain o performs a new transport request instead of reviving an empty ready instance. - Notification server and browser parity are now complete. Schedules/jobs, overview, cache/metrics, and real worker execution remain open Phase 3 gates. + +### 2026-08-07 — Phase 3 durable schedules, jobs, and worker foundation + +- Seven strict scheduling tables now persist the reviewed schedule directory, explicit disable + intents, one durable run state machine, bounded ordered events, worker instances, fenced + resource leases, and the singleton cross-process claim-pause control. SQL checks and triggers + protect immutable execution snapshots, legal lifecycle transitions, append-only history, + canonical resource sets, caller-scoped idempotency, event/byte reservations, and optimistic + versions even when writes bypass the service layer. +- Nine `jobs:read`/`jobs:write` procedures expose stable keyset-paginated run and schedule reads, + session-only cancel/pause/update operations, and a caller-scoped idempotent manual-run boundary. + Automation can invoke only registry actions explicitly marked for `jobs:write`; this slice + exposes only the harmless `system.worker-smoke` action. Durable audit rows and compact + `jobs.runs` / `schedules.records` invalidations commit with each externally visible mutation. +- Schedule cadence is distinct from operator configuration versioning. A due schedule with active + work retains its cursor and later coalesces exactly one occurrence; manual runs never move it. + Disabled schedules retain a dormant internal cursor, so expiry or re-enable resumes interval + cadence without drift. Expired intents close under a system actor and re-enable the schedule in + one admitted transaction, while disabling cancels only queued schedule-triggered work. +- The separate Bun worker now owns an Effect-coordinated single-capacity execution loop with + registration/heartbeat, bounded recovery and candidate scans, atomic resource claims, lease + renewal, persisted cooperative cancellation, retry-safe backoff, timeout, bounded progress and + output, fenced settlement, and ordered drain/stop. Unexpected heartbeat, scheduler, claim, or + coordinator completion fails the process rather than leaving a zombie worker. +- A migrated-database system test enqueues the code-owned smoke action through the shared + repository, lets the worker claim it, and observes a durable successful result without shell, + Gateway, or host-mutation authority. The parity inventory marks the nine Dashboard jobs and + schedules operations implemented. The five `openClawCron.*` operations and `/jobs` browser + remain planned, along with overview and cache/metrics. diff --git a/greenfield/docs/architecture/greenfield-rewrite/runtime-and-delivery.md b/greenfield/docs/architecture/greenfield-rewrite/runtime-and-delivery.md index b2225318a..fe31b5fd5 100644 --- a/greenfield/docs/architecture/greenfield-rewrite/runtime-and-delivery.md +++ b/greenfield/docs/architecture/greenfield-rewrite/runtime-and-delivery.md @@ -144,9 +144,9 @@ build path: releases contain prebuilt assets and production never compiles the f ## Configuration From Scratch -The greenfield web configuration parser accepts only its registered-key projection. The future -web and worker composition roots must invoke their role-specific parser exactly once; that startup -wiring is not implemented by this slice. App, server, and worker source has no scattered +The greenfield web configuration parser accepts only its registered-key projection. The web and +worker composition roots invoke their role-specific parser exactly once. App, server, and worker +source has no scattered runtime-environment reads and no truthy-string parsing. Repository scripts are greenfield-owned tools checked by the Bun graph and source-boundary policy; they do not import code outside the self-contained future root. @@ -164,10 +164,10 @@ non-secret settings, and encrypted secrets. A setting is not duplicated across e database with implicit precedence. If bootstrap requires a temporary precedence rule, it is explicitly modeled as a bootstrap state and disappears after completion. -The first web parser currently validates an injected registered-key projection. Its project-root -field is only a lexically normalized absolute staging value: the future process composition must -resolve its real path and enforce the managed-filesystem containment policy before opening host -paths. Startup wiring and that filesystem validation are not claimed by this slice. +Each process composition resolves the configured project root to a real path before deriving and +opening managed production paths. The parser's project-root field remains a lexically normalized +absolute value so parsing itself performs no host I/O; the composition boundary owns the stronger +filesystem identity and containment checks. The target repository has exactly three TypeScript configuration files. `tsconfig.json` owns all shared strict compiler options, has `files: []`, and references only `tsconfig.browser.json` and diff --git a/greenfield/docs/generated/procedures.md b/greenfield/docs/generated/procedures.md index 8f94cae31..726b1b46e 100644 --- a/greenfield/docs/generated/procedures.md +++ b/greenfield/docs/generated/procedures.md @@ -45,9 +45,13 @@ | `automationSecurity.replaceCapabilities` | mutation | automation-security | MFA enrollment required; recent MFA when enabled | [input](./schemas/automationSecurity.replaceCapabilities.input.schema.json) | [output](./schemas/automationSecurity.replaceCapabilities.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | `mfa_enrollment_required`, `step_up_required` | Atomically replaces a principal's least-privilege capability set. | | `automationSecurity.revokeCredential` | mutation | automation-security | MFA enrollment required; recent MFA when enabled | [input](./schemas/automationSecurity.revokeCredential.input.schema.json) | [output](./schemas/automationSecurity.revokeCredential.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | `mfa_enrollment_required`, `step_up_required` | Explicitly revokes one automation credential after client cutover. | | `automationSecurity.rotateCredential` | mutation | automation-security | MFA enrollment required; recent MFA when enabled | [input](./schemas/automationSecurity.rotateCredential.input.schema.json) | [output](./schemas/automationSecurity.rotateCredential.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `PRECONDITION_FAILED`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | `mfa_enrollment_required`, `step_up_required` | Stages a linked replacement credential without revoking its predecessor. | -| `events.stream` | subscription | events | Authenticated; per-topic: agents:read, notifications:read, reports:read, tasks:read | [input](./schemas/events.stream.input.schema.json) | [output](./schemas/events.stream.output.schema.json) | `BAD_REQUEST`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `TOO_MANY_REQUESTS`, `UNAUTHORIZED` | None | Streams authorized durable changes with tracked resume cursors. | +| `events.stream` | subscription | events | Authenticated; per-topic: agents:read, jobs:read, notifications:read, reports:read, tasks:read | [input](./schemas/events.stream.input.schema.json) | [output](./schemas/events.stream.output.schema.json) | `BAD_REQUEST`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `TOO_MANY_REQUESTS`, `UNAUTHORIZED` | None | Streams authorized durable changes with tracked resume cursors. | | `incidents.get` | query | incidents | Authenticated: reports:read | [input](./schemas/incidents.get.input.schema.json) | [output](./schemas/incidents.get.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Loads one exact incident lifecycle record. | | `incidents.list` | query | incidents | Authenticated: reports:read | [input](./schemas/incidents.list.input.schema.json) | [output](./schemas/incidents.list.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists stable incident lifecycle rows for report navigation. | +| `jobs.cancelRun` | mutation | jobs | Authenticated browser session: jobs:write | [input](./schemas/jobs.cancelRun.input.schema.json) | [output](./schemas/jobs.cancelRun.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Cancels a queued run or requests cooperative running cancellation. | +| `jobs.getRun` | query | jobs | Authenticated: jobs:read | [input](./schemas/jobs.getRun.input.schema.json) | [output](./schemas/jobs.getRun.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Loads one durable run with bounded newest-first events. | +| `jobs.listRuns` | query | jobs | Authenticated: jobs:read | [input](./schemas/jobs.listRuns.input.schema.json) | [output](./schemas/jobs.listRuns.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists stable newest-first durable run history and queue state. | +| `jobs.setClaimingPaused` | mutation | jobs | Authenticated browser session: jobs:write | [input](./schemas/jobs.setClaimingPaused.input.schema.json) | [output](./schemas/jobs.setClaimingPaused.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Pauses or resumes new cross-process claims under version control. | | `monitoring.submitCompleteSnapshot` | mutation | monitoring | Authenticated automation principal: monitoring:write | [input](./schemas/monitoring.submitCompleteSnapshot.input.schema.json) | [output](./schemas/monitoring.submitCompleteSnapshot.output.schema.json) | `BAD_REQUEST`, `CONFLICT`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Atomically ingests one complete monitor snapshot. | | `notifications.clearRead` | mutation | notifications | Authenticated browser session: notifications:write | [input](./schemas/notifications.clearRead.input.schema.json) | [output](./schemas/notifications.clearRead.output.schema.json) | `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Deletes one bounded page of matching read notifications. | | `notifications.delete` | mutation | notifications | Authenticated browser session: notifications:write | [input](./schemas/notifications.delete.input.schema.json) | [output](./schemas/notifications.delete.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Deletes one exact Dashboard notification. | @@ -59,6 +63,11 @@ | `reports.get` | query | reports | Authenticated: reports:read | [input](./schemas/reports.get.input.schema.json) | [output](./schemas/reports.get.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Loads one complete immutable Markdown report. | | `reports.list` | query | reports | Authenticated: reports:read | [input](./schemas/reports.list.input.schema.json) | [output](./schemas/reports.list.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists a stable filtered page of immutable report summaries. | | `reports.upsert` | mutation | reports | Authenticated: reports:write | [input](./schemas/reports.upsert.input.schema.json) | [output](./schemas/reports.upsert.output.schema.json) | `BAD_REQUEST`, `CONFLICT`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Creates a report or accepts an exact idempotent replay. | +| `schedules.get` | query | schedules | Authenticated: jobs:read | [input](./schemas/schedules.get.input.schema.json) | [output](./schemas/schedules.get.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Loads one code-owned schedule and its latest durable run state. | +| `schedules.list` | query | schedules | Authenticated: jobs:read | [input](./schemas/schedules.list.input.schema.json) | [output](./schemas/schedules.list.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists the stable code-owned Dashboard schedule directory. | +| `schedules.listRuns` | query | schedules | Authenticated: jobs:read | [input](./schemas/schedules.listRuns.input.schema.json) | [output](./schemas/schedules.listRuns.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Lists stable newest-first durable history for one schedule. | +| `schedules.run` | mutation | schedules | Authenticated: jobs:write | [input](./schemas/schedules.run.input.schema.json) | [output](./schemas/schedules.run.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Enqueues one caller-scoped idempotent manual schedule run. | +| `schedules.update` | mutation | schedules | Authenticated browser session: jobs:write | [input](./schemas/schedules.update.input.schema.json) | [output](./schemas/schedules.update.output.schema.json) | `BAD_REQUEST`, `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Updates one schedule or its explicit disable intent by version. | | `securityAudit.listEvents` | query | securityAudit | Authenticated browser session | [input](./schemas/securityAudit.listEvents.input.schema.json) | [output](./schemas/securityAudit.listEvents.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists redacted immutable security events in stable newest-first order. | | `system.runtimeIdentity` | query | system | Public | [input](./schemas/system.runtimeIdentity.input.schema.json) | [output](./schemas/system.runtimeIdentity.output.schema.json) | None | None | Returns the Bun runtime identity of the serving process. | | `tasks.addUpdate` | mutation | tasks | Authenticated: tasks:write | [input](./schemas/tasks.addUpdate.input.schema.json) | [output](./schemas/tasks.addUpdate.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Appends one authenticated progress update to a task. | diff --git a/greenfield/docs/generated/realtime-events.md b/greenfield/docs/generated/realtime-events.md index f46503a5d..4d992ba64 100644 --- a/greenfield/docs/generated/realtime-events.md +++ b/greenfield/docs/generated/realtime-events.md @@ -5,7 +5,9 @@ | Topic | Payload | Snapshot | Retention | Summary | | --- | --- | --- | --- | --- | | `agents.status` | [payload](./schemas/agents.status.realtime.payload.schema.json) | `agents.listStatuses` | 7 days | Invalidates one agent status row after a durable metadata change. | +| `jobs.runs` | [payload](./schemas/jobs.runs.realtime.payload.schema.json) | `jobs.listRuns` | 7 days | Invalidates durable run rows and exact queue state. | | `monitoring.incidents` | [payload](./schemas/monitoring.incidents.realtime.payload.schema.json) | `incidents.list` | 7 days | Invalidates incident lifecycle rows after a complete monitor snapshot. | | `monitoring.notifications` | [payload](./schemas/monitoring.notifications.realtime.payload.schema.json) | `notifications.list` | 7 days | Invalidates Dashboard notifications after catalog changes. | | `monitoring.reports` | [payload](./schemas/monitoring.reports.realtime.payload.schema.json) | `reports.list` | 7 days | Invalidates immutable reports after catalog changes. | +| `schedules.records` | [payload](./schemas/schedules.records.realtime.payload.schema.json) | `schedules.list` | 7 days | Invalidates the code-owned Dashboard schedule directory. | | `tasks.records` | [payload](./schemas/tasks.records.realtime.payload.schema.json) | `tasks.list` | 7 days | Invalidates one task row after a durable task-domain change. | diff --git a/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.input.schema.json b/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.input.schema.json index a0d4a5672..d4f283b46 100644 --- a/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.input.schema.json +++ b/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.input.schema.json @@ -8,6 +8,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -18,7 +20,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "id": { diff --git a/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.output.schema.json b/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.output.schema.json index 2975797d2..0a524675c 100644 --- a/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.output.schema.json +++ b/greenfield/docs/generated/schemas/automationSecurity.createPrincipal.output.schema.json @@ -77,6 +77,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -87,7 +89,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { @@ -153,6 +155,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -163,7 +167,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { diff --git a/greenfield/docs/generated/schemas/automationSecurity.disablePrincipal.output.schema.json b/greenfield/docs/generated/schemas/automationSecurity.disablePrincipal.output.schema.json index 80fa539de..498ff076f 100644 --- a/greenfield/docs/generated/schemas/automationSecurity.disablePrincipal.output.schema.json +++ b/greenfield/docs/generated/schemas/automationSecurity.disablePrincipal.output.schema.json @@ -26,6 +26,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -36,7 +38,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { @@ -102,6 +104,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -112,7 +116,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { diff --git a/greenfield/docs/generated/schemas/automationSecurity.listPrincipals.output.schema.json b/greenfield/docs/generated/schemas/automationSecurity.listPrincipals.output.schema.json index 5d155f62a..3828f1bbd 100644 --- a/greenfield/docs/generated/schemas/automationSecurity.listPrincipals.output.schema.json +++ b/greenfield/docs/generated/schemas/automationSecurity.listPrincipals.output.schema.json @@ -51,6 +51,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -61,7 +63,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { @@ -127,6 +129,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -137,7 +141,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { diff --git a/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.input.schema.json b/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.input.schema.json index 2622ef2e1..378257593 100644 --- a/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.input.schema.json +++ b/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.input.schema.json @@ -19,6 +19,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -29,7 +31,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true } }, diff --git a/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.output.schema.json b/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.output.schema.json index 5e96ab396..dccc48204 100644 --- a/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.output.schema.json +++ b/greenfield/docs/generated/schemas/automationSecurity.replaceCapabilities.output.schema.json @@ -26,6 +26,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -36,7 +38,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { @@ -102,6 +104,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -112,7 +116,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "createdAtMs": { diff --git a/greenfield/docs/generated/schemas/events.stream.input.schema.json b/greenfield/docs/generated/schemas/events.stream.input.schema.json index 4eddcfedc..76d3aa2cc 100644 --- a/greenfield/docs/generated/schemas/events.stream.input.schema.json +++ b/greenfield/docs/generated/schemas/events.stream.input.schema.json @@ -13,6 +13,8 @@ "items": { "enum": [ "agents.status", + "jobs.runs", + "schedules.records", "monitoring.incidents", "monitoring.notifications", "monitoring.reports", diff --git a/greenfield/docs/generated/schemas/events.stream.output.schema.json b/greenfield/docs/generated/schemas/events.stream.output.schema.json index d561c4dc1..e4ef55a8b 100644 --- a/greenfield/docs/generated/schemas/events.stream.output.schema.json +++ b/greenfield/docs/generated/schemas/events.stream.output.schema.json @@ -8,7 +8,7 @@ "type": "object", "properties": { "event": { - "oneOf": [ + "anyOf": [ { "type": "object", "properties": { @@ -61,6 +61,201 @@ ], "additionalProperties": false }, + { + "type": "object", + "properties": { + "entityId": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "entityType": { + "const": "job-run" + }, + "occurredAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "operation": { + "enum": [ + "created", + "updated" + ], + "type": "string" + }, + "payload": { + "type": "object", + "properties": { + "id": { + "anyOf": [ + { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + ] + } + }, + "required": [ + "id" + ], + "additionalProperties": false + }, + "topic": { + "const": "jobs.runs" + } + }, + "required": [ + "entityId", + "entityType", + "occurredAtMs", + "operation", + "payload", + "topic" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires the realtime entity and compact payload IDs to match exactly." + }, + { + "type": "object", + "properties": { + "entityId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "entityType": { + "const": "job-queue" + }, + "occurredAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "operation": { + "enum": [ + "snapshot-required" + ], + "type": "string" + }, + "payload": { + "type": "object", + "properties": { + "id": { + "anyOf": [ + { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + ] + } + }, + "required": [ + "id" + ], + "additionalProperties": false + }, + "topic": { + "const": "jobs.runs" + } + }, + "required": [ + "entityId", + "entityType", + "occurredAtMs", + "operation", + "payload", + "topic" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires the realtime entity and compact payload IDs to match exactly." + }, + { + "type": "object", + "properties": { + "entityId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "entityType": { + "const": "schedule" + }, + "occurredAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "operation": { + "enum": [ + "created", + "updated" + ], + "type": "string" + }, + "payload": { + "type": "object", + "properties": { + "id": { + "anyOf": [ + { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + ] + } + }, + "required": [ + "id" + ], + "additionalProperties": false + }, + "topic": { + "const": "schedules.records" + } + }, + "required": [ + "entityId", + "entityType", + "occurredAtMs", + "operation", + "payload", + "topic" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires the realtime entity and compact payload IDs to match exactly." + }, { "type": "object", "properties": { diff --git a/greenfield/docs/generated/schemas/jobs.cancelRun.input.schema.json b/greenfield/docs/generated/schemas/jobs.cancelRun.input.schema.json new file mode 100644 index 000000000..b6651138e --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.cancelRun.input.schema.json @@ -0,0 +1,18 @@ +{ + "$id": "urn:mira-dashboard:jobs.cancelRun.input", + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + } + }, + "required": [ + "id" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.cancelRun.output.schema.json b/greenfield/docs/generated/schemas/jobs.cancelRun.output.schema.json new file mode 100644 index 000000000..2c6b456db --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.cancelRun.output.schema.json @@ -0,0 +1,209 @@ +{ + "$id": "urn:mira-dashboard:jobs.cancelRun.output", + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.getRun.input.schema.json b/greenfield/docs/generated/schemas/jobs.getRun.input.schema.json new file mode 100644 index 000000000..e69289e99 --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.getRun.input.schema.json @@ -0,0 +1,38 @@ +{ + "$id": "urn:mira-dashboard:jobs.getRun.input", + "type": "object", + "properties": { + "eventCursor": { + "type": "object", + "properties": { + "sequence": { + "type": "integer", + "minimum": 1, + "maximum": 1000 + } + }, + "required": [ + "sequence" + ], + "additionalProperties": false + }, + "eventLimit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "default": 50 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + } + }, + "required": [ + "id" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.getRun.output.schema.json b/greenfield/docs/generated/schemas/jobs.getRun.output.schema.json new file mode 100644 index 000000000..2d36910ed --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.getRun.output.schema.json @@ -0,0 +1,315 @@ +{ + "$id": "urn:mira-dashboard:jobs.getRun.output", + "type": "object", + "properties": { + "events": { + "type": "array", + "items": { + "type": "object", + "properties": { + "attempt": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "kind": { + "enum": [ + "cancel-requested", + "cancelled", + "claimed", + "failed", + "lease-expired", + "output-truncated", + "progress", + "queued", + "retry-scheduled", + "stderr", + "stdout", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "message": { + "type": "string", + "maxLength": 4096, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ], + "$comment": "Live Valibot validation additionally limits the job-event message to its reviewed UTF-8 byte budget." + }, + "occurredAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "progress": { + "type": "object", + "$comment": "Live Valibot validation additionally requires an acyclic plain JSON object with bounded depth, finite safe-magnitude numbers, and no sparse arrays. Live Valibot validation additionally limits serialized job progress to its reviewed UTF-8 byte budget." + }, + "sequence": { + "type": "integer", + "minimum": 1, + "maximum": 1000 + }, + "workerInstanceId": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + } + }, + "required": [ + "attempt", + "kind", + "occurredAtMs", + "sequence" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires job-event payload fields to agree with the event kind." + }, + "maxItems": 100, + "$comment": "Live Valibot validation additionally requires strict newest-first job-event sequence order." + }, + "nextEventCursor": { + "type": "object", + "properties": { + "sequence": { + "type": "integer", + "minimum": 1, + "maximum": 1000 + } + }, + "required": [ + "sequence" + ], + "additionalProperties": false + }, + "result": { + "type": "object", + "$comment": "Live Valibot validation additionally requires an acyclic plain JSON object with bounded depth, finite safe-magnitude numbers, and no sparse arrays. Live Valibot validation additionally limits the serialized job result to its reviewed UTF-8 byte budget." + }, + "run": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + } + }, + "required": [ + "events", + "run" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally binds job result, events, cursor, and run state.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.listRuns.input.schema.json b/greenfield/docs/generated/schemas/jobs.listRuns.input.schema.json new file mode 100644 index 000000000..0ff474662 --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.listRuns.input.schema.json @@ -0,0 +1,98 @@ +{ + "$id": "urn:mira-dashboard:jobs.listRuns.input", + "type": "object", + "properties": { + "cursor": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "id", + "queuedAtMs" + ], + "additionalProperties": false + }, + "filters": { + "type": "object", + "properties": { + "resourceClasses": { + "type": "array", + "items": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "minItems": 1, + "maxItems": 16, + "uniqueItems": true + }, + "scheduleId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "states": { + "type": "array", + "items": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "minItems": 1, + "maxItems": 16, + "uniqueItems": true + }, + "triggerTypes": { + "type": "array", + "items": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "minItems": 1, + "maxItems": 16, + "uniqueItems": true + } + }, + "required": [], + "additionalProperties": false + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "default": 50 + } + }, + "required": [], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.listRuns.output.schema.json b/greenfield/docs/generated/schemas/jobs.listRuns.output.schema.json new file mode 100644 index 000000000..e689be1a1 --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.listRuns.output.schema.json @@ -0,0 +1,418 @@ +{ + "$id": "urn:mira-dashboard:jobs.listRuns.output", + "type": "object", + "properties": { + "nextCursor": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "id", + "queuedAtMs" + ], + "additionalProperties": false + }, + "runs": { + "type": "array", + "items": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "maxItems": 100, + "$comment": "Live Valibot validation additionally requires strict newest-first job-run ordering by queue timestamp and ID." + }, + "summary": { + "type": "object", + "properties": { + "activeResourceClasses": { + "type": "array", + "items": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "maxItems": 5, + "$comment": "Live Valibot validation additionally requires active resource classes in canonical unique order." + }, + "control": { + "type": "object", + "properties": { + "claimingPaused": { + "type": "boolean" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "version": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "required": [ + "claimingPaused", + "updatedAtMs", + "version" + ], + "additionalProperties": false + }, + "oldestQueuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "stateCounts": { + "type": "object", + "properties": { + "cancelled": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "failed": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "queued": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "running": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "succeeded": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "timed-out": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + } + }, + "required": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "additionalProperties": false + }, + "workers": { + "type": "array", + "items": { + "type": "object", + "properties": { + "activeRunCount": { + "type": "integer", + "minimum": 0, + "maximum": 16 + }, + "capacity": { + "type": "integer", + "minimum": 1, + "maximum": 16 + }, + "drainingAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "heartbeatAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "releaseId": { + "type": "string", + "pattern": "^[0-9a-f]{40}$" + }, + "startedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "state": { + "enum": [ + "draining", + "online", + "stopped" + ], + "type": "string" + }, + "stoppedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "activeRunCount", + "capacity", + "heartbeatAtMs", + "id", + "releaseId", + "startedAtMs", + "state" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires worker capacity and lifecycle timestamps to agree." + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires unique workers in canonical ID order." + } + }, + "required": [ + "activeResourceClasses", + "control", + "stateCounts", + "workers" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally binds queue-derived fields to their exact state counts." + } + }, + "required": [ + "runs", + "summary" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires a job-run cursor to identify the returned last row.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.runs.realtime.payload.schema.json b/greenfield/docs/generated/schemas/jobs.runs.realtime.payload.schema.json new file mode 100644 index 000000000..4e6ae5a26 --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.runs.realtime.payload.schema.json @@ -0,0 +1,28 @@ +{ + "$id": "urn:mira-dashboard:jobs.runs.realtime.payload", + "type": "object", + "properties": { + "id": { + "anyOf": [ + { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + ] + } + }, + "required": [ + "id" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.setClaimingPaused.input.schema.json b/greenfield/docs/generated/schemas/jobs.setClaimingPaused.input.schema.json new file mode 100644 index 000000000..196b7542b --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.setClaimingPaused.input.schema.json @@ -0,0 +1,20 @@ +{ + "$id": "urn:mira-dashboard:jobs.setClaimingPaused.input", + "type": "object", + "properties": { + "expectedVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "paused": { + "type": "boolean" + } + }, + "required": [ + "expectedVersion", + "paused" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/jobs.setClaimingPaused.output.schema.json b/greenfield/docs/generated/schemas/jobs.setClaimingPaused.output.schema.json new file mode 100644 index 000000000..f1a60a5ac --- /dev/null +++ b/greenfield/docs/generated/schemas/jobs.setClaimingPaused.output.schema.json @@ -0,0 +1,26 @@ +{ + "$id": "urn:mira-dashboard:jobs.setClaimingPaused.output", + "type": "object", + "properties": { + "claimingPaused": { + "type": "boolean" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "version": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "required": [ + "claimingPaused", + "updatedAtMs", + "version" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.get.input.schema.json b/greenfield/docs/generated/schemas/schedules.get.input.schema.json new file mode 100644 index 000000000..a7353a78d --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.get.input.schema.json @@ -0,0 +1,17 @@ +{ + "$id": "urn:mira-dashboard:schedules.get.input", + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + }, + "required": [ + "id" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.get.output.schema.json b/greenfield/docs/generated/schemas/schedules.get.output.schema.json new file mode 100644 index 000000000..84dde5114 --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.get.output.schema.json @@ -0,0 +1,667 @@ +{ + "$id": "urn:mira-dashboard:schedules.get.output", + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "activeDisableIntent": { + "type": "object", + "properties": { + "createdAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "expiresAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "reason": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + } + }, + "required": [ + "createdAtMs", + "id", + "reason" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires disable-intent expiry after creation." + }, + "activeRun": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "createdAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "description": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "enabled": { + "type": "boolean" + }, + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "latestRun": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "name": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "nextRunAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "schedule": { + "oneOf": [ + { + "type": "object", + "properties": { + "expression": { + "type": "string", + "maxLength": 200, + "description": "Five-field minute cron; live validation accepts JAN-DEC month and SUN-SAT weekday aliases, normalizes aliases and ASCII whitespace, and requires a future occurrence.", + "minLength": 9, + "$comment": "Live Valibot validation additionally requires a valid five-field minute cron with a future occurrence." + }, + "kind": { + "const": "cron" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "expression", + "kind", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "kind": { + "const": "daily" + }, + "timeOfDay": { + "type": "string", + "pattern": "^(?:[01]\\d|2[0-3]):[0-5]\\d$" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "kind", + "timeOfDay", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "intervalMs": { + "type": "integer", + "minimum": 60000, + "maximum": 31536000000 + }, + "kind": { + "const": "interval" + } + }, + "required": [ + "intervalMs", + "kind" + ], + "additionalProperties": false + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "version": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "required": [ + "actionKey", + "attemptLimit", + "cancellationPolicy", + "createdAtMs", + "description", + "enabled", + "id", + "name", + "priority", + "resourceClass", + "resourceKeys", + "retrySafe", + "schedule", + "timeoutMs", + "updatedAtMs", + "version" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally binds schedule state to its cursor, disable intent, and embedded runs.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.list.input.schema.json b/greenfield/docs/generated/schemas/schedules.list.input.schema.json new file mode 100644 index 000000000..2446bd88d --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.list.input.schema.json @@ -0,0 +1,39 @@ +{ + "$id": "urn:mira-dashboard:schedules.list.input", + "type": "object", + "properties": { + "cursor": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + }, + "required": [ + "id" + ], + "additionalProperties": false + }, + "enabled": { + "enum": [ + "all", + "disabled", + "enabled" + ], + "type": "string", + "default": "all" + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "default": 50 + } + }, + "required": [], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.list.output.schema.json b/greenfield/docs/generated/schemas/schedules.list.output.schema.json new file mode 100644 index 000000000..35ca82a94 --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.list.output.schema.json @@ -0,0 +1,697 @@ +{ + "$id": "urn:mira-dashboard:schedules.list.output", + "type": "object", + "properties": { + "nextCursor": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + }, + "required": [ + "id" + ], + "additionalProperties": false + }, + "schedules": { + "type": "array", + "items": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "activeDisableIntent": { + "type": "object", + "properties": { + "createdAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "expiresAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "reason": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + } + }, + "required": [ + "createdAtMs", + "id", + "reason" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires disable-intent expiry after creation." + }, + "activeRun": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "createdAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "description": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "enabled": { + "type": "boolean" + }, + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "latestRun": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "name": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "nextRunAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "schedule": { + "oneOf": [ + { + "type": "object", + "properties": { + "expression": { + "type": "string", + "maxLength": 200, + "description": "Five-field minute cron; live validation accepts JAN-DEC month and SUN-SAT weekday aliases, normalizes aliases and ASCII whitespace, and requires a future occurrence.", + "minLength": 9, + "$comment": "Live Valibot validation additionally requires a valid five-field minute cron with a future occurrence." + }, + "kind": { + "const": "cron" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "expression", + "kind", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "kind": { + "const": "daily" + }, + "timeOfDay": { + "type": "string", + "pattern": "^(?:[01]\\d|2[0-3]):[0-5]\\d$" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "kind", + "timeOfDay", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "intervalMs": { + "type": "integer", + "minimum": 60000, + "maximum": 31536000000 + }, + "kind": { + "const": "interval" + } + }, + "required": [ + "intervalMs", + "kind" + ], + "additionalProperties": false + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "version": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "required": [ + "actionKey", + "attemptLimit", + "cancellationPolicy", + "createdAtMs", + "description", + "enabled", + "id", + "name", + "priority", + "resourceClass", + "resourceKeys", + "retrySafe", + "schedule", + "timeoutMs", + "updatedAtMs", + "version" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally binds schedule state to its cursor, disable intent, and embedded runs." + }, + "maxItems": 100, + "$comment": "Live Valibot validation additionally requires strict ascending schedule ID order." + } + }, + "required": [ + "schedules" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires a schedule cursor to identify the returned last row.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.listRuns.input.schema.json b/greenfield/docs/generated/schemas/schedules.listRuns.input.schema.json new file mode 100644 index 000000000..de18a7432 --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.listRuns.input.schema.json @@ -0,0 +1,45 @@ +{ + "$id": "urn:mira-dashboard:schedules.listRuns.input", + "type": "object", + "properties": { + "cursor": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "id", + "queuedAtMs" + ], + "additionalProperties": false + }, + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "default": 50 + } + }, + "required": [ + "id" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.listRuns.output.schema.json b/greenfield/docs/generated/schemas/schedules.listRuns.output.schema.json new file mode 100644 index 000000000..be25eeffa --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.listRuns.output.schema.json @@ -0,0 +1,246 @@ +{ + "$id": "urn:mira-dashboard:schedules.listRuns.output", + "type": "object", + "properties": { + "nextCursor": { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "id", + "queuedAtMs" + ], + "additionalProperties": false + }, + "runs": { + "type": "array", + "items": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "maxItems": 100, + "$comment": "Live Valibot validation additionally requires strict newest-first job-run ordering by queue timestamp and ID." + } + }, + "required": [ + "runs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires a schedule-run cursor to identify the returned last row.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.records.realtime.payload.schema.json b/greenfield/docs/generated/schemas/schedules.records.realtime.payload.schema.json new file mode 100644 index 000000000..cdd8b475c --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.records.realtime.payload.schema.json @@ -0,0 +1,28 @@ +{ + "$id": "urn:mira-dashboard:schedules.records.realtime.payload", + "type": "object", + "properties": { + "id": { + "anyOf": [ + { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + } + ] + } + }, + "required": [ + "id" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.run.input.schema.json b/greenfield/docs/generated/schemas/schedules.run.input.schema.json new file mode 100644 index 000000000..6f3b1bc31 --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.run.input.schema.json @@ -0,0 +1,24 @@ +{ + "$id": "urn:mira-dashboard:schedules.run.input", + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "idempotencyKey": { + "type": "string", + "minLength": 32, + "maxLength": 128, + "pattern": "^(?:[A-Za-z0-9_-]{4})*(?:[A-Za-z0-9_-][AQgw]|[A-Za-z0-9_-]{2}[AEIMQUYcgkosw048])?$" + } + }, + "required": [ + "id", + "idempotencyKey" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.run.output.schema.json b/greenfield/docs/generated/schemas/schedules.run.output.schema.json new file mode 100644 index 000000000..c26dfebeb --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.run.output.schema.json @@ -0,0 +1,209 @@ +{ + "$id": "urn:mira-dashboard:schedules.run.output", + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.update.input.schema.json b/greenfield/docs/generated/schemas/schedules.update.input.schema.json new file mode 100644 index 000000000..e78bf063c --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.update.input.schema.json @@ -0,0 +1,141 @@ +{ + "$id": "urn:mira-dashboard:schedules.update.input", + "type": "object", + "properties": { + "expectedVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "patch": { + "type": "object", + "properties": { + "disableIntent": { + "anyOf": [ + { + "type": "object", + "properties": { + "expiresAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "reason": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + } + }, + "required": [ + "reason" + ], + "additionalProperties": false + }, + { + "type": "null" + } + ] + }, + "enabled": { + "type": "boolean" + }, + "schedule": { + "oneOf": [ + { + "type": "object", + "properties": { + "expression": { + "type": "string", + "maxLength": 400, + "description": "Five-field minute cron; live validation accepts JAN-DEC month and SUN-SAT weekday aliases, normalizes aliases and ASCII whitespace, and requires a future occurrence." + }, + "kind": { + "const": "cron" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "expression", + "kind", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "kind": { + "const": "daily" + }, + "timeOfDay": { + "type": "string", + "pattern": "^(?:[01]\\d|2[0-3]):[0-5]\\d$" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "kind", + "timeOfDay", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "intervalMs": { + "type": "integer", + "minimum": 60000, + "maximum": 31536000000 + }, + "kind": { + "const": "interval" + } + }, + "required": [ + "intervalMs", + "kind" + ], + "additionalProperties": false + } + ] + } + }, + "required": [], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires a non-empty schedule patch with an explicit disable transition." + } + }, + "required": [ + "expectedVersion", + "id", + "patch" + ], + "additionalProperties": false, + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/schedules.update.output.schema.json b/greenfield/docs/generated/schemas/schedules.update.output.schema.json new file mode 100644 index 000000000..f54f89886 --- /dev/null +++ b/greenfield/docs/generated/schemas/schedules.update.output.schema.json @@ -0,0 +1,667 @@ +{ + "$id": "urn:mira-dashboard:schedules.update.output", + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "activeDisableIntent": { + "type": "object", + "properties": { + "createdAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "expiresAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "reason": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + } + }, + "required": [ + "createdAtMs", + "id", + "reason" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires disable-intent expiry after creation." + }, + "activeRun": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "createdAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "description": { + "type": "string", + "maxLength": 1000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "enabled": { + "type": "boolean" + }, + "id": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "latestRun": { + "type": "object", + "properties": { + "actionKey": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "attemptCount": { + "type": "integer", + "minimum": 0, + "maximum": 10 + }, + "attemptLimit": { + "type": "integer", + "minimum": 1, + "maximum": 10 + }, + "availableAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "cancellationPolicy": { + "enum": [ + "cooperative", + "never", + "queued-only" + ], + "type": "string" + }, + "cancelRequestedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "displayName": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "eventCount": { + "type": "integer", + "minimum": 0, + "maximum": 1000 + }, + "finishedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "firstStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "id": { + "type": "string", + "minLength": 36, + "maxLength": 36, + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" + }, + "lastAttemptStartedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "queuedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "scheduledForAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "scheduledJobId": { + "type": "string", + "minLength": 1, + "maxLength": 80, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "scheduledJobVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "state": { + "enum": [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out" + ], + "type": "string" + }, + "stateVersion": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + }, + "terminalCode": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._/-]*$" + }, + "terminalMessage": { + "type": "string", + "maxLength": 2000, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "triggerType": { + "enum": [ + "manual", + "schedule", + "startup", + "system" + ], + "type": "string" + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + } + }, + "required": [ + "actionKey", + "attemptCount", + "attemptLimit", + "availableAtMs", + "cancellationPolicy", + "displayName", + "eventCount", + "id", + "priority", + "queuedAtMs", + "resourceClass", + "resourceKeys", + "retrySafe", + "state", + "stateVersion", + "timeoutMs", + "triggerType", + "updatedAtMs" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree." + }, + "name": { + "type": "string", + "maxLength": 160, + "minLength": 1, + "pattern": "\\S", + "allOf": [ + { + "pattern": "^[^\\u0000]*$" + }, + { + "pattern": "^(?![\\s\\S]*(?:[\\u0000-\\u001F\\u007F-\\u009F\\u00AD\\u0600-\\u0605\\u061C\\u06DD\\u070F\\u0890-\\u0891\\u08E2\\u180E\\u200B-\\u200F\\u202A-\\u202E\\u2060-\\u2064\\u2066-\\u206F\\uFEFF\\uFFF9-\\uFFFB]|\\uD804[\\uDCBD\\uDCCD]|\\uD80D[\\uDC30-\\uDC3F]|\\uD82F[\\uDCA0-\\uDCA3]|\\uD834[\\uDD73-\\uDD7A]|\\uDB40(?:\\uDC01|[\\uDC20-\\uDC7F])))[\\s\\S]*$" + } + ] + }, + "nextRunAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "priority": { + "type": "integer", + "minimum": -100, + "maximum": 100 + }, + "resourceClass": { + "enum": [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network" + ], + "type": "string" + }, + "resourceKeys": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "maxLength": 128, + "pattern": "^[a-z0-9][a-z0-9._-]*$" + }, + "maxItems": 32, + "$comment": "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget." + }, + "retrySafe": { + "type": "boolean" + }, + "schedule": { + "oneOf": [ + { + "type": "object", + "properties": { + "expression": { + "type": "string", + "maxLength": 200, + "description": "Five-field minute cron; live validation accepts JAN-DEC month and SUN-SAT weekday aliases, normalizes aliases and ASCII whitespace, and requires a future occurrence.", + "minLength": 9, + "$comment": "Live Valibot validation additionally requires a valid five-field minute cron with a future occurrence." + }, + "kind": { + "const": "cron" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "expression", + "kind", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "kind": { + "const": "daily" + }, + "timeOfDay": { + "type": "string", + "pattern": "^(?:[01]\\d|2[0-3]):[0-5]\\d$" + }, + "timeZone": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "$comment": "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier." + } + }, + "required": [ + "kind", + "timeOfDay", + "timeZone" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "intervalMs": { + "type": "integer", + "minimum": 60000, + "maximum": 31536000000 + }, + "kind": { + "const": "interval" + } + }, + "required": [ + "intervalMs", + "kind" + ], + "additionalProperties": false + } + ] + }, + "timeoutMs": { + "type": "integer", + "minimum": 1000, + "maximum": 86400000 + }, + "updatedAtMs": { + "type": "integer", + "minimum": 0, + "maximum": 8640000000000000 + }, + "version": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991 + } + }, + "required": [ + "actionKey", + "attemptLimit", + "cancellationPolicy", + "createdAtMs", + "description", + "enabled", + "id", + "name", + "priority", + "resourceClass", + "resourceKeys", + "retrySafe", + "schedule", + "timeoutMs", + "updatedAtMs", + "version" + ], + "additionalProperties": false, + "$comment": "Live Valibot validation additionally binds schedule state to its cursor, disable intent, and embedded runs.", + "$schema": "https://json-schema.org/draft/2020-12/schema" +} diff --git a/greenfield/docs/generated/schemas/securityAudit.listEvents.output.schema.json b/greenfield/docs/generated/schemas/securityAudit.listEvents.output.schema.json index 59764a04f..191814cca 100644 --- a/greenfield/docs/generated/schemas/securityAudit.listEvents.output.schema.json +++ b/greenfield/docs/generated/schemas/securityAudit.listEvents.output.schema.json @@ -135,6 +135,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -145,7 +147,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "method": { @@ -190,6 +192,8 @@ "enum": [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -200,7 +204,7 @@ ], "type": "string" }, - "maxItems": 9, + "maxItems": 11, "uniqueItems": true }, "replacementCredentialId": { diff --git a/greenfield/migrations/20260804022252_dashboard-foundation/migration.sql b/greenfield/migrations/20260804022252_dashboard-foundation/migration.sql index a22601f75..9eca51019 100644 --- a/greenfield/migrations/20260804022252_dashboard-foundation/migration.sql +++ b/greenfield/migrations/20260804022252_dashboard-foundation/migration.sql @@ -139,7 +139,7 @@ CREATE TABLE `automation_principal_capabilities` ( `principal_id` text NOT NULL, CONSTRAINT `automation_principal_capabilities_pk` PRIMARY KEY(`principal_id`, `capability`), CONSTRAINT `fk_automation_principal_capabilities_principal_id_automation_principals_id_fk` FOREIGN KEY (`principal_id`) REFERENCES `automation_principals`(`id`) ON DELETE CASCADE, - CONSTRAINT "automation_principal_capabilities_capability_check" CHECK("capability" IN ('agents:read', 'agents:write', 'monitoring:write', 'notifications:read', 'notifications:write', 'reports:read', 'reports:write', 'tasks:read', 'tasks:write')), + CONSTRAINT "automation_principal_capabilities_capability_check" CHECK("capability" IN ('agents:read', 'agents:write', 'jobs:read', 'jobs:write', 'monitoring:write', 'notifications:read', 'notifications:write', 'reports:read', 'reports:write', 'tasks:read', 'tasks:write')), CONSTRAINT "automation_principal_capabilities_granted_at_check" CHECK("granted_at" BETWEEN 0 AND 8640000000000000) ) STRICT; --> statement-breakpoint @@ -587,6 +587,1332 @@ BEGIN SELECT RAISE(ABORT, 'automation credential predecessor must share principal'); END; --> statement-breakpoint +CREATE TABLE `job_disable_intents` ( + `created_at` integer NOT NULL, + `created_by_id` text NOT NULL, + `created_by_kind` text NOT NULL, + `ended_at` integer, + `ended_by_id` text, + `ended_by_kind` text, + `ended_reason` text, + `expires_at` integer, + `external_job_id` text, + `external_provider` text, + `id` text PRIMARY KEY, + `reason` text NOT NULL, + `scheduled_job_id` text, + `target_kind` text NOT NULL, + CONSTRAINT `fk_job_disable_intents_scheduled_job_id_scheduled_jobs_id_fk` FOREIGN KEY (`scheduled_job_id`) REFERENCES `scheduled_jobs`(`id`) ON UPDATE RESTRICT ON DELETE RESTRICT, + CONSTRAINT "job_disable_intents_created_at_check" CHECK("created_at" BETWEEN 0 AND 8640000000000000), + CONSTRAINT "job_disable_intents_created_actor_check" CHECK((("created_by_kind" = 'user' AND length("created_by_id") = 36 AND instr("created_by_id", char(0)) = 0 AND length(replace("created_by_id", '-', '')) = 32 AND replace("created_by_id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("created_by_id", 9, 1) = '-' AND substr("created_by_id", 14, 1) = '-' AND substr("created_by_id", 15, 1) = '7' AND substr("created_by_id", 19, 1) = '-' AND substr("created_by_id", 20, 1) GLOB '[89ab]' AND substr("created_by_id", 24, 1) = '-') OR ("created_by_kind" = 'automation' AND length("created_by_id") BETWEEN 1 AND 64 AND instr("created_by_id", char(0)) = 0 AND "created_by_id" = lower("created_by_id") AND substr("created_by_id", 1, 1) GLOB '[a-z0-9]' AND "created_by_id" NOT GLOB '*[^a-z0-9._-]*'))), + CONSTRAINT "job_disable_intents_end_check" CHECK(("ended_at" IS NULL AND "ended_by_kind" IS NULL AND "ended_by_id" IS NULL AND "ended_reason" IS NULL) OR ("ended_at" IS NOT NULL AND "ended_at" BETWEEN 0 AND 8640000000000000 AND "ended_at" >= "created_at" AND "ended_by_kind" IS NOT NULL AND "ended_by_id" IS NOT NULL AND "ended_reason" IN ('expired', 're-enabled', 'replaced') AND (("ended_by_kind" = 'user' AND length("ended_by_id") = 36 AND instr("ended_by_id", char(0)) = 0 AND length(replace("ended_by_id", '-', '')) = 32 AND replace("ended_by_id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("ended_by_id", 9, 1) = '-' AND substr("ended_by_id", 14, 1) = '-' AND substr("ended_by_id", 15, 1) = '7' AND substr("ended_by_id", 19, 1) = '-' AND substr("ended_by_id", 20, 1) GLOB '[89ab]' AND substr("ended_by_id", 24, 1) = '-') OR ("ended_by_kind" = 'automation' AND length("ended_by_id") BETWEEN 1 AND 64 AND instr("ended_by_id", char(0)) = 0 AND "ended_by_id" = lower("ended_by_id") AND substr("ended_by_id", 1, 1) GLOB '[a-z0-9]' AND "ended_by_id" NOT GLOB '*[^a-z0-9._-]*') OR ("ended_by_kind" = 'system' AND length("ended_by_id") BETWEEN 1 AND 128 AND instr("ended_by_id", char(0)) = 0 AND "ended_by_id" = lower("ended_by_id") AND substr("ended_by_id", 1, 1) GLOB '[a-z0-9]' AND "ended_by_id" NOT GLOB '*[^a-z0-9._-]*')) AND ("ended_reason" <> 'expired' OR ("ended_by_kind" = 'system' AND "expires_at" IS NOT NULL AND "ended_at" >= "expires_at")))), + CONSTRAINT "job_disable_intents_expiry_check" CHECK("expires_at" IS NULL OR ("expires_at" BETWEEN 0 AND 8640000000000000 AND "expires_at" > "created_at")), + CONSTRAINT "job_disable_intents_external_job_id_check" CHECK("external_job_id" IS NULL OR (length("external_job_id") BETWEEN 1 AND 256 AND instr("external_job_id", char(0)) = 0 AND length(trim("external_job_id", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0)), + CONSTRAINT "job_disable_intents_id_check" CHECK(length("id") = 36 AND instr("id", char(0)) = 0 AND length(replace("id", '-', '')) = 32 AND replace("id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("id", 9, 1) = '-' AND substr("id", 14, 1) = '-' AND substr("id", 15, 1) = '7' AND substr("id", 19, 1) = '-' AND substr("id", 20, 1) GLOB '[89ab]' AND substr("id", 24, 1) = '-'), + CONSTRAINT "job_disable_intents_reason_check" CHECK(length("reason") BETWEEN 1 AND 1000 AND instr("reason", char(0)) = 0 AND length(trim("reason", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND "reason" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST("reason" AS BLOB)) <= 4000), + CONSTRAINT "job_disable_intents_target_check" CHECK(("target_kind" = 'dashboard-schedule' AND "scheduled_job_id" IS NOT NULL AND "external_provider" IS NULL AND "external_job_id" IS NULL) OR ("target_kind" = 'openclaw-cron' AND "scheduled_job_id" IS NULL AND "external_provider" = 'openclaw' AND "external_job_id" IS NOT NULL)) +) STRICT, WITHOUT ROWID; +--> statement-breakpoint +CREATE TABLE `job_run_events` ( + `attempt` integer NOT NULL, + `job_run_id` text NOT NULL, + `kind` text NOT NULL, + `message` text, + `occurred_at` integer NOT NULL, + `progress_json` text, + `sequence` integer NOT NULL, + `worker_instance_id` text, + CONSTRAINT `job_run_events_pk` PRIMARY KEY(`job_run_id`, `sequence`), + CONSTRAINT `fk_job_run_events_job_run_id_job_runs_id_fk` FOREIGN KEY (`job_run_id`) REFERENCES `job_runs`(`id`) ON UPDATE RESTRICT ON DELETE CASCADE, + CONSTRAINT `fk_job_run_events_worker_instance_id_worker_instances_id_fk` FOREIGN KEY (`worker_instance_id`) REFERENCES `worker_instances`(`id`) ON UPDATE RESTRICT ON DELETE SET NULL, + CONSTRAINT "job_run_events_attempt_check" CHECK("attempt" BETWEEN 0 AND 10), + CONSTRAINT "job_run_events_kind_check" CHECK("kind" IN ('cancel-requested', 'cancelled', 'claimed', 'failed', 'lease-expired', 'output-truncated', 'progress', 'queued', 'retry-scheduled', 'stderr', 'stdout', 'succeeded', 'timed-out')), + CONSTRAINT "job_run_events_message_check" CHECK(("message" IS NULL OR (length("message") BETWEEN 1 AND 4096 AND instr("message", char(0)) = 0 AND length(trim("message", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND "message" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST("message" AS BLOB)) <= 4096))), + CONSTRAINT "job_run_events_occurred_at_check" CHECK("occurred_at" BETWEEN 0 AND 8640000000000000), + CONSTRAINT "job_run_events_payload_shape_check" CHECK(("kind" = 'progress' AND "progress_json" IS NOT NULL AND length(CAST("progress_json" AS BLOB)) <= 16384 AND CASE WHEN json_valid("progress_json") THEN json_type("progress_json") = 'object' ELSE 0 END) OR ("kind" IN ('stderr', 'stdout') AND "message" IS NOT NULL AND "progress_json" IS NULL) OR ("kind" NOT IN ('progress', 'stderr', 'stdout') AND "progress_json" IS NULL)), + CONSTRAINT "job_run_events_sequence_check" CHECK("sequence" BETWEEN 1 AND 1000) +) STRICT, WITHOUT ROWID; +--> statement-breakpoint +CREATE TABLE `job_runs` ( + `action_key` text NOT NULL, + `attempt_count` integer DEFAULT 0 NOT NULL, + `attempt_limit` integer NOT NULL, + `available_at` integer NOT NULL, + `cancellation_policy` text NOT NULL, + `cancel_requested_at` integer, + `cancel_requested_by_id` text, + `cancel_requested_by_kind` text, + `display_name` text NOT NULL, + `enqueue_sha256` text NOT NULL, + `event_bytes` integer DEFAULT 0 NOT NULL, + `event_count` integer DEFAULT 0 NOT NULL, + `finished_at` integer, + `first_started_at` integer, + `heartbeat_at` integer, + `id` text PRIMARY KEY, + `idempotency_key` text NOT NULL, + `last_attempt_started_at` integer, + `lease_expires_at` integer, + `lease_owner_id` text, + `lease_token` text, + `payload_event_count` integer DEFAULT 0 NOT NULL, + `payload_json` text NOT NULL, + `priority` integer NOT NULL, + `queued_at` integer NOT NULL, + `requested_by_id` text NOT NULL, + `requested_by_kind` text NOT NULL, + `resource_class` text NOT NULL, + `resource_keys_json` text NOT NULL, + `result_json` text, + `retry_safe` integer NOT NULL, + `scheduled_for_at` integer, + `scheduled_job_id` text, + `scheduled_job_version` integer, + `state` text NOT NULL, + `state_version` integer DEFAULT 1 NOT NULL, + `terminal_code` text, + `terminal_message` text, + `timeout_ms` integer NOT NULL, + `trigger_type` text NOT NULL, + `updated_at` integer NOT NULL, + CONSTRAINT `fk_job_runs_lease_owner_id_worker_instances_id_fk` FOREIGN KEY (`lease_owner_id`) REFERENCES `worker_instances`(`id`) ON UPDATE RESTRICT ON DELETE RESTRICT, + CONSTRAINT `fk_job_runs_scheduled_job_id_scheduled_jobs_id_fk` FOREIGN KEY (`scheduled_job_id`) REFERENCES `scheduled_jobs`(`id`) ON UPDATE RESTRICT ON DELETE RESTRICT, + CONSTRAINT "job_runs_action_key_check" CHECK(length("action_key") BETWEEN 1 AND 128 AND instr("action_key", char(0)) = 0 AND "action_key" = lower("action_key") AND substr("action_key", 1, 1) GLOB '[a-z0-9]' AND "action_key" NOT GLOB '*[^a-z0-9._-]*'), + CONSTRAINT "job_runs_attempt_check" CHECK("attempt_limit" BETWEEN 1 AND 10 AND "attempt_count" BETWEEN 0 AND "attempt_limit" AND (("attempt_count" = 0 AND "first_started_at" IS NULL AND "last_attempt_started_at" IS NULL) OR ("attempt_count" > 0 AND "first_started_at" IS NOT NULL AND "last_attempt_started_at" IS NOT NULL))), + CONSTRAINT "job_runs_available_at_check" CHECK("available_at" BETWEEN 0 AND 8640000000000000 AND "available_at" >= "queued_at"), + CONSTRAINT "job_runs_cancellation_policy_check" CHECK("cancellation_policy" IN ('cooperative', 'never', 'queued-only')), + CONSTRAINT "job_runs_cancel_request_check" CHECK(("state" <> 'cancelled' AND "cancel_requested_at" IS NULL AND "cancel_requested_by_kind" IS NULL AND "cancel_requested_by_id" IS NULL) OR ("cancellation_policy" <> 'never' AND "cancel_requested_at" IS NOT NULL AND "cancel_requested_at" BETWEEN 0 AND 8640000000000000 AND "cancel_requested_at" >= "queued_at" AND "cancel_requested_at" <= "updated_at" AND "cancel_requested_by_kind" IS NOT NULL AND "cancel_requested_by_id" IS NOT NULL AND (("cancel_requested_by_kind" = 'user' AND length("cancel_requested_by_id") = 36 AND instr("cancel_requested_by_id", char(0)) = 0 AND length(replace("cancel_requested_by_id", '-', '')) = 32 AND replace("cancel_requested_by_id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("cancel_requested_by_id", 9, 1) = '-' AND substr("cancel_requested_by_id", 14, 1) = '-' AND substr("cancel_requested_by_id", 15, 1) = '7' AND substr("cancel_requested_by_id", 19, 1) = '-' AND substr("cancel_requested_by_id", 20, 1) GLOB '[89ab]' AND substr("cancel_requested_by_id", 24, 1) = '-') OR ("cancel_requested_by_kind" = 'automation' AND length("cancel_requested_by_id") BETWEEN 1 AND 64 AND instr("cancel_requested_by_id", char(0)) = 0 AND "cancel_requested_by_id" = lower("cancel_requested_by_id") AND substr("cancel_requested_by_id", 1, 1) GLOB '[a-z0-9]' AND "cancel_requested_by_id" NOT GLOB '*[^a-z0-9._-]*') OR ("cancel_requested_by_kind" = 'system' AND length("cancel_requested_by_id") BETWEEN 1 AND 128 AND instr("cancel_requested_by_id", char(0)) = 0 AND "cancel_requested_by_id" = lower("cancel_requested_by_id") AND substr("cancel_requested_by_id", 1, 1) GLOB '[a-z0-9]' AND "cancel_requested_by_id" NOT GLOB '*[^a-z0-9._-]*')))), + CONSTRAINT "job_runs_display_name_check" CHECK(length("display_name") BETWEEN 1 AND 160 AND instr("display_name", char(0)) = 0 AND length(trim("display_name", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND "display_name" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST("display_name" AS BLOB)) <= 640), + CONSTRAINT "job_runs_enqueue_sha256_check" CHECK(length("enqueue_sha256") = 64 AND instr("enqueue_sha256", char(0)) = 0 AND "enqueue_sha256" NOT GLOB '*[^0-9a-f]*'), + CONSTRAINT "job_runs_event_budget_check" CHECK("event_count" BETWEEN 0 AND 1000 AND "payload_event_count" BETWEEN 0 AND 967 AND "payload_event_count" <= "event_count" AND "event_bytes" BETWEEN 0 AND 1048576), + CONSTRAINT "job_runs_id_check" CHECK(length("id") = 36 AND instr("id", char(0)) = 0 AND length(replace("id", '-', '')) = 32 AND replace("id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("id", 9, 1) = '-' AND substr("id", 14, 1) = '-' AND substr("id", 15, 1) = '7' AND substr("id", 19, 1) = '-' AND substr("id", 20, 1) GLOB '[89ab]' AND substr("id", 24, 1) = '-'), + CONSTRAINT "job_runs_idempotency_key_check" CHECK(length("idempotency_key") BETWEEN 32 AND 128 AND instr("idempotency_key", char(0)) = 0 AND "idempotency_key" NOT GLOB '*[^A-Za-z0-9_-]*' AND (length("idempotency_key") % 4 = 0 OR (length("idempotency_key") % 4 = 2 AND substr("idempotency_key", -1, 1) GLOB '[AQgw]') OR (length("idempotency_key") % 4 = 3 AND substr("idempotency_key", -1, 1) GLOB '[AEIMQUYcgkosw048]'))), + CONSTRAINT "job_runs_lease_check" CHECK(("state" <> 'running' AND "lease_owner_id" IS NULL AND "lease_token" IS NULL AND "lease_expires_at" IS NULL AND "heartbeat_at" IS NULL) OR ("state" = 'running' AND "lease_owner_id" IS NOT NULL AND "lease_token" IS NOT NULL AND length("lease_token") = 36 AND instr("lease_token", char(0)) = 0 AND length(replace("lease_token", '-', '')) = 32 AND replace("lease_token", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("lease_token", 9, 1) = '-' AND substr("lease_token", 14, 1) = '-' AND substr("lease_token", 15, 1) = '7' AND substr("lease_token", 19, 1) = '-' AND substr("lease_token", 20, 1) GLOB '[89ab]' AND substr("lease_token", 24, 1) = '-' AND "lease_expires_at" IS NOT NULL AND "heartbeat_at" IS NOT NULL AND "heartbeat_at" BETWEEN 0 AND 8640000000000000 AND "lease_expires_at" BETWEEN 0 AND 8640000000000000 AND "heartbeat_at" >= "last_attempt_started_at" AND "lease_expires_at" > "heartbeat_at")), + CONSTRAINT "job_runs_payload_json_check" CHECK(length(CAST("payload_json" AS BLOB)) <= 65536 AND CASE WHEN json_valid("payload_json") THEN json_type("payload_json") = 'object' ELSE 0 END), + CONSTRAINT "job_runs_priority_check" CHECK("priority" BETWEEN -100 AND 100), + CONSTRAINT "job_runs_requested_actor_check" CHECK((("requested_by_kind" = 'user' AND length("requested_by_id") = 36 AND instr("requested_by_id", char(0)) = 0 AND length(replace("requested_by_id", '-', '')) = 32 AND replace("requested_by_id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("requested_by_id", 9, 1) = '-' AND substr("requested_by_id", 14, 1) = '-' AND substr("requested_by_id", 15, 1) = '7' AND substr("requested_by_id", 19, 1) = '-' AND substr("requested_by_id", 20, 1) GLOB '[89ab]' AND substr("requested_by_id", 24, 1) = '-') OR ("requested_by_kind" = 'automation' AND length("requested_by_id") BETWEEN 1 AND 64 AND instr("requested_by_id", char(0)) = 0 AND "requested_by_id" = lower("requested_by_id") AND substr("requested_by_id", 1, 1) GLOB '[a-z0-9]' AND "requested_by_id" NOT GLOB '*[^a-z0-9._-]*') OR ("requested_by_kind" = 'system' AND length("requested_by_id") BETWEEN 1 AND 128 AND instr("requested_by_id", char(0)) = 0 AND "requested_by_id" = lower("requested_by_id") AND substr("requested_by_id", 1, 1) GLOB '[a-z0-9]' AND "requested_by_id" NOT GLOB '*[^a-z0-9._-]*'))), + CONSTRAINT "job_runs_resource_class_check" CHECK("resource_class" IN ('exclusive', 'host-heavy', 'interactive', 'light', 'network')), + CONSTRAINT "job_runs_resource_keys_json_check" CHECK(length(CAST("resource_keys_json" AS BLOB)) <= 4096 AND CASE WHEN json_valid("resource_keys_json") THEN json_type("resource_keys_json") = 'array' ELSE 0 END), + CONSTRAINT "job_runs_result_json_check" CHECK("result_json" IS NULL OR (length(CAST("result_json" AS BLOB)) <= 65536 AND CASE WHEN json_valid("result_json") THEN json_type("result_json") = 'object' ELSE 0 END)), + CONSTRAINT "job_runs_retry_safe_check" CHECK("retry_safe" IN (0, 1)), + CONSTRAINT "job_runs_schedule_check" CHECK(("trigger_type" = 'schedule' AND "scheduled_job_id" IS NOT NULL AND "scheduled_job_version" BETWEEN 1 AND 9007199254740991 AND "scheduled_for_at" IS NOT NULL AND "scheduled_for_at" BETWEEN 0 AND 8640000000000000 AND "scheduled_for_at" <= "queued_at") OR ("trigger_type" = 'manual' AND "scheduled_job_id" IS NOT NULL AND "scheduled_job_version" BETWEEN 1 AND 9007199254740991 AND "scheduled_for_at" IS NULL) OR ("trigger_type" IN ('startup', 'system') AND "scheduled_job_id" IS NULL AND "scheduled_job_version" IS NULL AND "scheduled_for_at" IS NULL)), + CONSTRAINT "job_runs_state_check" CHECK("state" IN ('cancelled', 'failed', 'queued', 'running', 'succeeded', 'timed-out') AND (("state" = 'queued' AND "finished_at" IS NULL AND "result_json" IS NULL AND "terminal_code" IS NULL AND "terminal_message" IS NULL) OR ("state" = 'running' AND "attempt_count" > 0 AND "finished_at" IS NULL AND "result_json" IS NULL AND "terminal_code" IS NULL AND "terminal_message" IS NULL) OR ("state" = 'succeeded' AND "attempt_count" > 0 AND "finished_at" IS NOT NULL AND "result_json" IS NOT NULL AND "terminal_code" IS NULL AND "terminal_message" IS NULL) OR ("state" IN ('failed', 'timed-out') AND "attempt_count" > 0 AND "finished_at" IS NOT NULL AND "result_json" IS NULL AND "terminal_code" IS NOT NULL AND "terminal_message" IS NOT NULL) OR ("state" = 'cancelled' AND "finished_at" IS NOT NULL AND "result_json" IS NULL AND "terminal_code" IS NOT NULL AND "terminal_message" IS NOT NULL))), + CONSTRAINT "job_runs_state_version_check" CHECK("state_version" BETWEEN 1 AND 9007199254740991), + CONSTRAINT "job_runs_terminal_code_check" CHECK(("terminal_code" IS NULL OR (length("terminal_code") BETWEEN 1 AND 128 AND instr("terminal_code", char(0)) = 0 AND "terminal_code" = lower("terminal_code") AND substr("terminal_code", 1, 1) GLOB '[a-z0-9]' AND "terminal_code" NOT GLOB '*[^a-z0-9._/-]*'))), + CONSTRAINT "job_runs_terminal_message_check" CHECK(("terminal_message" IS NULL OR (length("terminal_message") BETWEEN 1 AND 2000 AND instr("terminal_message", char(0)) = 0 AND length(trim("terminal_message", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND "terminal_message" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST("terminal_message" AS BLOB)) <= 8000))), + CONSTRAINT "job_runs_timeout_check" CHECK("timeout_ms" BETWEEN 1000 AND 86400000), + CONSTRAINT "job_runs_time_check" CHECK("queued_at" BETWEEN 0 AND 8640000000000000 AND "updated_at" BETWEEN 0 AND 8640000000000000 AND "updated_at" >= "queued_at" AND ("first_started_at" IS NULL OR ("first_started_at" BETWEEN 0 AND 8640000000000000 AND "first_started_at" BETWEEN "queued_at" AND "updated_at")) AND ("last_attempt_started_at" IS NULL OR ("first_started_at" IS NOT NULL AND "last_attempt_started_at" BETWEEN 0 AND 8640000000000000 AND "last_attempt_started_at" BETWEEN "first_started_at" AND "updated_at")) AND ("heartbeat_at" IS NULL OR ("last_attempt_started_at" IS NOT NULL AND "heartbeat_at" BETWEEN 0 AND 8640000000000000 AND "heartbeat_at" BETWEEN "last_attempt_started_at" AND "updated_at")) AND ("cancel_requested_at" IS NULL OR ("cancel_requested_at" BETWEEN 0 AND 8640000000000000 AND "cancel_requested_at" BETWEEN "queued_at" AND "updated_at")) AND ("finished_at" IS NULL OR ("finished_at" BETWEEN 0 AND 8640000000000000 AND "finished_at" BETWEEN COALESCE("last_attempt_started_at", "queued_at") AND "updated_at"))) +) STRICT, WITHOUT ROWID; +--> statement-breakpoint +CREATE TABLE `job_worker_control` ( + `claiming_paused` integer NOT NULL, + `id` integer PRIMARY KEY, + `updated_at` integer NOT NULL, + `updated_by_id` text, + `updated_by_kind` text, + `version` integer NOT NULL, + CONSTRAINT "job_worker_control_actor_check" CHECK(("updated_by_kind" IS NULL AND "updated_by_id" IS NULL) OR ("updated_by_kind" IS NOT NULL AND "updated_by_id" IS NOT NULL AND (("updated_by_kind" = 'user' AND length("updated_by_id") = 36 AND instr("updated_by_id", char(0)) = 0 AND length(replace("updated_by_id", '-', '')) = 32 AND replace("updated_by_id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("updated_by_id", 9, 1) = '-' AND substr("updated_by_id", 14, 1) = '-' AND substr("updated_by_id", 15, 1) = '7' AND substr("updated_by_id", 19, 1) = '-' AND substr("updated_by_id", 20, 1) GLOB '[89ab]' AND substr("updated_by_id", 24, 1) = '-') OR ("updated_by_kind" = 'automation' AND length("updated_by_id") BETWEEN 1 AND 64 AND instr("updated_by_id", char(0)) = 0 AND "updated_by_id" = lower("updated_by_id") AND substr("updated_by_id", 1, 1) GLOB '[a-z0-9]' AND "updated_by_id" NOT GLOB '*[^a-z0-9._-]*')))), + CONSTRAINT "job_worker_control_claiming_paused_check" CHECK("claiming_paused" IN (0, 1)), + CONSTRAINT "job_worker_control_id_check" CHECK("id" = 1), + CONSTRAINT "job_worker_control_updated_at_check" CHECK("updated_at" BETWEEN 0 AND 8640000000000000), + CONSTRAINT "job_worker_control_version_check" CHECK("version" BETWEEN 1 AND 9007199254740991) +) STRICT; +--> statement-breakpoint +INSERT INTO job_worker_control ( + id, claiming_paused, updated_at, updated_by_kind, updated_by_id, version +) VALUES (1, 0, 0, NULL, NULL, 1); +--> statement-breakpoint +CREATE TABLE `resource_leases` ( + `acquired_at` integer NOT NULL, + `expires_at` integer NOT NULL, + `job_run_id` text NOT NULL, + `lease_token` text NOT NULL, + `renewed_at` integer NOT NULL, + `resource_key` text PRIMARY KEY, + `worker_instance_id` text NOT NULL, + CONSTRAINT `fk_resource_leases_job_run_id_job_runs_id_fk` FOREIGN KEY (`job_run_id`) REFERENCES `job_runs`(`id`) ON UPDATE RESTRICT ON DELETE RESTRICT, + CONSTRAINT `fk_resource_leases_worker_instance_id_worker_instances_id_fk` FOREIGN KEY (`worker_instance_id`) REFERENCES `worker_instances`(`id`) ON UPDATE RESTRICT ON DELETE RESTRICT, + CONSTRAINT "resource_leases_lease_token_check" CHECK(length("lease_token") = 36 AND instr("lease_token", char(0)) = 0 AND length(replace("lease_token", '-', '')) = 32 AND replace("lease_token", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("lease_token", 9, 1) = '-' AND substr("lease_token", 14, 1) = '-' AND substr("lease_token", 15, 1) = '7' AND substr("lease_token", 19, 1) = '-' AND substr("lease_token", 20, 1) GLOB '[89ab]' AND substr("lease_token", 24, 1) = '-'), + CONSTRAINT "resource_leases_resource_key_check" CHECK(length("resource_key") BETWEEN 1 AND 128 AND instr("resource_key", char(0)) = 0 AND "resource_key" = lower("resource_key") AND substr("resource_key", 1, 1) GLOB '[a-z0-9]' AND "resource_key" NOT GLOB '*[^a-z0-9._-]*'), + CONSTRAINT "resource_leases_time_check" CHECK("acquired_at" BETWEEN 0 AND 8640000000000000 AND "renewed_at" BETWEEN 0 AND 8640000000000000 AND "expires_at" BETWEEN 0 AND 8640000000000000 AND "renewed_at" >= "acquired_at" AND "expires_at" > "renewed_at") +) STRICT, WITHOUT ROWID; +--> statement-breakpoint +CREATE TABLE `scheduled_jobs` ( + `action_key` text NOT NULL, + `action_payload_json` text NOT NULL, + `attempt_limit` integer NOT NULL, + `cancellation_policy` text NOT NULL, + `created_at` integer NOT NULL, + `cron_expression` text, + `description` text NOT NULL, + `enabled` integer NOT NULL, + `id` text PRIMARY KEY, + `interval_ms` integer, + `name` text NOT NULL, + `next_run_at` integer, + `priority` integer NOT NULL, + `resource_class` text NOT NULL, + `resource_keys_json` text NOT NULL, + `retry_safe` integer NOT NULL, + `schedule_kind` text NOT NULL, + `time_of_day` text, + `time_zone` text, + `timeout_ms` integer NOT NULL, + `updated_at` integer NOT NULL, + `version` integer NOT NULL, + CONSTRAINT "scheduled_jobs_action_key_check" CHECK(length("action_key") BETWEEN 1 AND 128 AND instr("action_key", char(0)) = 0 AND "action_key" = lower("action_key") AND substr("action_key", 1, 1) GLOB '[a-z0-9]' AND "action_key" NOT GLOB '*[^a-z0-9._-]*'), + CONSTRAINT "scheduled_jobs_action_payload_json_check" CHECK(length(CAST("action_payload_json" AS BLOB)) <= 65536 AND CASE WHEN json_valid("action_payload_json") THEN json_type("action_payload_json") = 'object' ELSE 0 END), + CONSTRAINT "scheduled_jobs_attempt_limit_check" CHECK("attempt_limit" BETWEEN 1 AND 10), + CONSTRAINT "scheduled_jobs_cancellation_policy_check" CHECK("cancellation_policy" IN ('cooperative', 'never', 'queued-only')), + CONSTRAINT "scheduled_jobs_created_at_check" CHECK("created_at" BETWEEN 0 AND 8640000000000000), + CONSTRAINT "scheduled_jobs_description_check" CHECK(length("description") BETWEEN 1 AND 1000 AND instr("description", char(0)) = 0 AND length(trim("description", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND "description" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST("description" AS BLOB)) <= 4000), + CONSTRAINT "scheduled_jobs_enabled_check" CHECK("enabled" IN (0, 1)), + CONSTRAINT "scheduled_jobs_id_check" CHECK(length("id") BETWEEN 1 AND 80 AND instr("id", char(0)) = 0 AND "id" = lower("id") AND substr("id", 1, 1) GLOB '[a-z0-9]' AND "id" NOT GLOB '*[^a-z0-9._-]*'), + CONSTRAINT "scheduled_jobs_name_check" CHECK(length("name") BETWEEN 1 AND 160 AND instr("name", char(0)) = 0 AND length(trim("name", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND "name" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST("name" AS BLOB)) <= 640), + CONSTRAINT "scheduled_jobs_next_run_check" CHECK(("next_run_at" IS NULL OR "next_run_at" BETWEEN 0 AND 8640000000000000) AND ("enabled" = 0 OR "next_run_at" IS NOT NULL)), + CONSTRAINT "scheduled_jobs_priority_check" CHECK("priority" BETWEEN -100 AND 100), + CONSTRAINT "scheduled_jobs_resource_class_check" CHECK("resource_class" IN ('exclusive', 'host-heavy', 'interactive', 'light', 'network')), + CONSTRAINT "scheduled_jobs_resource_keys_json_check" CHECK(length(CAST("resource_keys_json" AS BLOB)) <= 4096 AND CASE WHEN json_valid("resource_keys_json") THEN json_type("resource_keys_json") = 'array' ELSE 0 END), + CONSTRAINT "scheduled_jobs_retry_safe_check" CHECK("retry_safe" IN (0, 1)), + CONSTRAINT "scheduled_jobs_schedule_shape_check" CHECK(("schedule_kind" = 'interval' AND "interval_ms" BETWEEN 60000 AND 31536000000 AND "time_of_day" IS NULL AND "cron_expression" IS NULL AND "time_zone" IS NULL) OR ("schedule_kind" = 'daily' AND "interval_ms" IS NULL AND "time_of_day" IS NOT NULL AND instr("time_of_day", char(0)) = 0 AND "time_of_day" GLOB '[0-2][0-9]:[0-5][0-9]' AND CAST(substr("time_of_day", 1, 2) AS INTEGER) BETWEEN 0 AND 23 AND "cron_expression" IS NULL AND "time_zone" IS NOT NULL) OR ("schedule_kind" = 'cron' AND "interval_ms" IS NULL AND "time_of_day" IS NULL AND "cron_expression" IS NOT NULL AND length("cron_expression") BETWEEN 9 AND 200 AND instr("cron_expression", char(0)) = 0 AND "cron_expression" = trim("cron_expression") AND "cron_expression" NOT LIKE '% %' AND "cron_expression" NOT GLOB '*[^-0-9*,/ ]*' AND length("cron_expression") - length(replace("cron_expression", ' ', '')) = 4 AND "time_zone" IS NOT NULL)), + CONSTRAINT "scheduled_jobs_time_zone_check" CHECK("time_zone" IS NULL OR "time_zone" IN ('Africa/Abidjan', 'Africa/Accra', 'Africa/Addis_Ababa', 'Africa/Algiers', 'Africa/Asmara', 'Africa/Bamako', 'Africa/Bangui', 'Africa/Banjul', 'Africa/Bissau', 'Africa/Blantyre', 'Africa/Brazzaville', 'Africa/Bujumbura', 'Africa/Cairo', 'Africa/Casablanca', 'Africa/Ceuta', 'Africa/Conakry', 'Africa/Dakar', 'Africa/Dar_es_Salaam', 'Africa/Djibouti', 'Africa/Douala', 'Africa/El_Aaiun', 'Africa/Freetown', 'Africa/Gaborone', 'Africa/Harare', 'Africa/Johannesburg', 'Africa/Juba', 'Africa/Kampala', 'Africa/Khartoum', 'Africa/Kigali', 'Africa/Kinshasa', 'Africa/Lagos', 'Africa/Libreville', 'Africa/Lome', 'Africa/Luanda', 'Africa/Lubumbashi', 'Africa/Lusaka', 'Africa/Malabo', 'Africa/Maputo', 'Africa/Maseru', 'Africa/Mbabane', 'Africa/Mogadishu', 'Africa/Monrovia', 'Africa/Nairobi', 'Africa/Ndjamena', 'Africa/Niamey', 'Africa/Nouakchott', 'Africa/Ouagadougou', 'Africa/Porto-Novo', 'Africa/Sao_Tome', 'Africa/Tripoli', 'Africa/Tunis', 'Africa/Windhoek', 'America/Adak', 'America/Anchorage', 'America/Anguilla', 'America/Antigua', 'America/Araguaina', 'America/Argentina/Buenos_Aires', 'America/Argentina/Catamarca', 'America/Argentina/Cordoba', 'America/Argentina/Jujuy', 'America/Argentina/La_Rioja', 'America/Argentina/Mendoza', 'America/Argentina/Rio_Gallegos', 'America/Argentina/Salta', 'America/Argentina/San_Juan', 'America/Argentina/San_Luis', 'America/Argentina/Tucuman', 'America/Argentina/Ushuaia', 'America/Aruba', 'America/Asuncion', 'America/Atikokan', 'America/Bahia', 'America/Bahia_Banderas', 'America/Barbados', 'America/Belem', 'America/Belize', 'America/Blanc-Sablon', 'America/Boa_Vista', 'America/Bogota', 'America/Boise', 'America/Cambridge_Bay', 'America/Campo_Grande', 'America/Cancun', 'America/Caracas', 'America/Cayenne', 'America/Cayman', 'America/Chicago', 'America/Chihuahua', 'America/Ciudad_Juarez', 'America/Costa_Rica', 'America/Creston', 'America/Cuiaba', 'America/Curacao', 'America/Danmarkshavn', 'America/Dawson', 'America/Dawson_Creek', 'America/Denver', 'America/Detroit', 'America/Dominica', 'America/Edmonton', 'America/Eirunepe', 'America/El_Salvador', 'America/Fort_Nelson', 'America/Fortaleza', 'America/Glace_Bay', 'America/Goose_Bay', 'America/Grand_Turk', 'America/Grenada', 'America/Guadeloupe', 'America/Guatemala', 'America/Guayaquil', 'America/Guyana', 'America/Halifax', 'America/Havana', 'America/Hermosillo', 'America/Indiana/Indianapolis', 'America/Indiana/Knox', 'America/Indiana/Marengo', 'America/Indiana/Petersburg', 'America/Indiana/Tell_City', 'America/Indiana/Vevay', 'America/Indiana/Vincennes', 'America/Indiana/Winamac', 'America/Inuvik', 'America/Iqaluit', 'America/Jamaica', 'America/Juneau', 'America/Kentucky/Louisville', 'America/Kentucky/Monticello', 'America/Kralendijk', 'America/La_Paz', 'America/Lima', 'America/Los_Angeles', 'America/Lower_Princes', 'America/Maceio', 'America/Managua', 'America/Manaus', 'America/Marigot', 'America/Martinique', 'America/Matamoros', 'America/Mazatlan', 'America/Menominee', 'America/Merida', 'America/Metlakatla', 'America/Mexico_City', 'America/Miquelon', 'America/Moncton', 'America/Monterrey', 'America/Montevideo', 'America/Montserrat', 'America/Nassau', 'America/New_York', 'America/Nome', 'America/Noronha', 'America/North_Dakota/Beulah', 'America/North_Dakota/Center', 'America/North_Dakota/New_Salem', 'America/Nuuk', 'America/Ojinaga', 'America/Panama', 'America/Paramaribo', 'America/Phoenix', 'America/Port-au-Prince', 'America/Port_of_Spain', 'America/Porto_Velho', 'America/Puerto_Rico', 'America/Punta_Arenas', 'America/Rankin_Inlet', 'America/Recife', 'America/Regina', 'America/Resolute', 'America/Rio_Branco', 'America/Santarem', 'America/Santiago', 'America/Santo_Domingo', 'America/Sao_Paulo', 'America/Scoresbysund', 'America/Sitka', 'America/St_Barthelemy', 'America/St_Johns', 'America/St_Kitts', 'America/St_Lucia', 'America/St_Thomas', 'America/St_Vincent', 'America/Swift_Current', 'America/Tegucigalpa', 'America/Thule', 'America/Tijuana', 'America/Toronto', 'America/Tortola', 'America/Vancouver', 'America/Whitehorse', 'America/Winnipeg', 'America/Yakutat', 'Antarctica/Casey', 'Antarctica/Davis', 'Antarctica/DumontDUrville', 'Antarctica/Macquarie', 'Antarctica/Mawson', 'Antarctica/McMurdo', 'Antarctica/Palmer', 'Antarctica/Rothera', 'Antarctica/Syowa', 'Antarctica/Troll', 'Antarctica/Vostok', 'Arctic/Longyearbyen', 'Asia/Aden', 'Asia/Almaty', 'Asia/Amman', 'Asia/Anadyr', 'Asia/Aqtau', 'Asia/Aqtobe', 'Asia/Ashgabat', 'Asia/Atyrau', 'Asia/Baghdad', 'Asia/Bahrain', 'Asia/Baku', 'Asia/Bangkok', 'Asia/Barnaul', 'Asia/Beirut', 'Asia/Bishkek', 'Asia/Brunei', 'Asia/Chita', 'Asia/Choibalsan', 'Asia/Colombo', 'Asia/Damascus', 'Asia/Dhaka', 'Asia/Dili', 'Asia/Dubai', 'Asia/Dushanbe', 'Asia/Famagusta', 'Asia/Gaza', 'Asia/Hebron', 'Asia/Ho_Chi_Minh', 'Asia/Hong_Kong', 'Asia/Hovd', 'Asia/Irkutsk', 'Asia/Jakarta', 'Asia/Jayapura', 'Asia/Jerusalem', 'Asia/Kabul', 'Asia/Kamchatka', 'Asia/Karachi', 'Asia/Kathmandu', 'Asia/Khandyga', 'Asia/Kolkata', 'Asia/Krasnoyarsk', 'Asia/Kuala_Lumpur', 'Asia/Kuching', 'Asia/Kuwait', 'Asia/Macau', 'Asia/Magadan', 'Asia/Makassar', 'Asia/Manila', 'Asia/Muscat', 'Asia/Nicosia', 'Asia/Novokuznetsk', 'Asia/Novosibirsk', 'Asia/Omsk', 'Asia/Oral', 'Asia/Phnom_Penh', 'Asia/Pontianak', 'Asia/Pyongyang', 'Asia/Qatar', 'Asia/Qostanay', 'Asia/Qyzylorda', 'Asia/Riyadh', 'Asia/Sakhalin', 'Asia/Samarkand', 'Asia/Seoul', 'Asia/Shanghai', 'Asia/Singapore', 'Asia/Srednekolymsk', 'Asia/Taipei', 'Asia/Tashkent', 'Asia/Tbilisi', 'Asia/Tehran', 'Asia/Thimphu', 'Asia/Tokyo', 'Asia/Tomsk', 'Asia/Ulaanbaatar', 'Asia/Urumqi', 'Asia/Ust-Nera', 'Asia/Vientiane', 'Asia/Vladivostok', 'Asia/Yakutsk', 'Asia/Yangon', 'Asia/Yekaterinburg', 'Asia/Yerevan', 'Atlantic/Azores', 'Atlantic/Bermuda', 'Atlantic/Canary', 'Atlantic/Cape_Verde', 'Atlantic/Faroe', 'Atlantic/Madeira', 'Atlantic/Reykjavik', 'Atlantic/South_Georgia', 'Atlantic/St_Helena', 'Atlantic/Stanley', 'Australia/Adelaide', 'Australia/Brisbane', 'Australia/Broken_Hill', 'Australia/Darwin', 'Australia/Eucla', 'Australia/Hobart', 'Australia/Lindeman', 'Australia/Lord_Howe', 'Australia/Melbourne', 'Australia/Perth', 'Australia/Sydney', 'Etc/GMT+1', 'Etc/GMT+10', 'Etc/GMT+11', 'Etc/GMT+12', 'Etc/GMT+2', 'Etc/GMT+3', 'Etc/GMT+4', 'Etc/GMT+5', 'Etc/GMT+6', 'Etc/GMT+7', 'Etc/GMT+8', 'Etc/GMT+9', 'Etc/GMT-1', 'Etc/GMT-10', 'Etc/GMT-11', 'Etc/GMT-12', 'Etc/GMT-13', 'Etc/GMT-14', 'Etc/GMT-2', 'Etc/GMT-3', 'Etc/GMT-4', 'Etc/GMT-5', 'Etc/GMT-6', 'Etc/GMT-7', 'Etc/GMT-8', 'Etc/GMT-9', 'Europe/Amsterdam', 'Europe/Andorra', 'Europe/Astrakhan', 'Europe/Athens', 'Europe/Belgrade', 'Europe/Berlin', 'Europe/Bratislava', 'Europe/Brussels', 'Europe/Bucharest', 'Europe/Budapest', 'Europe/Busingen', 'Europe/Chisinau', 'Europe/Copenhagen', 'Europe/Dublin', 'Europe/Gibraltar', 'Europe/Guernsey', 'Europe/Helsinki', 'Europe/Isle_of_Man', 'Europe/Istanbul', 'Europe/Jersey', 'Europe/Kaliningrad', 'Europe/Kirov', 'Europe/Kyiv', 'Europe/Lisbon', 'Europe/Ljubljana', 'Europe/London', 'Europe/Luxembourg', 'Europe/Madrid', 'Europe/Malta', 'Europe/Mariehamn', 'Europe/Minsk', 'Europe/Monaco', 'Europe/Moscow', 'Europe/Oslo', 'Europe/Paris', 'Europe/Podgorica', 'Europe/Prague', 'Europe/Riga', 'Europe/Rome', 'Europe/Samara', 'Europe/San_Marino', 'Europe/Sarajevo', 'Europe/Saratov', 'Europe/Simferopol', 'Europe/Skopje', 'Europe/Sofia', 'Europe/Stockholm', 'Europe/Tallinn', 'Europe/Tirane', 'Europe/Ulyanovsk', 'Europe/Vaduz', 'Europe/Vatican', 'Europe/Vienna', 'Europe/Vilnius', 'Europe/Volgograd', 'Europe/Warsaw', 'Europe/Zagreb', 'Europe/Zurich', 'Indian/Antananarivo', 'Indian/Chagos', 'Indian/Christmas', 'Indian/Cocos', 'Indian/Comoro', 'Indian/Kerguelen', 'Indian/Mahe', 'Indian/Maldives', 'Indian/Mauritius', 'Indian/Mayotte', 'Indian/Reunion', 'Pacific/Apia', 'Pacific/Auckland', 'Pacific/Bougainville', 'Pacific/Chatham', 'Pacific/Chuuk', 'Pacific/Easter', 'Pacific/Efate', 'Pacific/Fakaofo', 'Pacific/Fiji', 'Pacific/Funafuti', 'Pacific/Galapagos', 'Pacific/Gambier', 'Pacific/Guadalcanal', 'Pacific/Guam', 'Pacific/Honolulu', 'Pacific/Kanton', 'Pacific/Kiritimati', 'Pacific/Kosrae', 'Pacific/Kwajalein', 'Pacific/Majuro', 'Pacific/Marquesas', 'Pacific/Midway', 'Pacific/Nauru', 'Pacific/Niue', 'Pacific/Norfolk', 'Pacific/Noumea', 'Pacific/Pago_Pago', 'Pacific/Palau', 'Pacific/Pitcairn', 'Pacific/Pohnpei', 'Pacific/Port_Moresby', 'Pacific/Rarotonga', 'Pacific/Saipan', 'Pacific/Tahiti', 'Pacific/Tarawa', 'Pacific/Tongatapu', 'Pacific/Wake', 'Pacific/Wallis', 'UTC')), + CONSTRAINT "scheduled_jobs_timeout_check" CHECK("timeout_ms" BETWEEN 1000 AND 86400000), + CONSTRAINT "scheduled_jobs_updated_at_check" CHECK("updated_at" BETWEEN 0 AND 8640000000000000 AND "updated_at" >= "created_at"), + CONSTRAINT "scheduled_jobs_version_check" CHECK("version" BETWEEN 1 AND 9007199254740991) +) STRICT, WITHOUT ROWID; +--> statement-breakpoint +CREATE TABLE `worker_instances` ( + `capacity` integer NOT NULL, + `draining_at` integer, + `heartbeat_at` integer NOT NULL, + `id` text PRIMARY KEY, + `pid` integer NOT NULL, + `release_id` text NOT NULL, + `started_at` integer NOT NULL, + `state` text NOT NULL, + `stopped_at` integer, + CONSTRAINT "worker_instances_capacity_check" CHECK("capacity" BETWEEN 1 AND 16), + CONSTRAINT "worker_instances_id_check" CHECK(length("id") = 36 AND instr("id", char(0)) = 0 AND length(replace("id", '-', '')) = 32 AND replace("id", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr("id", 9, 1) = '-' AND substr("id", 14, 1) = '-' AND substr("id", 15, 1) = '7' AND substr("id", 19, 1) = '-' AND substr("id", 20, 1) GLOB '[89ab]' AND substr("id", 24, 1) = '-'), + CONSTRAINT "worker_instances_pid_check" CHECK("pid" BETWEEN 1 AND 2147483647), + CONSTRAINT "worker_instances_release_id_check" CHECK(length("release_id") = 40 AND instr("release_id", char(0)) = 0 AND "release_id" NOT GLOB '*[^0-9a-f]*'), + CONSTRAINT "worker_instances_state_check" CHECK(("state" = 'online' AND "draining_at" IS NULL AND "stopped_at" IS NULL) OR ("state" = 'draining' AND "draining_at" IS NOT NULL AND "stopped_at" IS NULL) OR ("state" = 'stopped' AND "draining_at" IS NOT NULL AND "stopped_at" IS NOT NULL)), + CONSTRAINT "worker_instances_time_check" CHECK("started_at" BETWEEN 0 AND 8640000000000000 AND "heartbeat_at" BETWEEN 0 AND 8640000000000000 AND "heartbeat_at" >= "started_at" AND ("draining_at" IS NULL OR ("draining_at" BETWEEN 0 AND 8640000000000000 AND "draining_at" >= "started_at")) AND ("stopped_at" IS NULL OR ("stopped_at" BETWEEN 0 AND 8640000000000000 AND "stopped_at" >= "draining_at"))) +) STRICT, WITHOUT ROWID; +--> statement-breakpoint +CREATE UNIQUE INDEX `job_disable_intents_active_schedule_unique` ON `job_disable_intents` (`scheduled_job_id`) WHERE "job_disable_intents"."scheduled_job_id" IS NOT NULL AND "job_disable_intents"."ended_at" IS NULL;--> statement-breakpoint +CREATE UNIQUE INDEX `job_disable_intents_active_external_unique` ON `job_disable_intents` (`external_provider`,`external_job_id`) WHERE "job_disable_intents"."external_job_id" IS NOT NULL AND "job_disable_intents"."ended_at" IS NULL;--> statement-breakpoint +CREATE INDEX `job_disable_intents_active_expiry_idx` ON `job_disable_intents` (`expires_at`,`id`) WHERE "job_disable_intents"."expires_at" IS NOT NULL AND "job_disable_intents"."ended_at" IS NULL;--> statement-breakpoint +CREATE INDEX `job_disable_intents_schedule_created_id_idx` ON `job_disable_intents` (`scheduled_job_id`,`created_at`,`id`);--> statement-breakpoint +CREATE INDEX `job_disable_intents_external_created_id_idx` ON `job_disable_intents` (`external_provider`,`external_job_id`,`created_at`,`id`);--> statement-breakpoint +CREATE INDEX `job_run_events_occurred_run_sequence_idx` ON `job_run_events` (`occurred_at`,`job_run_id`,`sequence`);--> statement-breakpoint +CREATE UNIQUE INDEX `job_runs_idempotency_unique` ON `job_runs` (`requested_by_kind`,`requested_by_id`,`idempotency_key`);--> statement-breakpoint +CREATE INDEX `job_runs_claim_idx` ON `job_runs` ("available_at" asc,"priority" desc,"queued_at" asc,"id" asc) WHERE "job_runs"."state" = 'queued';--> statement-breakpoint +CREATE UNIQUE INDEX `job_runs_one_active_schedule_idx` ON `job_runs` (`scheduled_job_id`) WHERE "job_runs"."scheduled_job_id" IS NOT NULL AND "job_runs"."state" IN ('queued', 'running');--> statement-breakpoint +CREATE INDEX `job_runs_queued_id_idx` ON `job_runs` (`queued_at`,`id`);--> statement-breakpoint +CREATE INDEX `job_runs_schedule_queued_id_idx` ON `job_runs` (`scheduled_job_id`,`queued_at`,`id`);--> statement-breakpoint +CREATE INDEX `job_runs_running_lease_idx` ON `job_runs` (`lease_expires_at`,`id`) WHERE "job_runs"."state" = 'running';--> statement-breakpoint +CREATE INDEX `job_runs_running_owner_id_idx` ON `job_runs` (`lease_owner_id`,`id`) WHERE "job_runs"."state" = 'running';--> statement-breakpoint +CREATE INDEX `resource_leases_expiry_key_idx` ON `resource_leases` (`expires_at`,`resource_key`);--> statement-breakpoint +CREATE INDEX `resource_leases_run_key_idx` ON `resource_leases` (`job_run_id`,`resource_key`);--> statement-breakpoint +CREATE INDEX `scheduled_jobs_due_idx` ON `scheduled_jobs` (`next_run_at`,`id`) WHERE "scheduled_jobs"."enabled" = 1;--> statement-breakpoint +CREATE INDEX `scheduled_jobs_updated_id_idx` ON `scheduled_jobs` (`updated_at`,`id`);--> statement-breakpoint +CREATE INDEX `worker_instances_heartbeat_id_idx` ON `worker_instances` (`heartbeat_at`,`id`); +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_validate_resource_keys_insert +BEFORE INSERT ON scheduled_jobs +WHEN json_array_length(NEW.resource_keys_json) > 32 + OR EXISTS ( + SELECT 1 + FROM json_each(NEW.resource_keys_json) AS entry + WHERE entry.type <> 'text' + OR length(CAST(entry.value AS TEXT)) NOT BETWEEN 1 AND 128 + OR CAST(entry.value AS TEXT) <> lower(CAST(entry.value AS TEXT)) + OR substr(CAST(entry.value AS TEXT), 1, 1) NOT GLOB '[a-z0-9]' + OR CAST(entry.value AS TEXT) GLOB '*[^a-z0-9._-]*' + ) + OR EXISTS ( + SELECT 1 + FROM json_each(NEW.resource_keys_json) AS current + JOIN json_each(NEW.resource_keys_json) AS previous + ON previous.key = current.key - 1 + WHERE CAST(current.value AS TEXT) <= CAST(previous.value AS TEXT) + ) +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs resource keys must be canonical'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_validate_resource_keys_update +BEFORE UPDATE OF resource_keys_json ON scheduled_jobs +WHEN json_array_length(NEW.resource_keys_json) > 32 + OR EXISTS ( + SELECT 1 + FROM json_each(NEW.resource_keys_json) AS entry + WHERE entry.type <> 'text' + OR length(CAST(entry.value AS TEXT)) NOT BETWEEN 1 AND 128 + OR CAST(entry.value AS TEXT) <> lower(CAST(entry.value AS TEXT)) + OR substr(CAST(entry.value AS TEXT), 1, 1) NOT GLOB '[a-z0-9]' + OR CAST(entry.value AS TEXT) GLOB '*[^a-z0-9._-]*' + ) + OR EXISTS ( + SELECT 1 + FROM json_each(NEW.resource_keys_json) AS current + JOIN json_each(NEW.resource_keys_json) AS previous + ON previous.key = current.key - 1 + WHERE CAST(current.value AS TEXT) <= CAST(previous.value AS TEXT) + ) +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs resource keys must be canonical'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_validate_cron_insert +BEFORE INSERT ON scheduled_jobs +WHEN NEW.schedule_kind = 'cron' + AND NEW.cron_expression IS NOT NULL + AND length(NEW.cron_expression) BETWEEN 9 AND 200 + AND instr(NEW.cron_expression, char(0)) = 0 + AND NEW.cron_expression = trim(NEW.cron_expression) + AND NEW.cron_expression NOT LIKE '% %' + AND NEW.cron_expression NOT GLOB '*[^-0-9*,/ ]*' + AND length(NEW.cron_expression) - length(replace(NEW.cron_expression, ' ', '')) = 4 + AND EXISTS ( + WITH RECURSIVE + cron_fields(field_index, field_value, remaining, minimum_value, maximum_value) AS ( + SELECT + 0, + substr(NEW.cron_expression, 1, instr(NEW.cron_expression, ' ') - 1), + substr(NEW.cron_expression, instr(NEW.cron_expression, ' ') + 1), + 0, + 59 + UNION ALL + SELECT + field_index + 1, + CASE + WHEN instr(remaining, ' ') = 0 THEN remaining + ELSE substr(remaining, 1, instr(remaining, ' ') - 1) + END, + CASE + WHEN instr(remaining, ' ') = 0 THEN '' + ELSE substr(remaining, instr(remaining, ' ') + 1) + END, + CASE field_index + 1 WHEN 2 THEN 1 WHEN 3 THEN 1 ELSE 0 END, + CASE field_index + 1 + WHEN 0 THEN 59 + WHEN 1 THEN 23 + WHEN 2 THEN 31 + WHEN 3 THEN 12 + ELSE 7 + END + FROM cron_fields + WHERE field_index < 4 + ), + cron_parts( + field_index, + part_index, + minimum_value, + maximum_value, + part_value, + remaining + ) AS ( + SELECT + field_index, + 0, + minimum_value, + maximum_value, + substr(field_value || ',', 1, instr(field_value || ',', ',') - 1), + substr(field_value || ',', instr(field_value || ',', ',') + 1) + FROM cron_fields + UNION ALL + SELECT + field_index, + part_index + 1, + minimum_value, + maximum_value, + substr(remaining, 1, instr(remaining, ',') - 1), + substr(remaining, instr(remaining, ',') + 1) + FROM cron_parts + WHERE remaining <> '' + ), + parsed_parts AS ( + SELECT + *, + length(part_value) - length(replace(part_value, '/', '')) AS slash_count, + CASE + WHEN instr(part_value, '/') = 0 THEN part_value + ELSE substr(part_value, 1, instr(part_value, '/') - 1) + END AS base_value, + CASE + WHEN instr(part_value, '/') = 0 THEN NULL + ELSE substr(part_value, instr(part_value, '/') + 1) + END AS step_value + FROM cron_parts + ), + parsed_ranges AS ( + SELECT + *, + length(base_value) - length(replace(base_value, '-', '')) AS range_count, + CASE + WHEN instr(base_value, '-') = 0 THEN base_value + ELSE substr(base_value, 1, instr(base_value, '-') - 1) + END AS left_value, + CASE + WHEN instr(base_value, '-') = 0 THEN NULL + ELSE substr(base_value, instr(base_value, '-') + 1) + END AS right_value + FROM parsed_parts + ), + invalid_parts AS ( + SELECT 1 + FROM parsed_ranges + WHERE length(part_value) = 0 + OR slash_count > 1 + OR ( + slash_count = 1 + AND ( + length(step_value) = 0 + OR step_value GLOB '*[^0-9]*' + OR CAST(step_value AS INTEGER) < 1 + OR CAST(step_value AS INTEGER) > maximum_value + ) + ) + OR ( + base_value <> '*' + AND ( + range_count > 1 + OR ( + range_count = 0 + AND ( + length(left_value) = 0 + OR left_value GLOB '*[^0-9]*' + OR CAST(left_value AS INTEGER) < minimum_value + OR CAST(left_value AS INTEGER) > maximum_value + ) + ) + OR ( + range_count = 1 + AND ( + length(left_value) = 0 + OR left_value GLOB '*[^0-9]*' + OR length(right_value) = 0 + OR right_value GLOB '*[^0-9]*' + OR CAST(left_value AS INTEGER) < minimum_value + OR CAST(left_value AS INTEGER) > maximum_value + OR CAST(right_value AS INTEGER) < minimum_value + OR CAST(right_value AS INTEGER) > maximum_value + OR CAST(left_value AS INTEGER) > CAST(right_value AS INTEGER) + ) + ) + ) + ) + ), + field_modes AS ( + SELECT + field_index, + max( + CASE + WHEN part_index = 0 AND base_value = '*' THEN 1 + ELSE 0 + END + ) AS starts_with_wildcard, + max( + CASE + WHEN part_index = 0 + AND base_value = '*' + AND ( + step_value IS NULL + OR CAST(step_value AS INTEGER) = 1 + ) + THEN 1 + ELSE 0 + END + ) AS unrestricted + FROM parsed_ranges + GROUP BY field_index + ), + domain_values(value) AS ( + SELECT 0 + UNION ALL + SELECT value + 1 + FROM domain_values + WHERE value < 59 + ), + expanded_values(field_index, value) AS ( + SELECT DISTINCT + parsed_ranges.field_index, + CASE + WHEN parsed_ranges.field_index = 4 AND domain_values.value = 7 + THEN 0 + ELSE domain_values.value + END + FROM parsed_ranges + JOIN domain_values + ON domain_values.value BETWEEN parsed_ranges.minimum_value + AND parsed_ranges.maximum_value + WHERE ( + parsed_ranges.base_value = '*' + AND ( + domain_values.value - parsed_ranges.minimum_value + ) % coalesce(CAST(parsed_ranges.step_value AS INTEGER), 1) = 0 + ) + OR ( + parsed_ranges.base_value <> '*' + AND parsed_ranges.range_count = 0 + AND domain_values.value >= CAST(parsed_ranges.left_value AS INTEGER) + AND domain_values.value <= CASE + WHEN parsed_ranges.step_value IS NULL + THEN CAST(parsed_ranges.left_value AS INTEGER) + ELSE parsed_ranges.maximum_value + END + AND ( + domain_values.value - CAST(parsed_ranges.left_value AS INTEGER) + ) % coalesce(CAST(parsed_ranges.step_value AS INTEGER), 1) = 0 + ) + OR ( + parsed_ranges.base_value <> '*' + AND parsed_ranges.range_count = 1 + AND domain_values.value BETWEEN + CAST(parsed_ranges.left_value AS INTEGER) + AND CAST(parsed_ranges.right_value AS INTEGER) + AND ( + domain_values.value - CAST(parsed_ranges.left_value AS INTEGER) + ) % coalesce(CAST(parsed_ranges.step_value AS INTEGER), 1) = 0 + ) + ), + viability_required AS ( + SELECT 1 + FROM field_modes AS day_field + JOIN field_modes AS weekday_field + ON weekday_field.field_index = 4 + WHERE day_field.field_index = 2 + AND day_field.unrestricted = 0 + AND ( + weekday_field.unrestricted = 1 + OR day_field.starts_with_wildcard = 1 + OR weekday_field.starts_with_wildcard = 1 + ) + ), + viable_day_month AS ( + SELECT 1 + FROM expanded_values AS day_value + JOIN expanded_values AS month_value + ON month_value.field_index = 3 + WHERE day_value.field_index = 2 + AND day_value.value <= CASE month_value.value + WHEN 2 THEN 29 + WHEN 4 THEN 30 + WHEN 6 THEN 30 + WHEN 9 THEN 30 + WHEN 11 THEN 30 + ELSE 31 + END + LIMIT 1 + ) + SELECT 1 + FROM invalid_parts + UNION ALL + SELECT 1 + WHERE EXISTS (SELECT 1 FROM viability_required) + AND NOT EXISTS (SELECT 1 FROM viable_day_month) + LIMIT 1 + ) +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs cron expression must be semantically valid'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_validate_cron_update +BEFORE UPDATE OF schedule_kind, cron_expression ON scheduled_jobs +WHEN NEW.schedule_kind = 'cron' + AND NEW.cron_expression IS NOT NULL + AND length(NEW.cron_expression) BETWEEN 9 AND 200 + AND instr(NEW.cron_expression, char(0)) = 0 + AND NEW.cron_expression = trim(NEW.cron_expression) + AND NEW.cron_expression NOT LIKE '% %' + AND NEW.cron_expression NOT GLOB '*[^-0-9*,/ ]*' + AND length(NEW.cron_expression) - length(replace(NEW.cron_expression, ' ', '')) = 4 + AND EXISTS ( + WITH RECURSIVE + cron_fields(field_index, field_value, remaining, minimum_value, maximum_value) AS ( + SELECT + 0, + substr(NEW.cron_expression, 1, instr(NEW.cron_expression, ' ') - 1), + substr(NEW.cron_expression, instr(NEW.cron_expression, ' ') + 1), + 0, + 59 + UNION ALL + SELECT + field_index + 1, + CASE + WHEN instr(remaining, ' ') = 0 THEN remaining + ELSE substr(remaining, 1, instr(remaining, ' ') - 1) + END, + CASE + WHEN instr(remaining, ' ') = 0 THEN '' + ELSE substr(remaining, instr(remaining, ' ') + 1) + END, + CASE field_index + 1 WHEN 2 THEN 1 WHEN 3 THEN 1 ELSE 0 END, + CASE field_index + 1 + WHEN 0 THEN 59 + WHEN 1 THEN 23 + WHEN 2 THEN 31 + WHEN 3 THEN 12 + ELSE 7 + END + FROM cron_fields + WHERE field_index < 4 + ), + cron_parts( + field_index, + part_index, + minimum_value, + maximum_value, + part_value, + remaining + ) AS ( + SELECT + field_index, + 0, + minimum_value, + maximum_value, + substr(field_value || ',', 1, instr(field_value || ',', ',') - 1), + substr(field_value || ',', instr(field_value || ',', ',') + 1) + FROM cron_fields + UNION ALL + SELECT + field_index, + part_index + 1, + minimum_value, + maximum_value, + substr(remaining, 1, instr(remaining, ',') - 1), + substr(remaining, instr(remaining, ',') + 1) + FROM cron_parts + WHERE remaining <> '' + ), + parsed_parts AS ( + SELECT + *, + length(part_value) - length(replace(part_value, '/', '')) AS slash_count, + CASE + WHEN instr(part_value, '/') = 0 THEN part_value + ELSE substr(part_value, 1, instr(part_value, '/') - 1) + END AS base_value, + CASE + WHEN instr(part_value, '/') = 0 THEN NULL + ELSE substr(part_value, instr(part_value, '/') + 1) + END AS step_value + FROM cron_parts + ), + parsed_ranges AS ( + SELECT + *, + length(base_value) - length(replace(base_value, '-', '')) AS range_count, + CASE + WHEN instr(base_value, '-') = 0 THEN base_value + ELSE substr(base_value, 1, instr(base_value, '-') - 1) + END AS left_value, + CASE + WHEN instr(base_value, '-') = 0 THEN NULL + ELSE substr(base_value, instr(base_value, '-') + 1) + END AS right_value + FROM parsed_parts + ), + invalid_parts AS ( + SELECT 1 + FROM parsed_ranges + WHERE length(part_value) = 0 + OR slash_count > 1 + OR ( + slash_count = 1 + AND ( + length(step_value) = 0 + OR step_value GLOB '*[^0-9]*' + OR CAST(step_value AS INTEGER) < 1 + OR CAST(step_value AS INTEGER) > maximum_value + ) + ) + OR ( + base_value <> '*' + AND ( + range_count > 1 + OR ( + range_count = 0 + AND ( + length(left_value) = 0 + OR left_value GLOB '*[^0-9]*' + OR CAST(left_value AS INTEGER) < minimum_value + OR CAST(left_value AS INTEGER) > maximum_value + ) + ) + OR ( + range_count = 1 + AND ( + length(left_value) = 0 + OR left_value GLOB '*[^0-9]*' + OR length(right_value) = 0 + OR right_value GLOB '*[^0-9]*' + OR CAST(left_value AS INTEGER) < minimum_value + OR CAST(left_value AS INTEGER) > maximum_value + OR CAST(right_value AS INTEGER) < minimum_value + OR CAST(right_value AS INTEGER) > maximum_value + OR CAST(left_value AS INTEGER) > CAST(right_value AS INTEGER) + ) + ) + ) + ) + ), + field_modes AS ( + SELECT + field_index, + max( + CASE + WHEN part_index = 0 AND base_value = '*' THEN 1 + ELSE 0 + END + ) AS starts_with_wildcard, + max( + CASE + WHEN part_index = 0 + AND base_value = '*' + AND ( + step_value IS NULL + OR CAST(step_value AS INTEGER) = 1 + ) + THEN 1 + ELSE 0 + END + ) AS unrestricted + FROM parsed_ranges + GROUP BY field_index + ), + domain_values(value) AS ( + SELECT 0 + UNION ALL + SELECT value + 1 + FROM domain_values + WHERE value < 59 + ), + expanded_values(field_index, value) AS ( + SELECT DISTINCT + parsed_ranges.field_index, + CASE + WHEN parsed_ranges.field_index = 4 AND domain_values.value = 7 + THEN 0 + ELSE domain_values.value + END + FROM parsed_ranges + JOIN domain_values + ON domain_values.value BETWEEN parsed_ranges.minimum_value + AND parsed_ranges.maximum_value + WHERE ( + parsed_ranges.base_value = '*' + AND ( + domain_values.value - parsed_ranges.minimum_value + ) % coalesce(CAST(parsed_ranges.step_value AS INTEGER), 1) = 0 + ) + OR ( + parsed_ranges.base_value <> '*' + AND parsed_ranges.range_count = 0 + AND domain_values.value >= CAST(parsed_ranges.left_value AS INTEGER) + AND domain_values.value <= CASE + WHEN parsed_ranges.step_value IS NULL + THEN CAST(parsed_ranges.left_value AS INTEGER) + ELSE parsed_ranges.maximum_value + END + AND ( + domain_values.value - CAST(parsed_ranges.left_value AS INTEGER) + ) % coalesce(CAST(parsed_ranges.step_value AS INTEGER), 1) = 0 + ) + OR ( + parsed_ranges.base_value <> '*' + AND parsed_ranges.range_count = 1 + AND domain_values.value BETWEEN + CAST(parsed_ranges.left_value AS INTEGER) + AND CAST(parsed_ranges.right_value AS INTEGER) + AND ( + domain_values.value - CAST(parsed_ranges.left_value AS INTEGER) + ) % coalesce(CAST(parsed_ranges.step_value AS INTEGER), 1) = 0 + ) + ), + viability_required AS ( + SELECT 1 + FROM field_modes AS day_field + JOIN field_modes AS weekday_field + ON weekday_field.field_index = 4 + WHERE day_field.field_index = 2 + AND day_field.unrestricted = 0 + AND ( + weekday_field.unrestricted = 1 + OR day_field.starts_with_wildcard = 1 + OR weekday_field.starts_with_wildcard = 1 + ) + ), + viable_day_month AS ( + SELECT 1 + FROM expanded_values AS day_value + JOIN expanded_values AS month_value + ON month_value.field_index = 3 + WHERE day_value.field_index = 2 + AND day_value.value <= CASE month_value.value + WHEN 2 THEN 29 + WHEN 4 THEN 30 + WHEN 6 THEN 30 + WHEN 9 THEN 30 + WHEN 11 THEN 30 + ELSE 31 + END + LIMIT 1 + ) + SELECT 1 + FROM invalid_parts + UNION ALL + SELECT 1 + WHERE EXISTS (SELECT 1 FROM viability_required) + AND NOT EXISTS (SELECT 1 FROM viable_day_month) + LIMIT 1 + ) +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs cron expression must be semantically valid'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_reject_replace +BEFORE INSERT ON scheduled_jobs +WHEN EXISTS (SELECT 1 FROM scheduled_jobs WHERE id = NEW.id) +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_reject_identity_update +BEFORE UPDATE OF id, created_at ON scheduled_jobs +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_validate_version_update +BEFORE UPDATE ON scheduled_jobs +WHEN ( + ( + NEW.action_key IS NOT OLD.action_key + OR NEW.action_payload_json IS NOT OLD.action_payload_json + OR NEW.attempt_limit IS NOT OLD.attempt_limit + OR NEW.cancellation_policy IS NOT OLD.cancellation_policy + OR NEW.cron_expression IS NOT OLD.cron_expression + OR NEW.description IS NOT OLD.description + OR NEW.enabled IS NOT OLD.enabled + OR NEW.interval_ms IS NOT OLD.interval_ms + OR NEW.name IS NOT OLD.name + OR NEW.priority IS NOT OLD.priority + OR NEW.resource_class IS NOT OLD.resource_class + OR NEW.resource_keys_json IS NOT OLD.resource_keys_json + OR NEW.retry_safe IS NOT OLD.retry_safe + OR NEW.schedule_kind IS NOT OLD.schedule_kind + OR NEW.time_of_day IS NOT OLD.time_of_day + OR NEW.time_zone IS NOT OLD.time_zone + OR NEW.timeout_ms IS NOT OLD.timeout_ms + ) + AND ( + NEW.version <> OLD.version + 1 + OR NEW.updated_at < OLD.updated_at + ) + ) + OR ( + NEW.action_key IS OLD.action_key + AND NEW.action_payload_json IS OLD.action_payload_json + AND NEW.attempt_limit IS OLD.attempt_limit + AND NEW.cancellation_policy IS OLD.cancellation_policy + AND NEW.cron_expression IS OLD.cron_expression + AND NEW.description IS OLD.description + AND NEW.enabled IS OLD.enabled + AND NEW.interval_ms IS OLD.interval_ms + AND NEW.name IS OLD.name + AND NEW.priority IS OLD.priority + AND NEW.resource_class IS OLD.resource_class + AND NEW.resource_keys_json IS OLD.resource_keys_json + AND NEW.retry_safe IS OLD.retry_safe + AND NEW.schedule_kind IS OLD.schedule_kind + AND NEW.time_of_day IS OLD.time_of_day + AND NEW.time_zone IS OLD.time_zone + AND NEW.timeout_ms IS OLD.timeout_ms + AND ( + NEW.version <> OLD.version + OR NEW.updated_at <> OLD.updated_at + ) + AND NOT ( + OLD.enabled = 0 + AND NEW.enabled = 0 + AND NEW.next_run_at IS OLD.next_run_at + AND NEW.version = OLD.version + 1 + AND NEW.updated_at >= OLD.updated_at + AND EXISTS ( + SELECT 1 + FROM job_disable_intents AS replacement + JOIN job_disable_intents AS replaced + ON replaced.scheduled_job_id = replacement.scheduled_job_id + AND replaced.id <> replacement.id + WHERE replacement.scheduled_job_id = NEW.id + AND replacement.ended_at IS NULL + AND replacement.created_at = NEW.updated_at + AND replacement.created_at >= OLD.updated_at + AND replaced.created_at <= OLD.updated_at + AND replaced.ended_at = NEW.updated_at + AND replaced.ended_reason = 'replaced' + AND replaced.ended_by_kind = replacement.created_by_kind + AND replaced.ended_by_id = replacement.created_by_id + ) + ) + ) +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs version transition is invalid'); +END; +--> statement-breakpoint +CREATE TRIGGER scheduled_jobs_reject_delete +BEFORE DELETE ON scheduled_jobs +BEGIN + SELECT RAISE(ABORT, 'scheduled_jobs history cannot be deleted'); +END; +--> statement-breakpoint +CREATE TRIGGER job_disable_intents_reject_replace +BEFORE INSERT ON job_disable_intents +WHEN EXISTS (SELECT 1 FROM job_disable_intents WHERE id = NEW.id) +BEGIN + SELECT RAISE(ABORT, 'job_disable_intents identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER job_disable_intents_reject_content_update +BEFORE UPDATE OF + id, target_kind, scheduled_job_id, external_provider, external_job_id, + reason, created_by_kind, created_by_id, created_at, expires_at +ON job_disable_intents +BEGIN + SELECT RAISE(ABORT, 'job_disable_intents content is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER job_disable_intents_reject_closed_update +BEFORE UPDATE ON job_disable_intents +WHEN OLD.ended_at IS NOT NULL +BEGIN + SELECT RAISE(ABORT, 'closed job_disable_intents are immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER job_disable_intents_reject_delete +BEFORE DELETE ON job_disable_intents +BEGIN + SELECT RAISE(ABORT, 'job_disable_intents history cannot be deleted'); +END; +--> statement-breakpoint +CREATE TRIGGER worker_instances_reject_replace +BEFORE INSERT ON worker_instances +WHEN EXISTS (SELECT 1 FROM worker_instances WHERE id = NEW.id) +BEGIN + SELECT RAISE(ABORT, 'worker_instances identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER worker_instances_reject_identity_update +BEFORE UPDATE OF id, release_id, pid, capacity, started_at ON worker_instances +BEGIN + SELECT RAISE(ABORT, 'worker_instances identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER worker_instances_validate_lifecycle_update +BEFORE UPDATE ON worker_instances +WHEN NEW.heartbeat_at < OLD.heartbeat_at + OR NOT ( + (OLD.state = 'online' AND NEW.state IN ('online', 'draining')) + OR (OLD.state = 'draining' AND NEW.state IN ('draining', 'stopped')) + OR (OLD.state = 'stopped' AND NEW.state = 'stopped') + ) + OR (OLD.draining_at IS NOT NULL AND NEW.draining_at IS NOT OLD.draining_at) + OR (OLD.stopped_at IS NOT NULL AND NEW.stopped_at IS NOT OLD.stopped_at) + OR ( + OLD.state = 'stopped' + AND ( + NEW.heartbeat_at IS NOT OLD.heartbeat_at + OR NEW.draining_at IS NOT OLD.draining_at + OR NEW.stopped_at IS NOT OLD.stopped_at + ) + ) +BEGIN + SELECT RAISE(ABORT, 'worker_instances lifecycle transition is invalid'); +END; +--> statement-breakpoint +CREATE TRIGGER worker_instances_reject_active_delete +BEFORE DELETE ON worker_instances +WHEN OLD.state <> 'stopped' +BEGIN + SELECT RAISE(ABORT, 'active worker_instances cannot be deleted'); +END; +--> statement-breakpoint +CREATE TRIGGER job_worker_control_reject_replace +BEFORE INSERT ON job_worker_control +WHEN EXISTS (SELECT 1 FROM job_worker_control WHERE id = 1) +BEGIN + SELECT RAISE(ABORT, 'job_worker_control singleton already exists'); +END; +--> statement-breakpoint +CREATE TRIGGER job_worker_control_validate_update +BEFORE UPDATE ON job_worker_control +WHEN NEW.id <> OLD.id + OR NEW.updated_at < OLD.updated_at + OR NEW.version <> OLD.version + 1 + OR NEW.updated_by_kind IS NULL + OR NEW.updated_by_id IS NULL +BEGIN + SELECT RAISE(ABORT, 'job_worker_control transition is invalid'); +END; +--> statement-breakpoint +CREATE TRIGGER job_worker_control_reject_delete +BEFORE DELETE ON job_worker_control +BEGIN + SELECT RAISE(ABORT, 'job_worker_control singleton cannot be deleted'); +END; +--> statement-breakpoint +CREATE TRIGGER job_runs_validate_resource_keys_insert +BEFORE INSERT ON job_runs +WHEN json_array_length(NEW.resource_keys_json) > 32 + OR EXISTS ( + SELECT 1 + FROM json_each(NEW.resource_keys_json) AS entry + WHERE entry.type <> 'text' + OR length(CAST(entry.value AS TEXT)) NOT BETWEEN 1 AND 128 + OR CAST(entry.value AS TEXT) <> lower(CAST(entry.value AS TEXT)) + OR substr(CAST(entry.value AS TEXT), 1, 1) NOT GLOB '[a-z0-9]' + OR CAST(entry.value AS TEXT) GLOB '*[^a-z0-9._-]*' + ) + OR EXISTS ( + SELECT 1 + FROM json_each(NEW.resource_keys_json) AS current + JOIN json_each(NEW.resource_keys_json) AS previous + ON previous.key = current.key - 1 + WHERE CAST(current.value AS TEXT) <= CAST(previous.value AS TEXT) + ) +BEGIN + SELECT RAISE(ABORT, 'job_runs resource keys must be canonical'); +END; +--> statement-breakpoint +CREATE TRIGGER job_runs_reject_replace +BEFORE INSERT ON job_runs +WHEN EXISTS ( + SELECT 1 + FROM job_runs + WHERE id = NEW.id + OR ( + requested_by_kind = NEW.requested_by_kind + AND requested_by_id = NEW.requested_by_id + AND idempotency_key = NEW.idempotency_key + ) +) +BEGIN + SELECT RAISE(ABORT, 'job_runs identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER job_runs_reject_snapshot_update +BEFORE UPDATE OF + id, scheduled_job_id, scheduled_job_version, action_key, display_name, + trigger_type, requested_by_kind, requested_by_id, idempotency_key, + enqueue_sha256, payload_json, resource_class, resource_keys_json, + priority, timeout_ms, attempt_limit, retry_safe, cancellation_policy, + queued_at, scheduled_for_at +ON job_runs +BEGIN + SELECT RAISE(ABORT, 'job_runs execution snapshot is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER job_runs_validate_lifecycle_update +BEFORE UPDATE ON job_runs +WHEN ( + OLD.state IN ('cancelled', 'failed', 'succeeded', 'timed-out') + AND ( + NEW.state IS NOT OLD.state + OR NEW.attempt_count IS NOT OLD.attempt_count + OR NEW.available_at IS NOT OLD.available_at + OR NEW.cancel_requested_at IS NOT OLD.cancel_requested_at + OR NEW.cancel_requested_by_kind IS NOT OLD.cancel_requested_by_kind + OR NEW.cancel_requested_by_id IS NOT OLD.cancel_requested_by_id + OR NEW.finished_at IS NOT OLD.finished_at + OR NEW.first_started_at IS NOT OLD.first_started_at + OR NEW.heartbeat_at IS NOT OLD.heartbeat_at + OR NEW.last_attempt_started_at IS NOT OLD.last_attempt_started_at + OR NEW.lease_expires_at IS NOT OLD.lease_expires_at + OR NEW.lease_owner_id IS NOT OLD.lease_owner_id + OR NEW.lease_token IS NOT OLD.lease_token + OR NEW.result_json IS NOT OLD.result_json + OR NEW.state_version IS NOT OLD.state_version + OR NEW.terminal_code IS NOT OLD.terminal_code + OR NEW.terminal_message IS NOT OLD.terminal_message + OR NEW.updated_at IS NOT OLD.updated_at + ) + ) + OR NEW.updated_at < OLD.updated_at + OR NEW.attempt_count < OLD.attempt_count + OR NEW.attempt_count > OLD.attempt_count + 1 + OR ( + OLD.state = 'queued' + AND NEW.state = 'running' + AND NEW.attempt_count <> OLD.attempt_count + 1 + ) + OR ( + NOT (OLD.state = 'queued' AND NEW.state = 'running') + AND NEW.attempt_count <> OLD.attempt_count + ) + OR ( + OLD.state = 'queued' + AND NEW.state NOT IN ('queued', 'running', 'cancelled') + ) + OR ( + OLD.state = 'running' + AND NEW.state NOT IN ( + 'running', 'queued', 'succeeded', 'failed', 'cancelled', 'timed-out' + ) + ) + OR ( + OLD.state = 'running' + AND NEW.state = 'queued' + AND ( + OLD.retry_safe <> 1 + OR OLD.attempt_count >= OLD.attempt_limit + OR OLD.cancel_requested_at IS NOT NULL + ) + ) + OR ( + NEW.state = 'cancelled' + AND ( + OLD.cancellation_policy = 'never' + OR ( + OLD.state = 'running' + AND OLD.cancellation_policy <> 'cooperative' + ) + ) + ) + OR ( + OLD.cancel_requested_at IS NOT NULL + AND ( + NEW.cancel_requested_at IS NOT OLD.cancel_requested_at + OR NEW.cancel_requested_by_kind IS NOT OLD.cancel_requested_by_kind + OR NEW.cancel_requested_by_id IS NOT OLD.cancel_requested_by_id + ) + ) + OR ( + OLD.state = 'running' + AND OLD.cancel_requested_at IS NULL + AND NEW.cancel_requested_at IS NOT NULL + AND OLD.cancellation_policy <> 'cooperative' + ) + OR ( + OLD.first_started_at IS NOT NULL + AND NEW.first_started_at IS NOT OLD.first_started_at + ) + OR ( + OLD.last_attempt_started_at IS NOT NULL + AND NEW.last_attempt_started_at < OLD.last_attempt_started_at + ) + OR NEW.event_count < OLD.event_count + OR NEW.event_count > OLD.event_count + 1 + OR NEW.payload_event_count < OLD.payload_event_count + OR NEW.payload_event_count > OLD.payload_event_count + 1 + OR NEW.event_bytes < OLD.event_bytes + OR ( + ( + NEW.state IS NOT OLD.state + OR NEW.cancel_requested_at IS NOT OLD.cancel_requested_at + OR NEW.cancel_requested_by_kind IS NOT OLD.cancel_requested_by_kind + OR NEW.cancel_requested_by_id IS NOT OLD.cancel_requested_by_id + ) + AND NEW.state_version <> OLD.state_version + 1 + ) + OR ( + NEW.state IS OLD.state + AND NEW.cancel_requested_at IS OLD.cancel_requested_at + AND NEW.cancel_requested_by_kind IS OLD.cancel_requested_by_kind + AND NEW.cancel_requested_by_id IS OLD.cancel_requested_by_id + AND NEW.state_version <> OLD.state_version + ) +BEGIN + SELECT RAISE(ABORT, 'job_runs lifecycle transition is invalid'); +END; +--> statement-breakpoint +CREATE TRIGGER job_runs_reject_delete +BEFORE DELETE ON job_runs +BEGIN + SELECT RAISE(ABORT, 'job_runs history cannot be deleted'); +END; +--> statement-breakpoint +CREATE TRIGGER resource_leases_validate_insert +BEFORE INSERT ON resource_leases +WHEN NOT EXISTS ( + SELECT 1 + FROM job_runs AS run + JOIN worker_instances AS worker + ON worker.id = NEW.worker_instance_id + WHERE run.id = NEW.job_run_id + AND run.state = 'running' + AND run.lease_owner_id = NEW.worker_instance_id + AND run.lease_token = NEW.lease_token + AND run.lease_expires_at = NEW.expires_at + AND worker.state = 'online' + AND EXISTS ( + SELECT 1 + FROM json_each(run.resource_keys_json) AS resource + WHERE resource.value = NEW.resource_key + ) +) +BEGIN + SELECT RAISE(ABORT, 'resource_leases must match one active fenced claim'); +END; +--> statement-breakpoint +CREATE TRIGGER resource_leases_reject_identity_update +BEFORE UPDATE OF + resource_key, job_run_id, worker_instance_id, lease_token, acquired_at +ON resource_leases +BEGIN + SELECT RAISE(ABORT, 'resource_leases identity is immutable'); +END; +--> statement-breakpoint +CREATE TRIGGER resource_leases_validate_renewal_update +BEFORE UPDATE ON resource_leases +WHEN NEW.renewed_at < OLD.renewed_at + OR NEW.expires_at <= OLD.expires_at + OR NOT EXISTS ( + SELECT 1 + FROM job_runs AS run + WHERE run.id = OLD.job_run_id + AND run.state = 'running' + AND run.lease_owner_id = OLD.worker_instance_id + AND run.lease_token = OLD.lease_token + AND run.lease_expires_at = NEW.expires_at + ) +BEGIN + SELECT RAISE(ABORT, 'resource_leases renewal is not fenced'); +END; +--> statement-breakpoint +CREATE TRIGGER job_run_events_validate_insert +BEFORE INSERT ON job_run_events +WHEN NOT EXISTS ( + SELECT 1 + FROM job_runs AS run + WHERE run.id = NEW.job_run_id + AND run.event_count < 1000 + AND NEW.sequence = run.event_count + 1 + AND run.event_count = ( + SELECT count(*) + FROM job_run_events AS existing + WHERE existing.job_run_id = NEW.job_run_id + ) + AND NEW.attempt <= run.attempt_count + AND ( + NEW.kind IN ('cancel-requested', 'cancelled', 'queued') + OR NEW.attempt > 0 + ) + AND ( + NEW.kind <> 'queued' + OR NEW.worker_instance_id IS NULL + ) + AND NEW.occurred_at BETWEEN run.queued_at AND run.updated_at + AND ( + run.event_count = 0 + OR NEW.occurred_at >= ( + SELECT previous.occurred_at + FROM job_run_events AS previous + WHERE previous.job_run_id = NEW.job_run_id + ORDER BY previous.sequence DESC + LIMIT 1 + ) + ) + AND ( + (run.event_count = 0 AND NEW.kind = 'queued' AND NEW.attempt = 0) + OR (run.event_count > 0 AND NEW.kind <> 'queued') + ) + AND ( + NEW.kind NOT IN ('progress', 'stderr', 'stdout') + OR run.payload_event_count < 967 + ) + AND ( + NEW.kind NOT IN ('progress', 'stderr', 'stdout') + OR run.event_bytes + + length(CAST(COALESCE(NEW.message, '') AS BLOB)) + + length(CAST(COALESCE(NEW.progress_json, '') AS BLOB)) + <= 1007616 + ) + AND run.event_bytes + + length(CAST(COALESCE(NEW.message, '') AS BLOB)) + + length(CAST(COALESCE(NEW.progress_json, '') AS BLOB)) + <= 1048576 + AND ( + NEW.kind <> 'claimed' + OR ( + run.state = 'running' + AND NEW.attempt = run.attempt_count + ) + ) + AND ( + NEW.kind NOT IN ('progress', 'stderr', 'stdout', 'output-truncated') + OR run.state = 'running' + ) + AND ( + NEW.kind <> 'retry-scheduled' + OR run.state = 'queued' + ) + AND ( + NEW.kind <> 'cancel-requested' + OR run.cancel_requested_at IS NOT NULL + ) + AND ( + NEW.kind <> 'cancelled' + OR run.state = 'cancelled' + ) + AND ( + NEW.kind <> 'succeeded' + OR run.state = 'succeeded' + ) + AND ( + NEW.kind <> 'failed' + OR run.state IN ('running', 'failed') + ) + AND ( + NEW.kind <> 'timed-out' + OR run.state = 'timed-out' + ) +) +BEGIN + SELECT RAISE(ABORT, 'job_run_events must follow the parent run lifecycle'); +END; +--> statement-breakpoint +CREATE TRIGGER job_run_events_reject_replace +BEFORE INSERT ON job_run_events +WHEN EXISTS ( + SELECT 1 + FROM job_run_events + WHERE job_run_id = NEW.job_run_id + AND sequence = NEW.sequence +) +BEGIN + SELECT RAISE(ABORT, 'job_run_events are append-only'); +END; +--> statement-breakpoint +CREATE TRIGGER job_run_events_update_parent_counters +AFTER INSERT ON job_run_events +BEGIN + UPDATE job_runs + SET event_count = event_count + 1, + payload_event_count = payload_event_count + + CASE + WHEN NEW.kind IN ('progress', 'stderr', 'stdout') THEN 1 + ELSE 0 + END, + event_bytes = event_bytes + + length(CAST(COALESCE(NEW.message, '') AS BLOB)) + + length(CAST(COALESCE(NEW.progress_json, '') AS BLOB)) + WHERE id = NEW.job_run_id; +END; +--> statement-breakpoint +CREATE TRIGGER job_run_events_reject_update +BEFORE UPDATE ON job_run_events +BEGIN + SELECT RAISE(ABORT, 'job_run_events are append-only'); +END; +--> statement-breakpoint +CREATE TRIGGER job_run_events_reject_delete +BEFORE DELETE ON job_run_events +BEGIN + SELECT RAISE(ABORT, 'job_run_events are append-only'); +END; +--> statement-breakpoint CREATE UNIQUE INDEX `incident_observations_run_incident_unique` ON `incident_observations` (`monitor_run_id`,`incident_id`);--> statement-breakpoint CREATE INDEX `incident_observations_incident_observed_id_idx` ON `incident_observations` (`incident_id`,`observed_at`,`id`);--> statement-breakpoint CREATE INDEX `incident_observations_run_idx` ON `incident_observations` (`monitor_run_id`,`id`);--> statement-breakpoint diff --git a/greenfield/migrations/20260804022252_dashboard-foundation/snapshot.json b/greenfield/migrations/20260804022252_dashboard-foundation/snapshot.json index 163f8bcac..51574d4b9 100644 --- a/greenfield/migrations/20260804022252_dashboard-foundation/snapshot.json +++ b/greenfield/migrations/20260804022252_dashboard-foundation/snapshot.json @@ -1,7 +1,7 @@ { "version": "7", "dialect": "sqlite", - "id": "fccb7937-15cf-4628-a561-1093e2ee23ed", + "id": "21335169-a9af-4cae-a2c4-732aa16d7b80", "prevIds": [ "00000000-0000-0000-0000-000000000000" ], @@ -50,6 +50,22 @@ "name": "incidents", "entityType": "tables" }, + { + "name": "job_disable_intents", + "entityType": "tables" + }, + { + "name": "job_run_events", + "entityType": "tables" + }, + { + "name": "job_runs", + "entityType": "tables" + }, + { + "name": "job_worker_control", + "entityType": "tables" + }, { "name": "monitor_runs", "entityType": "tables" @@ -66,6 +82,14 @@ "name": "reports", "entityType": "tables" }, + { + "name": "resource_leases", + "entityType": "tables" + }, + { + "name": "scheduled_jobs", + "entityType": "tables" + }, { "name": "schema_migrations", "entityType": "tables" @@ -110,6 +134,10 @@ "name": "users", "entityType": "tables" }, + { + "name": "worker_instances", + "entityType": "tables" + }, { "type": "text", "notNull": true, @@ -1170,35 +1198,25 @@ "entityType": "columns", "table": "incidents" }, - { - "type": "integer", - "notNull": false, - "autoincrement": false, - "default": null, - "generated": null, - "name": "completed_at", - "entityType": "columns", - "table": "monitor_runs" - }, { "type": "integer", "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "complete_snapshot", + "name": "created_at", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "id", + "name": "created_by_id", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { "type": "text", @@ -1206,59 +1224,59 @@ "autoincrement": false, "default": null, "generated": null, - "name": "monitor_key", + "name": "created_by_kind", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { - "type": "text", + "type": "integer", "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "report_id", + "name": "ended_at", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { - "type": "integer", - "notNull": true, + "type": "text", + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "started_at", + "name": "ended_by_id", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "state", + "name": "ended_by_kind", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "submission_sha256", + "name": "ended_reason", "entityType": "columns", - "table": "monitor_runs" + "table": "job_disable_intents" }, { - "type": "text", - "notNull": true, + "type": "integer", + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "channel", + "name": "expires_at", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { "type": "text", @@ -1266,19 +1284,19 @@ "autoincrement": false, "default": null, "generated": null, - "name": "id", + "name": "external_job_id", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { - "type": "integer", + "type": "text", "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "incident_generation", + "name": "external_provider", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { "type": "text", @@ -1286,9 +1304,9 @@ "autoincrement": false, "default": null, "generated": null, - "name": "incident_id", + "name": "id", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { "type": "text", @@ -1296,9 +1314,9 @@ "autoincrement": false, "default": null, "generated": null, - "name": "kind", + "name": "reason", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { "type": "text", @@ -1306,9 +1324,9 @@ "autoincrement": false, "default": null, "generated": null, - "name": "link_url", + "name": "scheduled_job_id", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { "type": "text", @@ -1316,9 +1334,9 @@ "autoincrement": false, "default": null, "generated": null, - "name": "message", + "name": "target_kind", "entityType": "columns", - "table": "notifications" + "table": "job_disable_intents" }, { "type": "integer", @@ -1326,39 +1344,49 @@ "autoincrement": false, "default": null, "generated": null, - "name": "occurred_at", + "name": "attempt", "entityType": "columns", - "table": "notifications" + "table": "job_run_events" }, { - "type": "integer", - "notNull": false, + "type": "text", + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "read_at", + "name": "job_run_id", "entityType": "columns", - "table": "notifications" + "table": "job_run_events" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "report_id", + "name": "kind", "entityType": "columns", - "table": "notifications" + "table": "job_run_events" }, { "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "message", + "entityType": "columns", + "table": "job_run_events" + }, + { + "type": "integer", "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "severity", + "name": "occurred_at", "entityType": "columns", - "table": "notifications" + "table": "job_run_events" }, { "type": "text", @@ -1366,29 +1394,29 @@ "autoincrement": false, "default": null, "generated": null, - "name": "source", + "name": "progress_json", "entityType": "columns", - "table": "notifications" + "table": "job_run_events" }, { - "type": "text", + "type": "integer", "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "title", + "name": "sequence", "entityType": "columns", - "table": "notifications" + "table": "job_run_events" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "entity_id", + "name": "worker_instance_id", "entityType": "columns", - "table": "realtime_events" + "table": "job_run_events" }, { "type": "text", @@ -1396,29 +1424,29 @@ "autoincrement": false, "default": null, "generated": null, - "name": "entity_type", + "name": "action_key", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { "type": "integer", "notNull": true, "autoincrement": false, - "default": null, + "default": "0", "generated": null, - "name": "expires_at", + "name": "attempt_count", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { "type": "integer", - "notNull": false, - "autoincrement": true, + "notNull": true, + "autoincrement": false, "default": null, "generated": null, - "name": "id", + "name": "attempt_limit", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { "type": "integer", @@ -1426,9 +1454,9 @@ "autoincrement": false, "default": null, "generated": null, - "name": "occurred_at", + "name": "available_at", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { "type": "text", @@ -1436,49 +1464,49 @@ "autoincrement": false, "default": null, "generated": null, - "name": "operation", + "name": "cancellation_policy", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { - "type": "text", - "notNull": true, + "type": "integer", + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "payload_json", + "name": "cancel_requested_at", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "topic", + "name": "cancel_requested_by_id", "entityType": "columns", - "table": "realtime_events" + "table": "job_runs" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "body_markdown", + "name": "cancel_requested_by_kind", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "id", + "name": "display_name", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { "type": "text", @@ -1486,59 +1514,59 @@ "autoincrement": false, "default": null, "generated": null, - "name": "kind", + "name": "enqueue_sha256", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { - "type": "text", + "type": "integer", "notNull": true, "autoincrement": false, - "default": "'{}'", + "default": "0", "generated": null, - "name": "metadata_json", + "name": "event_bytes", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { "type": "integer", "notNull": true, "autoincrement": false, - "default": null, + "default": "0", "generated": null, - "name": "occurred_at", + "name": "event_count", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { - "type": "text", - "notNull": true, + "type": "integer", + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "source", + "name": "finished_at", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { - "type": "text", + "type": "integer", "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "source_job_id", + "name": "first_started_at", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { - "type": "text", - "notNull": true, + "type": "integer", + "notNull": false, "autoincrement": false, - "default": "'ok'", + "default": null, "generated": null, - "name": "status", + "name": "heartbeat_at", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { "type": "text", @@ -1546,9 +1574,9 @@ "autoincrement": false, "default": null, "generated": null, - "name": "summary", + "name": "id", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { "type": "text", @@ -1556,29 +1584,29 @@ "autoincrement": false, "default": null, "generated": null, - "name": "title", + "name": "idempotency_key", "entityType": "columns", - "table": "reports" + "table": "job_runs" }, { "type": "integer", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "applied_at", + "name": "last_attempt_started_at", "entityType": "columns", - "table": "schema_migrations" + "table": "job_runs" }, { - "type": "text", - "notNull": true, + "type": "integer", + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "checksum", + "name": "lease_expires_at", "entityType": "columns", - "table": "schema_migrations" + "table": "job_runs" }, { "type": "text", @@ -1586,29 +1614,29 @@ "autoincrement": false, "default": null, "generated": null, - "name": "id", + "name": "lease_owner_id", "entityType": "columns", - "table": "schema_migrations" + "table": "job_runs" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "release_id", + "name": "lease_token", "entityType": "columns", - "table": "schema_migrations" + "table": "job_runs" }, { - "type": "text", + "type": "integer", "notNull": true, "autoincrement": false, - "default": null, + "default": "0", "generated": null, - "name": "cron_job_id", + "name": "payload_event_count", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "text", @@ -1616,19 +1644,19 @@ "autoincrement": false, "default": null, "generated": null, - "name": "kind", + "name": "payload_json", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { - "type": "text", - "notNull": false, + "type": "integer", + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "model", + "name": "priority", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "integer", @@ -1636,97 +1664,1077 @@ "autoincrement": false, "default": null, "generated": null, - "name": "recurring", + "name": "queued_at", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "schedule_summary", + "name": "requested_by_id", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "session_target", + "name": "requested_by_kind", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "task_id", + "name": "resource_class", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "text", - "notNull": false, + "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "thinking", + "name": "resource_keys_json", "entityType": "columns", - "table": "task_automation_profiles" + "table": "job_runs" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "actor_id", + "name": "result_json", "entityType": "columns", - "table": "task_events" + "table": "job_runs" }, { - "type": "text", + "type": "integer", "notNull": true, "autoincrement": false, "default": null, "generated": null, - "name": "actor_kind", + "name": "retry_safe", "entityType": "columns", - "table": "task_events" + "table": "job_runs" }, { "type": "integer", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "created_at", + "name": "scheduled_for_at", "entityType": "columns", - "table": "task_events" + "table": "job_runs" }, { "type": "text", - "notNull": true, + "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "event_type", + "name": "scheduled_job_id", "entityType": "columns", - "table": "task_events" + "table": "job_runs" }, { - "type": "text", + "type": "integer", "notNull": false, "autoincrement": false, "default": null, "generated": null, - "name": "id", + "name": "scheduled_job_version", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "state", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": "1", + "generated": null, + "name": "state_version", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "terminal_code", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "terminal_message", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "timeout_ms", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "trigger_type", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_at", + "entityType": "columns", + "table": "job_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "claiming_paused", + "entityType": "columns", + "table": "job_worker_control" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "job_worker_control" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_at", + "entityType": "columns", + "table": "job_worker_control" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_by_id", + "entityType": "columns", + "table": "job_worker_control" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_by_kind", + "entityType": "columns", + "table": "job_worker_control" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "version", + "entityType": "columns", + "table": "job_worker_control" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "completed_at", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "complete_snapshot", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "monitor_key", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "report_id", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "started_at", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "state", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "submission_sha256", + "entityType": "columns", + "table": "monitor_runs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "channel", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "incident_generation", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "incident_id", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "kind", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "link_url", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "message", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "occurred_at", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "read_at", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "report_id", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "severity", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "source", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "title", + "entityType": "columns", + "table": "notifications" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "entity_id", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "entity_type", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "expires_at", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": true, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "occurred_at", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "operation", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "payload_json", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "topic", + "entityType": "columns", + "table": "realtime_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "body_markdown", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "kind", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": "'{}'", + "generated": null, + "name": "metadata_json", + "entityType": "columns", + "table": "reports" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "occurred_at", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "source", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "source_job_id", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": "'ok'", + "generated": null, + "name": "status", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "summary", + "entityType": "columns", + "table": "reports" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "title", + "entityType": "columns", + "table": "reports" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "acquired_at", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "expires_at", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "job_run_id", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "lease_token", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "renewed_at", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "resource_key", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "worker_instance_id", + "entityType": "columns", + "table": "resource_leases" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "action_key", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "action_payload_json", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "attempt_limit", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "cancellation_policy", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "created_at", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "cron_expression", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "description", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "enabled", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "interval_ms", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "name", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "next_run_at", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "priority", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "resource_class", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "resource_keys_json", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "retry_safe", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "schedule_kind", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "time_of_day", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "time_zone", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "timeout_ms", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "updated_at", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "version", + "entityType": "columns", + "table": "scheduled_jobs" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "applied_at", + "entityType": "columns", + "table": "schema_migrations" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "checksum", + "entityType": "columns", + "table": "schema_migrations" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "schema_migrations" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "release_id", + "entityType": "columns", + "table": "schema_migrations" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "cron_job_id", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "kind", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "model", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "recurring", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "schedule_summary", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "session_target", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "task_id", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "thinking", + "entityType": "columns", + "table": "task_automation_profiles" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "actor_id", + "entityType": "columns", + "table": "task_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "actor_kind", + "entityType": "columns", + "table": "task_events" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "created_at", + "entityType": "columns", + "table": "task_events" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "event_type", + "entityType": "columns", + "table": "task_events" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", "entityType": "columns", "table": "task_events" }, @@ -2380,6 +3388,96 @@ "entityType": "columns", "table": "users" }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "capacity", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "draining_at", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "heartbeat_at", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "id", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "pid", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "release_id", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "integer", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "started_at", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "text", + "notNull": true, + "autoincrement": false, + "default": null, + "generated": null, + "name": "state", + "entityType": "columns", + "table": "worker_instances" + }, + { + "type": "integer", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "stopped_at", + "entityType": "columns", + "table": "worker_instances" + }, { "columns": [ "pending_login_id" @@ -2530,6 +3628,81 @@ "entityType": "fks", "table": "incident_observations" }, + { + "columns": [ + "scheduled_job_id" + ], + "tableTo": "scheduled_jobs", + "columnsTo": [ + "id" + ], + "onUpdate": "RESTRICT", + "onDelete": "RESTRICT", + "nameExplicit": false, + "name": "fk_job_disable_intents_scheduled_job_id_scheduled_jobs_id_fk", + "entityType": "fks", + "table": "job_disable_intents" + }, + { + "columns": [ + "job_run_id" + ], + "tableTo": "job_runs", + "columnsTo": [ + "id" + ], + "onUpdate": "RESTRICT", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_job_run_events_job_run_id_job_runs_id_fk", + "entityType": "fks", + "table": "job_run_events" + }, + { + "columns": [ + "worker_instance_id" + ], + "tableTo": "worker_instances", + "columnsTo": [ + "id" + ], + "onUpdate": "RESTRICT", + "onDelete": "SET NULL", + "nameExplicit": false, + "name": "fk_job_run_events_worker_instance_id_worker_instances_id_fk", + "entityType": "fks", + "table": "job_run_events" + }, + { + "columns": [ + "lease_owner_id" + ], + "tableTo": "worker_instances", + "columnsTo": [ + "id" + ], + "onUpdate": "RESTRICT", + "onDelete": "RESTRICT", + "nameExplicit": false, + "name": "fk_job_runs_lease_owner_id_worker_instances_id_fk", + "entityType": "fks", + "table": "job_runs" + }, + { + "columns": [ + "scheduled_job_id" + ], + "tableTo": "scheduled_jobs", + "columnsTo": [ + "id" + ], + "onUpdate": "RESTRICT", + "onDelete": "RESTRICT", + "nameExplicit": false, + "name": "fk_job_runs_scheduled_job_id_scheduled_jobs_id_fk", + "entityType": "fks", + "table": "job_runs" + }, { "columns": [ "report_id" @@ -2547,33 +3720,63 @@ }, { "columns": [ - "incident_id" + "incident_id" + ], + "tableTo": "incidents", + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "RESTRICT", + "nameExplicit": false, + "name": "fk_notifications_incident_id_incidents_id_fk", + "entityType": "fks", + "table": "notifications" + }, + { + "columns": [ + "report_id" + ], + "tableTo": "reports", + "columnsTo": [ + "id" + ], + "onUpdate": "NO ACTION", + "onDelete": "CASCADE", + "nameExplicit": false, + "name": "fk_notifications_report_id_reports_id_fk", + "entityType": "fks", + "table": "notifications" + }, + { + "columns": [ + "job_run_id" ], - "tableTo": "incidents", + "tableTo": "job_runs", "columnsTo": [ "id" ], - "onUpdate": "NO ACTION", + "onUpdate": "RESTRICT", "onDelete": "RESTRICT", "nameExplicit": false, - "name": "fk_notifications_incident_id_incidents_id_fk", + "name": "fk_resource_leases_job_run_id_job_runs_id_fk", "entityType": "fks", - "table": "notifications" + "table": "resource_leases" }, { "columns": [ - "report_id" + "worker_instance_id" ], - "tableTo": "reports", + "tableTo": "worker_instances", "columnsTo": [ "id" ], - "onUpdate": "NO ACTION", - "onDelete": "CASCADE", + "onUpdate": "RESTRICT", + "onDelete": "RESTRICT", "nameExplicit": false, - "name": "fk_notifications_report_id_reports_id_fk", + "name": "fk_resource_leases_worker_instance_id_worker_instances_id_fk", "entityType": "fks", - "table": "notifications" + "table": "resource_leases" }, { "columns": [ @@ -2690,6 +3893,16 @@ "entityType": "pks", "table": "automation_principal_capabilities" }, + { + "columns": [ + "job_run_id", + "sequence" + ], + "nameExplicit": true, + "name": "job_run_events_pk", + "entityType": "pks", + "table": "job_run_events" + }, { "columns": [ "task_id", @@ -2790,6 +4003,33 @@ "table": "incidents", "entityType": "pks" }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "job_disable_intents_pk", + "table": "job_disable_intents", + "entityType": "pks" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "job_runs_pk", + "table": "job_runs", + "entityType": "pks" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "job_worker_control_pk", + "table": "job_worker_control", + "entityType": "pks" + }, { "columns": [ "id" @@ -2826,6 +4066,24 @@ "table": "reports", "entityType": "pks" }, + { + "columns": [ + "resource_key" + ], + "nameExplicit": false, + "name": "resource_leases_pk", + "table": "resource_leases", + "entityType": "pks" + }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "scheduled_jobs_pk", + "table": "scheduled_jobs", + "entityType": "pks" + }, { "columns": [ "id" @@ -2916,6 +4174,15 @@ "table": "users", "entityType": "pks" }, + { + "columns": [ + "id" + ], + "nameExplicit": false, + "name": "worker_instances_pk", + "table": "worker_instances", + "entityType": "pks" + }, { "columns": [ { @@ -3169,43 +4436,237 @@ "isExpression": false }, { - "value": "bucket_key", + "value": "bucket_key", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "auth_rate_limit_buckets_kind_updated_at_idx", + "entityType": "indexes", + "table": "auth_rate_limit_buckets" + }, + { + "columns": [ + { + "value": "expires_at", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "auth_sessions_expires_at_idx", + "entityType": "indexes", + "table": "auth_sessions" + }, + { + "columns": [ + { + "value": "user_id", + "isExpression": false + }, + { + "value": "last_seen_at", + "isExpression": false + }, + { + "value": "created_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "auth_sessions_user_last_seen_idx", + "entityType": "indexes", + "table": "auth_sessions" + }, + { + "columns": [ + { + "value": "validator_hash", + "isExpression": false + } + ], + "isUnique": true, + "where": null, + "origin": "manual", + "name": "auth_sessions_validator_hash_unique", + "entityType": "indexes", + "table": "auth_sessions" + }, + { + "columns": [ + { + "value": "principal_id", + "isExpression": false + }, + { + "value": "created_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "automation_credentials_principal_created_idx", + "entityType": "indexes", + "table": "automation_credentials" + }, + { + "columns": [ + { + "value": "principal_id", + "isExpression": false + }, + { + "value": "created_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": "\"automation_credentials\".\"revoked_at\" IS NULL", + "origin": "manual", + "name": "automation_credentials_active_principal_created_idx", + "entityType": "indexes", + "table": "automation_credentials" + }, + { + "columns": [ + { + "value": "replaces_credential_id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "automation_credentials_replacement_idx", + "entityType": "indexes", + "table": "automation_credentials" + }, + { + "columns": [ + { + "value": "replaces_credential_id", + "isExpression": false + } + ], + "isUnique": true, + "where": "\"automation_credentials\".\"replaces_credential_id\" IS NOT NULL AND \"automation_credentials\".\"revoked_at\" IS NULL", + "origin": "manual", + "name": "automation_credentials_active_replacement_unique", + "entityType": "indexes", + "table": "automation_credentials" + }, + { + "columns": [ + { + "value": "prefix", + "isExpression": false + } + ], + "isUnique": true, + "where": null, + "origin": "manual", + "name": "automation_credentials_prefix_unique", + "entityType": "indexes", + "table": "automation_credentials" + }, + { + "columns": [ + { + "value": "validator_version", + "isExpression": false + }, + { + "value": "validator_hash", + "isExpression": false + } + ], + "isUnique": true, + "where": null, + "origin": "manual", + "name": "automation_credentials_validator_unique", + "entityType": "indexes", + "table": "automation_credentials" + }, + { + "columns": [ + { + "value": "created_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "automation_principals_created_id_idx", + "entityType": "indexes", + "table": "automation_principals" + }, + { + "columns": [ + { + "value": "created_at", + "isExpression": false + }, + { + "value": "id", "isExpression": false } ], "isUnique": false, - "where": null, + "where": "\"automation_principals\".\"disabled_at\" IS NULL", "origin": "manual", - "name": "auth_rate_limit_buckets_kind_updated_at_idx", + "name": "automation_principals_active_created_id_idx", "entityType": "indexes", - "table": "auth_rate_limit_buckets" + "table": "automation_principals" }, { "columns": [ { - "value": "expires_at", + "value": "monitor_run_id", + "isExpression": false + }, + { + "value": "incident_id", "isExpression": false } ], - "isUnique": false, + "isUnique": true, "where": null, "origin": "manual", - "name": "auth_sessions_expires_at_idx", + "name": "incident_observations_run_incident_unique", "entityType": "indexes", - "table": "auth_sessions" + "table": "incident_observations" }, { "columns": [ { - "value": "user_id", - "isExpression": false - }, - { - "value": "last_seen_at", + "value": "incident_id", "isExpression": false }, { - "value": "created_at", + "value": "observed_at", "isExpression": false }, { @@ -3216,130 +4677,138 @@ "isUnique": false, "where": null, "origin": "manual", - "name": "auth_sessions_user_last_seen_idx", + "name": "incident_observations_incident_observed_id_idx", "entityType": "indexes", - "table": "auth_sessions" + "table": "incident_observations" }, { "columns": [ { - "value": "validator_hash", + "value": "monitor_run_id", + "isExpression": false + }, + { + "value": "id", "isExpression": false } ], - "isUnique": true, + "isUnique": false, "where": null, "origin": "manual", - "name": "auth_sessions_validator_hash_unique", + "name": "incident_observations_run_idx", "entityType": "indexes", - "table": "auth_sessions" + "table": "incident_observations" }, { "columns": [ { - "value": "principal_id", - "isExpression": false - }, - { - "value": "created_at", + "value": "monitor_key", "isExpression": false }, { - "value": "id", + "value": "fingerprint", "isExpression": false } ], - "isUnique": false, + "isUnique": true, "where": null, "origin": "manual", - "name": "automation_credentials_principal_created_idx", + "name": "incidents_monitor_fingerprint_unique", "entityType": "indexes", - "table": "automation_credentials" + "table": "incidents" }, { "columns": [ { - "value": "principal_id", - "isExpression": false - }, - { - "value": "created_at", + "value": "monitor_key", "isExpression": false }, { - "value": "id", + "value": "last_seen_at", "isExpression": false } ], "isUnique": false, - "where": "\"automation_credentials\".\"revoked_at\" IS NULL", + "where": "\"incidents\".\"state\" = 'active'", "origin": "manual", - "name": "automation_credentials_active_principal_created_idx", + "name": "incidents_active_monitor_seen_idx", "entityType": "indexes", - "table": "automation_credentials" + "table": "incidents" }, { "columns": [ { - "value": "replaces_credential_id", + "value": "last_seen_at", + "isExpression": false + }, + { + "value": "id", "isExpression": false } ], "isUnique": false, "where": null, "origin": "manual", - "name": "automation_credentials_replacement_idx", + "name": "incidents_last_seen_id_idx", "entityType": "indexes", - "table": "automation_credentials" + "table": "incidents" }, { "columns": [ { - "value": "replaces_credential_id", + "value": "scheduled_job_id", "isExpression": false } ], "isUnique": true, - "where": "\"automation_credentials\".\"replaces_credential_id\" IS NOT NULL AND \"automation_credentials\".\"revoked_at\" IS NULL", + "where": "\"job_disable_intents\".\"scheduled_job_id\" IS NOT NULL AND \"job_disable_intents\".\"ended_at\" IS NULL", "origin": "manual", - "name": "automation_credentials_active_replacement_unique", + "name": "job_disable_intents_active_schedule_unique", "entityType": "indexes", - "table": "automation_credentials" + "table": "job_disable_intents" }, { "columns": [ { - "value": "prefix", + "value": "external_provider", + "isExpression": false + }, + { + "value": "external_job_id", "isExpression": false } ], "isUnique": true, - "where": null, + "where": "\"job_disable_intents\".\"external_job_id\" IS NOT NULL AND \"job_disable_intents\".\"ended_at\" IS NULL", "origin": "manual", - "name": "automation_credentials_prefix_unique", + "name": "job_disable_intents_active_external_unique", "entityType": "indexes", - "table": "automation_credentials" + "table": "job_disable_intents" }, { "columns": [ { - "value": "validator_version", + "value": "expires_at", "isExpression": false }, { - "value": "validator_hash", + "value": "id", "isExpression": false } ], - "isUnique": true, - "where": null, + "isUnique": false, + "where": "\"job_disable_intents\".\"expires_at\" IS NOT NULL AND \"job_disable_intents\".\"ended_at\" IS NULL", "origin": "manual", - "name": "automation_credentials_validator_unique", + "name": "job_disable_intents_active_expiry_idx", "entityType": "indexes", - "table": "automation_credentials" + "table": "job_disable_intents" }, { "columns": [ + { + "value": "scheduled_job_id", + "isExpression": false + }, { "value": "created_at", "isExpression": false @@ -3352,12 +4821,20 @@ "isUnique": false, "where": null, "origin": "manual", - "name": "automation_principals_created_id_idx", + "name": "job_disable_intents_schedule_created_id_idx", "entityType": "indexes", - "table": "automation_principals" + "table": "job_disable_intents" }, { "columns": [ + { + "value": "external_provider", + "isExpression": false + }, + { + "value": "external_job_id", + "isExpression": false + }, { "value": "created_at", "isExpression": false @@ -3368,56 +4845,100 @@ } ], "isUnique": false, - "where": "\"automation_principals\".\"disabled_at\" IS NULL", + "where": null, "origin": "manual", - "name": "automation_principals_active_created_id_idx", + "name": "job_disable_intents_external_created_id_idx", "entityType": "indexes", - "table": "automation_principals" + "table": "job_disable_intents" }, { "columns": [ { - "value": "monitor_run_id", + "value": "occurred_at", "isExpression": false }, { - "value": "incident_id", + "value": "job_run_id", + "isExpression": false + }, + { + "value": "sequence", "isExpression": false } ], - "isUnique": true, + "isUnique": false, "where": null, "origin": "manual", - "name": "incident_observations_run_incident_unique", + "name": "job_run_events_occurred_run_sequence_idx", "entityType": "indexes", - "table": "incident_observations" + "table": "job_run_events" }, { "columns": [ { - "value": "incident_id", + "value": "requested_by_kind", "isExpression": false }, { - "value": "observed_at", + "value": "requested_by_id", "isExpression": false }, { - "value": "id", + "value": "idempotency_key", "isExpression": false } ], - "isUnique": false, + "isUnique": true, "where": null, "origin": "manual", - "name": "incident_observations_incident_observed_id_idx", + "name": "job_runs_idempotency_unique", "entityType": "indexes", - "table": "incident_observations" + "table": "job_runs" }, { "columns": [ { - "value": "monitor_run_id", + "value": "\"available_at\" asc", + "isExpression": true + }, + { + "value": "\"priority\" desc", + "isExpression": true + }, + { + "value": "\"queued_at\" asc", + "isExpression": true + }, + { + "value": "\"id\" asc", + "isExpression": true + } + ], + "isUnique": false, + "where": "\"job_runs\".\"state\" = 'queued'", + "origin": "manual", + "name": "job_runs_claim_idx", + "entityType": "indexes", + "table": "job_runs" + }, + { + "columns": [ + { + "value": "scheduled_job_id", + "isExpression": false + } + ], + "isUnique": true, + "where": "\"job_runs\".\"scheduled_job_id\" IS NOT NULL AND \"job_runs\".\"state\" IN ('queued', 'running')", + "origin": "manual", + "name": "job_runs_one_active_schedule_idx", + "entityType": "indexes", + "table": "job_runs" + }, + { + "columns": [ + { + "value": "queued_at", "isExpression": false }, { @@ -3428,50 +4949,54 @@ "isUnique": false, "where": null, "origin": "manual", - "name": "incident_observations_run_idx", + "name": "job_runs_queued_id_idx", "entityType": "indexes", - "table": "incident_observations" + "table": "job_runs" }, { "columns": [ { - "value": "monitor_key", + "value": "scheduled_job_id", "isExpression": false }, { - "value": "fingerprint", + "value": "queued_at", + "isExpression": false + }, + { + "value": "id", "isExpression": false } ], - "isUnique": true, + "isUnique": false, "where": null, "origin": "manual", - "name": "incidents_monitor_fingerprint_unique", + "name": "job_runs_schedule_queued_id_idx", "entityType": "indexes", - "table": "incidents" + "table": "job_runs" }, { "columns": [ { - "value": "monitor_key", + "value": "lease_expires_at", "isExpression": false }, { - "value": "last_seen_at", + "value": "id", "isExpression": false } ], "isUnique": false, - "where": "\"incidents\".\"state\" = 'active'", + "where": "\"job_runs\".\"state\" = 'running'", "origin": "manual", - "name": "incidents_active_monitor_seen_idx", + "name": "job_runs_running_lease_idx", "entityType": "indexes", - "table": "incidents" + "table": "job_runs" }, { "columns": [ { - "value": "last_seen_at", + "value": "lease_owner_id", "isExpression": false }, { @@ -3480,11 +5005,11 @@ } ], "isUnique": false, - "where": null, + "where": "\"job_runs\".\"state\" = 'running'", "origin": "manual", - "name": "incidents_last_seen_id_idx", + "name": "job_runs_running_owner_id_idx", "entityType": "indexes", - "table": "incidents" + "table": "job_runs" }, { "columns": [ @@ -3637,7 +5162,87 @@ "isExpression": false }, { - "value": "occurred_at", + "value": "occurred_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "reports_kind_occurred_id_idx", + "entityType": "indexes", + "table": "reports" + }, + { + "columns": [ + { + "value": "source", + "isExpression": false + }, + { + "value": "source_job_id", + "isExpression": false + }, + { + "value": "occurred_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "reports_source_job_occurred_id_idx", + "entityType": "indexes", + "table": "reports" + }, + { + "columns": [ + { + "value": "expires_at", + "isExpression": false + }, + { + "value": "resource_key", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "resource_leases_expiry_key_idx", + "entityType": "indexes", + "table": "resource_leases" + }, + { + "columns": [ + { + "value": "job_run_id", + "isExpression": false + }, + { + "value": "resource_key", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "resource_leases_run_key_idx", + "entityType": "indexes", + "table": "resource_leases" + }, + { + "columns": [ + { + "value": "next_run_at", "isExpression": false }, { @@ -3646,24 +5251,16 @@ } ], "isUnique": false, - "where": null, + "where": "\"scheduled_jobs\".\"enabled\" = 1", "origin": "manual", - "name": "reports_kind_occurred_id_idx", + "name": "scheduled_jobs_due_idx", "entityType": "indexes", - "table": "reports" + "table": "scheduled_jobs" }, { "columns": [ { - "value": "source", - "isExpression": false - }, - { - "value": "source_job_id", - "isExpression": false - }, - { - "value": "occurred_at", + "value": "updated_at", "isExpression": false }, { @@ -3674,9 +5271,9 @@ "isUnique": false, "where": null, "origin": "manual", - "name": "reports_source_job_occurred_id_idx", + "name": "scheduled_jobs_updated_id_idx", "entityType": "indexes", - "table": "reports" + "table": "scheduled_jobs" }, { "columns": [ @@ -4010,6 +5607,24 @@ "entityType": "indexes", "table": "users" }, + { + "columns": [ + { + "value": "heartbeat_at", + "isExpression": false + }, + { + "value": "id", + "isExpression": false + } + ], + "isUnique": false, + "where": null, + "origin": "manual", + "name": "worker_instances_heartbeat_id_idx", + "entityType": "indexes", + "table": "worker_instances" + }, { "value": "length(\"agent_id\") BETWEEN 1 AND 64 AND instr(\"agent_id\", char(0)) = 0 AND \"agent_id\" = lower(\"agent_id\") AND substr(\"agent_id\", 1, 1) GLOB '[a-z0-9]' AND \"agent_id\" NOT GLOB '*[^a-z0-9._-]*'", "name": "agent_task_runs_agent_id_check", @@ -4335,7 +5950,7 @@ "table": "automation_credentials" }, { - "value": "\"capability\" IN ('agents:read', 'agents:write', 'monitoring:write', 'notifications:read', 'notifications:write', 'reports:read', 'reports:write', 'tasks:read', 'tasks:write')", + "value": "\"capability\" IN ('agents:read', 'agents:write', 'jobs:read', 'jobs:write', 'monitoring:write', 'notifications:read', 'notifications:write', 'reports:read', 'reports:write', 'tasks:read', 'tasks:write')", "name": "automation_principal_capabilities_capability_check", "entityType": "checks", "table": "automation_principal_capabilities" @@ -4442,6 +6057,270 @@ "entityType": "checks", "table": "incidents" }, + { + "value": "\"created_at\" BETWEEN 0 AND 8640000000000000", + "name": "job_disable_intents_created_at_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "((\"created_by_kind\" = 'user' AND length(\"created_by_id\") = 36 AND instr(\"created_by_id\", char(0)) = 0 AND length(replace(\"created_by_id\", '-', '')) = 32 AND replace(\"created_by_id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"created_by_id\", 9, 1) = '-' AND substr(\"created_by_id\", 14, 1) = '-' AND substr(\"created_by_id\", 15, 1) = '7' AND substr(\"created_by_id\", 19, 1) = '-' AND substr(\"created_by_id\", 20, 1) GLOB '[89ab]' AND substr(\"created_by_id\", 24, 1) = '-') OR (\"created_by_kind\" = 'automation' AND length(\"created_by_id\") BETWEEN 1 AND 64 AND instr(\"created_by_id\", char(0)) = 0 AND \"created_by_id\" = lower(\"created_by_id\") AND substr(\"created_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"created_by_id\" NOT GLOB '*[^a-z0-9._-]*'))", + "name": "job_disable_intents_created_actor_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "(\"ended_at\" IS NULL AND \"ended_by_kind\" IS NULL AND \"ended_by_id\" IS NULL AND \"ended_reason\" IS NULL) OR (\"ended_at\" IS NOT NULL AND \"ended_at\" BETWEEN 0 AND 8640000000000000 AND \"ended_at\" >= \"created_at\" AND \"ended_by_kind\" IS NOT NULL AND \"ended_by_id\" IS NOT NULL AND \"ended_reason\" IN ('expired', 're-enabled', 'replaced') AND ((\"ended_by_kind\" = 'user' AND length(\"ended_by_id\") = 36 AND instr(\"ended_by_id\", char(0)) = 0 AND length(replace(\"ended_by_id\", '-', '')) = 32 AND replace(\"ended_by_id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"ended_by_id\", 9, 1) = '-' AND substr(\"ended_by_id\", 14, 1) = '-' AND substr(\"ended_by_id\", 15, 1) = '7' AND substr(\"ended_by_id\", 19, 1) = '-' AND substr(\"ended_by_id\", 20, 1) GLOB '[89ab]' AND substr(\"ended_by_id\", 24, 1) = '-') OR (\"ended_by_kind\" = 'automation' AND length(\"ended_by_id\") BETWEEN 1 AND 64 AND instr(\"ended_by_id\", char(0)) = 0 AND \"ended_by_id\" = lower(\"ended_by_id\") AND substr(\"ended_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"ended_by_id\" NOT GLOB '*[^a-z0-9._-]*') OR (\"ended_by_kind\" = 'system' AND length(\"ended_by_id\") BETWEEN 1 AND 128 AND instr(\"ended_by_id\", char(0)) = 0 AND \"ended_by_id\" = lower(\"ended_by_id\") AND substr(\"ended_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"ended_by_id\" NOT GLOB '*[^a-z0-9._-]*')) AND (\"ended_reason\" <> 'expired' OR (\"ended_by_kind\" = 'system' AND \"expires_at\" IS NOT NULL AND \"ended_at\" >= \"expires_at\")))", + "name": "job_disable_intents_end_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "\"expires_at\" IS NULL OR (\"expires_at\" BETWEEN 0 AND 8640000000000000 AND \"expires_at\" > \"created_at\")", + "name": "job_disable_intents_expiry_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "\"external_job_id\" IS NULL OR (length(\"external_job_id\") BETWEEN 1 AND 256 AND instr(\"external_job_id\", char(0)) = 0 AND length(trim(\"external_job_id\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0)", + "name": "job_disable_intents_external_job_id_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "length(\"id\") = 36 AND instr(\"id\", char(0)) = 0 AND length(replace(\"id\", '-', '')) = 32 AND replace(\"id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"id\", 9, 1) = '-' AND substr(\"id\", 14, 1) = '-' AND substr(\"id\", 15, 1) = '7' AND substr(\"id\", 19, 1) = '-' AND substr(\"id\", 20, 1) GLOB '[89ab]' AND substr(\"id\", 24, 1) = '-'", + "name": "job_disable_intents_id_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "length(\"reason\") BETWEEN 1 AND 1000 AND instr(\"reason\", char(0)) = 0 AND length(trim(\"reason\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND \"reason\" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST(\"reason\" AS BLOB)) <= 4000", + "name": "job_disable_intents_reason_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "(\"target_kind\" = 'dashboard-schedule' AND \"scheduled_job_id\" IS NOT NULL AND \"external_provider\" IS NULL AND \"external_job_id\" IS NULL) OR (\"target_kind\" = 'openclaw-cron' AND \"scheduled_job_id\" IS NULL AND \"external_provider\" = 'openclaw' AND \"external_job_id\" IS NOT NULL)", + "name": "job_disable_intents_target_check", + "entityType": "checks", + "table": "job_disable_intents" + }, + { + "value": "\"attempt\" BETWEEN 0 AND 10", + "name": "job_run_events_attempt_check", + "entityType": "checks", + "table": "job_run_events" + }, + { + "value": "\"kind\" IN ('cancel-requested', 'cancelled', 'claimed', 'failed', 'lease-expired', 'output-truncated', 'progress', 'queued', 'retry-scheduled', 'stderr', 'stdout', 'succeeded', 'timed-out')", + "name": "job_run_events_kind_check", + "entityType": "checks", + "table": "job_run_events" + }, + { + "value": "(\"message\" IS NULL OR (length(\"message\") BETWEEN 1 AND 4096 AND instr(\"message\", char(0)) = 0 AND length(trim(\"message\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND \"message\" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST(\"message\" AS BLOB)) <= 4096))", + "name": "job_run_events_message_check", + "entityType": "checks", + "table": "job_run_events" + }, + { + "value": "\"occurred_at\" BETWEEN 0 AND 8640000000000000", + "name": "job_run_events_occurred_at_check", + "entityType": "checks", + "table": "job_run_events" + }, + { + "value": "(\"kind\" = 'progress' AND \"progress_json\" IS NOT NULL AND length(CAST(\"progress_json\" AS BLOB)) <= 16384 AND CASE WHEN json_valid(\"progress_json\") THEN json_type(\"progress_json\") = 'object' ELSE 0 END) OR (\"kind\" IN ('stderr', 'stdout') AND \"message\" IS NOT NULL AND \"progress_json\" IS NULL) OR (\"kind\" NOT IN ('progress', 'stderr', 'stdout') AND \"progress_json\" IS NULL)", + "name": "job_run_events_payload_shape_check", + "entityType": "checks", + "table": "job_run_events" + }, + { + "value": "\"sequence\" BETWEEN 1 AND 1000", + "name": "job_run_events_sequence_check", + "entityType": "checks", + "table": "job_run_events" + }, + { + "value": "length(\"action_key\") BETWEEN 1 AND 128 AND instr(\"action_key\", char(0)) = 0 AND \"action_key\" = lower(\"action_key\") AND substr(\"action_key\", 1, 1) GLOB '[a-z0-9]' AND \"action_key\" NOT GLOB '*[^a-z0-9._-]*'", + "name": "job_runs_action_key_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"attempt_limit\" BETWEEN 1 AND 10 AND \"attempt_count\" BETWEEN 0 AND \"attempt_limit\" AND ((\"attempt_count\" = 0 AND \"first_started_at\" IS NULL AND \"last_attempt_started_at\" IS NULL) OR (\"attempt_count\" > 0 AND \"first_started_at\" IS NOT NULL AND \"last_attempt_started_at\" IS NOT NULL))", + "name": "job_runs_attempt_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"available_at\" BETWEEN 0 AND 8640000000000000 AND \"available_at\" >= \"queued_at\"", + "name": "job_runs_available_at_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"cancellation_policy\" IN ('cooperative', 'never', 'queued-only')", + "name": "job_runs_cancellation_policy_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "(\"state\" <> 'cancelled' AND \"cancel_requested_at\" IS NULL AND \"cancel_requested_by_kind\" IS NULL AND \"cancel_requested_by_id\" IS NULL) OR (\"cancellation_policy\" <> 'never' AND \"cancel_requested_at\" IS NOT NULL AND \"cancel_requested_at\" BETWEEN 0 AND 8640000000000000 AND \"cancel_requested_at\" >= \"queued_at\" AND \"cancel_requested_at\" <= \"updated_at\" AND \"cancel_requested_by_kind\" IS NOT NULL AND \"cancel_requested_by_id\" IS NOT NULL AND ((\"cancel_requested_by_kind\" = 'user' AND length(\"cancel_requested_by_id\") = 36 AND instr(\"cancel_requested_by_id\", char(0)) = 0 AND length(replace(\"cancel_requested_by_id\", '-', '')) = 32 AND replace(\"cancel_requested_by_id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"cancel_requested_by_id\", 9, 1) = '-' AND substr(\"cancel_requested_by_id\", 14, 1) = '-' AND substr(\"cancel_requested_by_id\", 15, 1) = '7' AND substr(\"cancel_requested_by_id\", 19, 1) = '-' AND substr(\"cancel_requested_by_id\", 20, 1) GLOB '[89ab]' AND substr(\"cancel_requested_by_id\", 24, 1) = '-') OR (\"cancel_requested_by_kind\" = 'automation' AND length(\"cancel_requested_by_id\") BETWEEN 1 AND 64 AND instr(\"cancel_requested_by_id\", char(0)) = 0 AND \"cancel_requested_by_id\" = lower(\"cancel_requested_by_id\") AND substr(\"cancel_requested_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"cancel_requested_by_id\" NOT GLOB '*[^a-z0-9._-]*') OR (\"cancel_requested_by_kind\" = 'system' AND length(\"cancel_requested_by_id\") BETWEEN 1 AND 128 AND instr(\"cancel_requested_by_id\", char(0)) = 0 AND \"cancel_requested_by_id\" = lower(\"cancel_requested_by_id\") AND substr(\"cancel_requested_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"cancel_requested_by_id\" NOT GLOB '*[^a-z0-9._-]*')))", + "name": "job_runs_cancel_request_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "length(\"display_name\") BETWEEN 1 AND 160 AND instr(\"display_name\", char(0)) = 0 AND length(trim(\"display_name\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND \"display_name\" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST(\"display_name\" AS BLOB)) <= 640", + "name": "job_runs_display_name_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "length(\"enqueue_sha256\") = 64 AND instr(\"enqueue_sha256\", char(0)) = 0 AND \"enqueue_sha256\" NOT GLOB '*[^0-9a-f]*'", + "name": "job_runs_enqueue_sha256_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"event_count\" BETWEEN 0 AND 1000 AND \"payload_event_count\" BETWEEN 0 AND 967 AND \"payload_event_count\" <= \"event_count\" AND \"event_bytes\" BETWEEN 0 AND 1048576", + "name": "job_runs_event_budget_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "length(\"id\") = 36 AND instr(\"id\", char(0)) = 0 AND length(replace(\"id\", '-', '')) = 32 AND replace(\"id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"id\", 9, 1) = '-' AND substr(\"id\", 14, 1) = '-' AND substr(\"id\", 15, 1) = '7' AND substr(\"id\", 19, 1) = '-' AND substr(\"id\", 20, 1) GLOB '[89ab]' AND substr(\"id\", 24, 1) = '-'", + "name": "job_runs_id_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "length(\"idempotency_key\") BETWEEN 32 AND 128 AND instr(\"idempotency_key\", char(0)) = 0 AND \"idempotency_key\" NOT GLOB '*[^A-Za-z0-9_-]*' AND (length(\"idempotency_key\") % 4 = 0 OR (length(\"idempotency_key\") % 4 = 2 AND substr(\"idempotency_key\", -1, 1) GLOB '[AQgw]') OR (length(\"idempotency_key\") % 4 = 3 AND substr(\"idempotency_key\", -1, 1) GLOB '[AEIMQUYcgkosw048]'))", + "name": "job_runs_idempotency_key_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "(\"state\" <> 'running' AND \"lease_owner_id\" IS NULL AND \"lease_token\" IS NULL AND \"lease_expires_at\" IS NULL AND \"heartbeat_at\" IS NULL) OR (\"state\" = 'running' AND \"lease_owner_id\" IS NOT NULL AND \"lease_token\" IS NOT NULL AND length(\"lease_token\") = 36 AND instr(\"lease_token\", char(0)) = 0 AND length(replace(\"lease_token\", '-', '')) = 32 AND replace(\"lease_token\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"lease_token\", 9, 1) = '-' AND substr(\"lease_token\", 14, 1) = '-' AND substr(\"lease_token\", 15, 1) = '7' AND substr(\"lease_token\", 19, 1) = '-' AND substr(\"lease_token\", 20, 1) GLOB '[89ab]' AND substr(\"lease_token\", 24, 1) = '-' AND \"lease_expires_at\" IS NOT NULL AND \"heartbeat_at\" IS NOT NULL AND \"heartbeat_at\" BETWEEN 0 AND 8640000000000000 AND \"lease_expires_at\" BETWEEN 0 AND 8640000000000000 AND \"heartbeat_at\" >= \"last_attempt_started_at\" AND \"lease_expires_at\" > \"heartbeat_at\")", + "name": "job_runs_lease_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "length(CAST(\"payload_json\" AS BLOB)) <= 65536 AND CASE WHEN json_valid(\"payload_json\") THEN json_type(\"payload_json\") = 'object' ELSE 0 END", + "name": "job_runs_payload_json_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"priority\" BETWEEN -100 AND 100", + "name": "job_runs_priority_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "((\"requested_by_kind\" = 'user' AND length(\"requested_by_id\") = 36 AND instr(\"requested_by_id\", char(0)) = 0 AND length(replace(\"requested_by_id\", '-', '')) = 32 AND replace(\"requested_by_id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"requested_by_id\", 9, 1) = '-' AND substr(\"requested_by_id\", 14, 1) = '-' AND substr(\"requested_by_id\", 15, 1) = '7' AND substr(\"requested_by_id\", 19, 1) = '-' AND substr(\"requested_by_id\", 20, 1) GLOB '[89ab]' AND substr(\"requested_by_id\", 24, 1) = '-') OR (\"requested_by_kind\" = 'automation' AND length(\"requested_by_id\") BETWEEN 1 AND 64 AND instr(\"requested_by_id\", char(0)) = 0 AND \"requested_by_id\" = lower(\"requested_by_id\") AND substr(\"requested_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"requested_by_id\" NOT GLOB '*[^a-z0-9._-]*') OR (\"requested_by_kind\" = 'system' AND length(\"requested_by_id\") BETWEEN 1 AND 128 AND instr(\"requested_by_id\", char(0)) = 0 AND \"requested_by_id\" = lower(\"requested_by_id\") AND substr(\"requested_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"requested_by_id\" NOT GLOB '*[^a-z0-9._-]*'))", + "name": "job_runs_requested_actor_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"resource_class\" IN ('exclusive', 'host-heavy', 'interactive', 'light', 'network')", + "name": "job_runs_resource_class_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "length(CAST(\"resource_keys_json\" AS BLOB)) <= 4096 AND CASE WHEN json_valid(\"resource_keys_json\") THEN json_type(\"resource_keys_json\") = 'array' ELSE 0 END", + "name": "job_runs_resource_keys_json_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"result_json\" IS NULL OR (length(CAST(\"result_json\" AS BLOB)) <= 65536 AND CASE WHEN json_valid(\"result_json\") THEN json_type(\"result_json\") = 'object' ELSE 0 END)", + "name": "job_runs_result_json_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"retry_safe\" IN (0, 1)", + "name": "job_runs_retry_safe_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "(\"trigger_type\" = 'schedule' AND \"scheduled_job_id\" IS NOT NULL AND \"scheduled_job_version\" BETWEEN 1 AND 9007199254740991 AND \"scheduled_for_at\" IS NOT NULL AND \"scheduled_for_at\" BETWEEN 0 AND 8640000000000000 AND \"scheduled_for_at\" <= \"queued_at\") OR (\"trigger_type\" = 'manual' AND \"scheduled_job_id\" IS NOT NULL AND \"scheduled_job_version\" BETWEEN 1 AND 9007199254740991 AND \"scheduled_for_at\" IS NULL) OR (\"trigger_type\" IN ('startup', 'system') AND \"scheduled_job_id\" IS NULL AND \"scheduled_job_version\" IS NULL AND \"scheduled_for_at\" IS NULL)", + "name": "job_runs_schedule_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"state\" IN ('cancelled', 'failed', 'queued', 'running', 'succeeded', 'timed-out') AND ((\"state\" = 'queued' AND \"finished_at\" IS NULL AND \"result_json\" IS NULL AND \"terminal_code\" IS NULL AND \"terminal_message\" IS NULL) OR (\"state\" = 'running' AND \"attempt_count\" > 0 AND \"finished_at\" IS NULL AND \"result_json\" IS NULL AND \"terminal_code\" IS NULL AND \"terminal_message\" IS NULL) OR (\"state\" = 'succeeded' AND \"attempt_count\" > 0 AND \"finished_at\" IS NOT NULL AND \"result_json\" IS NOT NULL AND \"terminal_code\" IS NULL AND \"terminal_message\" IS NULL) OR (\"state\" IN ('failed', 'timed-out') AND \"attempt_count\" > 0 AND \"finished_at\" IS NOT NULL AND \"result_json\" IS NULL AND \"terminal_code\" IS NOT NULL AND \"terminal_message\" IS NOT NULL) OR (\"state\" = 'cancelled' AND \"finished_at\" IS NOT NULL AND \"result_json\" IS NULL AND \"terminal_code\" IS NOT NULL AND \"terminal_message\" IS NOT NULL))", + "name": "job_runs_state_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"state_version\" BETWEEN 1 AND 9007199254740991", + "name": "job_runs_state_version_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "(\"terminal_code\" IS NULL OR (length(\"terminal_code\") BETWEEN 1 AND 128 AND instr(\"terminal_code\", char(0)) = 0 AND \"terminal_code\" = lower(\"terminal_code\") AND substr(\"terminal_code\", 1, 1) GLOB '[a-z0-9]' AND \"terminal_code\" NOT GLOB '*[^a-z0-9._/-]*'))", + "name": "job_runs_terminal_code_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "(\"terminal_message\" IS NULL OR (length(\"terminal_message\") BETWEEN 1 AND 2000 AND instr(\"terminal_message\", char(0)) = 0 AND length(trim(\"terminal_message\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND \"terminal_message\" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST(\"terminal_message\" AS BLOB)) <= 8000))", + "name": "job_runs_terminal_message_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"timeout_ms\" BETWEEN 1000 AND 86400000", + "name": "job_runs_timeout_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "\"queued_at\" BETWEEN 0 AND 8640000000000000 AND \"updated_at\" BETWEEN 0 AND 8640000000000000 AND \"updated_at\" >= \"queued_at\" AND (\"first_started_at\" IS NULL OR (\"first_started_at\" BETWEEN 0 AND 8640000000000000 AND \"first_started_at\" BETWEEN \"queued_at\" AND \"updated_at\")) AND (\"last_attempt_started_at\" IS NULL OR (\"first_started_at\" IS NOT NULL AND \"last_attempt_started_at\" BETWEEN 0 AND 8640000000000000 AND \"last_attempt_started_at\" BETWEEN \"first_started_at\" AND \"updated_at\")) AND (\"heartbeat_at\" IS NULL OR (\"last_attempt_started_at\" IS NOT NULL AND \"heartbeat_at\" BETWEEN 0 AND 8640000000000000 AND \"heartbeat_at\" BETWEEN \"last_attempt_started_at\" AND \"updated_at\")) AND (\"cancel_requested_at\" IS NULL OR (\"cancel_requested_at\" BETWEEN 0 AND 8640000000000000 AND \"cancel_requested_at\" BETWEEN \"queued_at\" AND \"updated_at\")) AND (\"finished_at\" IS NULL OR (\"finished_at\" BETWEEN 0 AND 8640000000000000 AND \"finished_at\" BETWEEN COALESCE(\"last_attempt_started_at\", \"queued_at\") AND \"updated_at\"))", + "name": "job_runs_time_check", + "entityType": "checks", + "table": "job_runs" + }, + { + "value": "(\"updated_by_kind\" IS NULL AND \"updated_by_id\" IS NULL) OR (\"updated_by_kind\" IS NOT NULL AND \"updated_by_id\" IS NOT NULL AND ((\"updated_by_kind\" = 'user' AND length(\"updated_by_id\") = 36 AND instr(\"updated_by_id\", char(0)) = 0 AND length(replace(\"updated_by_id\", '-', '')) = 32 AND replace(\"updated_by_id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"updated_by_id\", 9, 1) = '-' AND substr(\"updated_by_id\", 14, 1) = '-' AND substr(\"updated_by_id\", 15, 1) = '7' AND substr(\"updated_by_id\", 19, 1) = '-' AND substr(\"updated_by_id\", 20, 1) GLOB '[89ab]' AND substr(\"updated_by_id\", 24, 1) = '-') OR (\"updated_by_kind\" = 'automation' AND length(\"updated_by_id\") BETWEEN 1 AND 64 AND instr(\"updated_by_id\", char(0)) = 0 AND \"updated_by_id\" = lower(\"updated_by_id\") AND substr(\"updated_by_id\", 1, 1) GLOB '[a-z0-9]' AND \"updated_by_id\" NOT GLOB '*[^a-z0-9._-]*')))", + "name": "job_worker_control_actor_check", + "entityType": "checks", + "table": "job_worker_control" + }, + { + "value": "\"claiming_paused\" IN (0, 1)", + "name": "job_worker_control_claiming_paused_check", + "entityType": "checks", + "table": "job_worker_control" + }, + { + "value": "\"id\" = 1", + "name": "job_worker_control_id_check", + "entityType": "checks", + "table": "job_worker_control" + }, + { + "value": "\"updated_at\" BETWEEN 0 AND 8640000000000000", + "name": "job_worker_control_updated_at_check", + "entityType": "checks", + "table": "job_worker_control" + }, + { + "value": "\"version\" BETWEEN 1 AND 9007199254740991", + "name": "job_worker_control_version_check", + "entityType": "checks", + "table": "job_worker_control" + }, { "value": "\"complete_snapshot\" IN (0, 1)", "name": "monitor_runs_complete_snapshot_check", @@ -4532,6 +6411,138 @@ "entityType": "checks", "table": "reports" }, + { + "value": "length(\"lease_token\") = 36 AND instr(\"lease_token\", char(0)) = 0 AND length(replace(\"lease_token\", '-', '')) = 32 AND replace(\"lease_token\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"lease_token\", 9, 1) = '-' AND substr(\"lease_token\", 14, 1) = '-' AND substr(\"lease_token\", 15, 1) = '7' AND substr(\"lease_token\", 19, 1) = '-' AND substr(\"lease_token\", 20, 1) GLOB '[89ab]' AND substr(\"lease_token\", 24, 1) = '-'", + "name": "resource_leases_lease_token_check", + "entityType": "checks", + "table": "resource_leases" + }, + { + "value": "length(\"resource_key\") BETWEEN 1 AND 128 AND instr(\"resource_key\", char(0)) = 0 AND \"resource_key\" = lower(\"resource_key\") AND substr(\"resource_key\", 1, 1) GLOB '[a-z0-9]' AND \"resource_key\" NOT GLOB '*[^a-z0-9._-]*'", + "name": "resource_leases_resource_key_check", + "entityType": "checks", + "table": "resource_leases" + }, + { + "value": "\"acquired_at\" BETWEEN 0 AND 8640000000000000 AND \"renewed_at\" BETWEEN 0 AND 8640000000000000 AND \"expires_at\" BETWEEN 0 AND 8640000000000000 AND \"renewed_at\" >= \"acquired_at\" AND \"expires_at\" > \"renewed_at\"", + "name": "resource_leases_time_check", + "entityType": "checks", + "table": "resource_leases" + }, + { + "value": "length(\"action_key\") BETWEEN 1 AND 128 AND instr(\"action_key\", char(0)) = 0 AND \"action_key\" = lower(\"action_key\") AND substr(\"action_key\", 1, 1) GLOB '[a-z0-9]' AND \"action_key\" NOT GLOB '*[^a-z0-9._-]*'", + "name": "scheduled_jobs_action_key_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "length(CAST(\"action_payload_json\" AS BLOB)) <= 65536 AND CASE WHEN json_valid(\"action_payload_json\") THEN json_type(\"action_payload_json\") = 'object' ELSE 0 END", + "name": "scheduled_jobs_action_payload_json_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"attempt_limit\" BETWEEN 1 AND 10", + "name": "scheduled_jobs_attempt_limit_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"cancellation_policy\" IN ('cooperative', 'never', 'queued-only')", + "name": "scheduled_jobs_cancellation_policy_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"created_at\" BETWEEN 0 AND 8640000000000000", + "name": "scheduled_jobs_created_at_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "length(\"description\") BETWEEN 1 AND 1000 AND instr(\"description\", char(0)) = 0 AND length(trim(\"description\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND \"description\" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST(\"description\" AS BLOB)) <= 4000", + "name": "scheduled_jobs_description_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"enabled\" IN (0, 1)", + "name": "scheduled_jobs_enabled_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "length(\"id\") BETWEEN 1 AND 80 AND instr(\"id\", char(0)) = 0 AND \"id\" = lower(\"id\") AND substr(\"id\", 1, 1) GLOB '[a-z0-9]' AND \"id\" NOT GLOB '*[^a-z0-9._-]*'", + "name": "scheduled_jobs_id_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "length(\"name\") BETWEEN 1 AND 160 AND instr(\"name\", char(0)) = 0 AND length(trim(\"name\", char(9, 10, 11, 12, 13, 32, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288, 65279))) > 0 AND \"name\" NOT GLOB ('*[' || char(1) || '-' || char(31) || char(127) || '-' || char(159) || char(173) || char(1536) || '-' || char(1541) || char(1564) || char(1757) || char(1807) || char(2192) || '-' || char(2193) || char(2274) || char(6158) || char(8203) || '-' || char(8207) || char(8234) || '-' || char(8238) || char(8288) || '-' || char(8292) || char(8294) || '-' || char(8303) || char(65279) || char(65529) || '-' || char(65531) || char(69821) || char(69837) || char(78896) || '-' || char(78911) || char(113824) || '-' || char(113827) || char(119155) || '-' || char(119162) || char(917505) || char(917536) || '-' || char(917631) || ']*') AND length(CAST(\"name\" AS BLOB)) <= 640", + "name": "scheduled_jobs_name_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "(\"next_run_at\" IS NULL OR \"next_run_at\" BETWEEN 0 AND 8640000000000000) AND (\"enabled\" = 0 OR \"next_run_at\" IS NOT NULL)", + "name": "scheduled_jobs_next_run_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"priority\" BETWEEN -100 AND 100", + "name": "scheduled_jobs_priority_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"resource_class\" IN ('exclusive', 'host-heavy', 'interactive', 'light', 'network')", + "name": "scheduled_jobs_resource_class_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "length(CAST(\"resource_keys_json\" AS BLOB)) <= 4096 AND CASE WHEN json_valid(\"resource_keys_json\") THEN json_type(\"resource_keys_json\") = 'array' ELSE 0 END", + "name": "scheduled_jobs_resource_keys_json_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"retry_safe\" IN (0, 1)", + "name": "scheduled_jobs_retry_safe_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "(\"schedule_kind\" = 'interval' AND \"interval_ms\" BETWEEN 60000 AND 31536000000 AND \"time_of_day\" IS NULL AND \"cron_expression\" IS NULL AND \"time_zone\" IS NULL) OR (\"schedule_kind\" = 'daily' AND \"interval_ms\" IS NULL AND \"time_of_day\" IS NOT NULL AND instr(\"time_of_day\", char(0)) = 0 AND \"time_of_day\" GLOB '[0-2][0-9]:[0-5][0-9]' AND CAST(substr(\"time_of_day\", 1, 2) AS INTEGER) BETWEEN 0 AND 23 AND \"cron_expression\" IS NULL AND \"time_zone\" IS NOT NULL) OR (\"schedule_kind\" = 'cron' AND \"interval_ms\" IS NULL AND \"time_of_day\" IS NULL AND \"cron_expression\" IS NOT NULL AND length(\"cron_expression\") BETWEEN 9 AND 200 AND instr(\"cron_expression\", char(0)) = 0 AND \"cron_expression\" = trim(\"cron_expression\") AND \"cron_expression\" NOT LIKE '% %' AND \"cron_expression\" NOT GLOB '*[^-0-9*,/ ]*' AND length(\"cron_expression\") - length(replace(\"cron_expression\", ' ', '')) = 4 AND \"time_zone\" IS NOT NULL)", + "name": "scheduled_jobs_schedule_shape_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"time_zone\" IS NULL OR \"time_zone\" IN ('Africa/Abidjan', 'Africa/Accra', 'Africa/Addis_Ababa', 'Africa/Algiers', 'Africa/Asmara', 'Africa/Bamako', 'Africa/Bangui', 'Africa/Banjul', 'Africa/Bissau', 'Africa/Blantyre', 'Africa/Brazzaville', 'Africa/Bujumbura', 'Africa/Cairo', 'Africa/Casablanca', 'Africa/Ceuta', 'Africa/Conakry', 'Africa/Dakar', 'Africa/Dar_es_Salaam', 'Africa/Djibouti', 'Africa/Douala', 'Africa/El_Aaiun', 'Africa/Freetown', 'Africa/Gaborone', 'Africa/Harare', 'Africa/Johannesburg', 'Africa/Juba', 'Africa/Kampala', 'Africa/Khartoum', 'Africa/Kigali', 'Africa/Kinshasa', 'Africa/Lagos', 'Africa/Libreville', 'Africa/Lome', 'Africa/Luanda', 'Africa/Lubumbashi', 'Africa/Lusaka', 'Africa/Malabo', 'Africa/Maputo', 'Africa/Maseru', 'Africa/Mbabane', 'Africa/Mogadishu', 'Africa/Monrovia', 'Africa/Nairobi', 'Africa/Ndjamena', 'Africa/Niamey', 'Africa/Nouakchott', 'Africa/Ouagadougou', 'Africa/Porto-Novo', 'Africa/Sao_Tome', 'Africa/Tripoli', 'Africa/Tunis', 'Africa/Windhoek', 'America/Adak', 'America/Anchorage', 'America/Anguilla', 'America/Antigua', 'America/Araguaina', 'America/Argentina/Buenos_Aires', 'America/Argentina/Catamarca', 'America/Argentina/Cordoba', 'America/Argentina/Jujuy', 'America/Argentina/La_Rioja', 'America/Argentina/Mendoza', 'America/Argentina/Rio_Gallegos', 'America/Argentina/Salta', 'America/Argentina/San_Juan', 'America/Argentina/San_Luis', 'America/Argentina/Tucuman', 'America/Argentina/Ushuaia', 'America/Aruba', 'America/Asuncion', 'America/Atikokan', 'America/Bahia', 'America/Bahia_Banderas', 'America/Barbados', 'America/Belem', 'America/Belize', 'America/Blanc-Sablon', 'America/Boa_Vista', 'America/Bogota', 'America/Boise', 'America/Cambridge_Bay', 'America/Campo_Grande', 'America/Cancun', 'America/Caracas', 'America/Cayenne', 'America/Cayman', 'America/Chicago', 'America/Chihuahua', 'America/Ciudad_Juarez', 'America/Costa_Rica', 'America/Creston', 'America/Cuiaba', 'America/Curacao', 'America/Danmarkshavn', 'America/Dawson', 'America/Dawson_Creek', 'America/Denver', 'America/Detroit', 'America/Dominica', 'America/Edmonton', 'America/Eirunepe', 'America/El_Salvador', 'America/Fort_Nelson', 'America/Fortaleza', 'America/Glace_Bay', 'America/Goose_Bay', 'America/Grand_Turk', 'America/Grenada', 'America/Guadeloupe', 'America/Guatemala', 'America/Guayaquil', 'America/Guyana', 'America/Halifax', 'America/Havana', 'America/Hermosillo', 'America/Indiana/Indianapolis', 'America/Indiana/Knox', 'America/Indiana/Marengo', 'America/Indiana/Petersburg', 'America/Indiana/Tell_City', 'America/Indiana/Vevay', 'America/Indiana/Vincennes', 'America/Indiana/Winamac', 'America/Inuvik', 'America/Iqaluit', 'America/Jamaica', 'America/Juneau', 'America/Kentucky/Louisville', 'America/Kentucky/Monticello', 'America/Kralendijk', 'America/La_Paz', 'America/Lima', 'America/Los_Angeles', 'America/Lower_Princes', 'America/Maceio', 'America/Managua', 'America/Manaus', 'America/Marigot', 'America/Martinique', 'America/Matamoros', 'America/Mazatlan', 'America/Menominee', 'America/Merida', 'America/Metlakatla', 'America/Mexico_City', 'America/Miquelon', 'America/Moncton', 'America/Monterrey', 'America/Montevideo', 'America/Montserrat', 'America/Nassau', 'America/New_York', 'America/Nome', 'America/Noronha', 'America/North_Dakota/Beulah', 'America/North_Dakota/Center', 'America/North_Dakota/New_Salem', 'America/Nuuk', 'America/Ojinaga', 'America/Panama', 'America/Paramaribo', 'America/Phoenix', 'America/Port-au-Prince', 'America/Port_of_Spain', 'America/Porto_Velho', 'America/Puerto_Rico', 'America/Punta_Arenas', 'America/Rankin_Inlet', 'America/Recife', 'America/Regina', 'America/Resolute', 'America/Rio_Branco', 'America/Santarem', 'America/Santiago', 'America/Santo_Domingo', 'America/Sao_Paulo', 'America/Scoresbysund', 'America/Sitka', 'America/St_Barthelemy', 'America/St_Johns', 'America/St_Kitts', 'America/St_Lucia', 'America/St_Thomas', 'America/St_Vincent', 'America/Swift_Current', 'America/Tegucigalpa', 'America/Thule', 'America/Tijuana', 'America/Toronto', 'America/Tortola', 'America/Vancouver', 'America/Whitehorse', 'America/Winnipeg', 'America/Yakutat', 'Antarctica/Casey', 'Antarctica/Davis', 'Antarctica/DumontDUrville', 'Antarctica/Macquarie', 'Antarctica/Mawson', 'Antarctica/McMurdo', 'Antarctica/Palmer', 'Antarctica/Rothera', 'Antarctica/Syowa', 'Antarctica/Troll', 'Antarctica/Vostok', 'Arctic/Longyearbyen', 'Asia/Aden', 'Asia/Almaty', 'Asia/Amman', 'Asia/Anadyr', 'Asia/Aqtau', 'Asia/Aqtobe', 'Asia/Ashgabat', 'Asia/Atyrau', 'Asia/Baghdad', 'Asia/Bahrain', 'Asia/Baku', 'Asia/Bangkok', 'Asia/Barnaul', 'Asia/Beirut', 'Asia/Bishkek', 'Asia/Brunei', 'Asia/Chita', 'Asia/Choibalsan', 'Asia/Colombo', 'Asia/Damascus', 'Asia/Dhaka', 'Asia/Dili', 'Asia/Dubai', 'Asia/Dushanbe', 'Asia/Famagusta', 'Asia/Gaza', 'Asia/Hebron', 'Asia/Ho_Chi_Minh', 'Asia/Hong_Kong', 'Asia/Hovd', 'Asia/Irkutsk', 'Asia/Jakarta', 'Asia/Jayapura', 'Asia/Jerusalem', 'Asia/Kabul', 'Asia/Kamchatka', 'Asia/Karachi', 'Asia/Kathmandu', 'Asia/Khandyga', 'Asia/Kolkata', 'Asia/Krasnoyarsk', 'Asia/Kuala_Lumpur', 'Asia/Kuching', 'Asia/Kuwait', 'Asia/Macau', 'Asia/Magadan', 'Asia/Makassar', 'Asia/Manila', 'Asia/Muscat', 'Asia/Nicosia', 'Asia/Novokuznetsk', 'Asia/Novosibirsk', 'Asia/Omsk', 'Asia/Oral', 'Asia/Phnom_Penh', 'Asia/Pontianak', 'Asia/Pyongyang', 'Asia/Qatar', 'Asia/Qostanay', 'Asia/Qyzylorda', 'Asia/Riyadh', 'Asia/Sakhalin', 'Asia/Samarkand', 'Asia/Seoul', 'Asia/Shanghai', 'Asia/Singapore', 'Asia/Srednekolymsk', 'Asia/Taipei', 'Asia/Tashkent', 'Asia/Tbilisi', 'Asia/Tehran', 'Asia/Thimphu', 'Asia/Tokyo', 'Asia/Tomsk', 'Asia/Ulaanbaatar', 'Asia/Urumqi', 'Asia/Ust-Nera', 'Asia/Vientiane', 'Asia/Vladivostok', 'Asia/Yakutsk', 'Asia/Yangon', 'Asia/Yekaterinburg', 'Asia/Yerevan', 'Atlantic/Azores', 'Atlantic/Bermuda', 'Atlantic/Canary', 'Atlantic/Cape_Verde', 'Atlantic/Faroe', 'Atlantic/Madeira', 'Atlantic/Reykjavik', 'Atlantic/South_Georgia', 'Atlantic/St_Helena', 'Atlantic/Stanley', 'Australia/Adelaide', 'Australia/Brisbane', 'Australia/Broken_Hill', 'Australia/Darwin', 'Australia/Eucla', 'Australia/Hobart', 'Australia/Lindeman', 'Australia/Lord_Howe', 'Australia/Melbourne', 'Australia/Perth', 'Australia/Sydney', 'Etc/GMT+1', 'Etc/GMT+10', 'Etc/GMT+11', 'Etc/GMT+12', 'Etc/GMT+2', 'Etc/GMT+3', 'Etc/GMT+4', 'Etc/GMT+5', 'Etc/GMT+6', 'Etc/GMT+7', 'Etc/GMT+8', 'Etc/GMT+9', 'Etc/GMT-1', 'Etc/GMT-10', 'Etc/GMT-11', 'Etc/GMT-12', 'Etc/GMT-13', 'Etc/GMT-14', 'Etc/GMT-2', 'Etc/GMT-3', 'Etc/GMT-4', 'Etc/GMT-5', 'Etc/GMT-6', 'Etc/GMT-7', 'Etc/GMT-8', 'Etc/GMT-9', 'Europe/Amsterdam', 'Europe/Andorra', 'Europe/Astrakhan', 'Europe/Athens', 'Europe/Belgrade', 'Europe/Berlin', 'Europe/Bratislava', 'Europe/Brussels', 'Europe/Bucharest', 'Europe/Budapest', 'Europe/Busingen', 'Europe/Chisinau', 'Europe/Copenhagen', 'Europe/Dublin', 'Europe/Gibraltar', 'Europe/Guernsey', 'Europe/Helsinki', 'Europe/Isle_of_Man', 'Europe/Istanbul', 'Europe/Jersey', 'Europe/Kaliningrad', 'Europe/Kirov', 'Europe/Kyiv', 'Europe/Lisbon', 'Europe/Ljubljana', 'Europe/London', 'Europe/Luxembourg', 'Europe/Madrid', 'Europe/Malta', 'Europe/Mariehamn', 'Europe/Minsk', 'Europe/Monaco', 'Europe/Moscow', 'Europe/Oslo', 'Europe/Paris', 'Europe/Podgorica', 'Europe/Prague', 'Europe/Riga', 'Europe/Rome', 'Europe/Samara', 'Europe/San_Marino', 'Europe/Sarajevo', 'Europe/Saratov', 'Europe/Simferopol', 'Europe/Skopje', 'Europe/Sofia', 'Europe/Stockholm', 'Europe/Tallinn', 'Europe/Tirane', 'Europe/Ulyanovsk', 'Europe/Vaduz', 'Europe/Vatican', 'Europe/Vienna', 'Europe/Vilnius', 'Europe/Volgograd', 'Europe/Warsaw', 'Europe/Zagreb', 'Europe/Zurich', 'Indian/Antananarivo', 'Indian/Chagos', 'Indian/Christmas', 'Indian/Cocos', 'Indian/Comoro', 'Indian/Kerguelen', 'Indian/Mahe', 'Indian/Maldives', 'Indian/Mauritius', 'Indian/Mayotte', 'Indian/Reunion', 'Pacific/Apia', 'Pacific/Auckland', 'Pacific/Bougainville', 'Pacific/Chatham', 'Pacific/Chuuk', 'Pacific/Easter', 'Pacific/Efate', 'Pacific/Fakaofo', 'Pacific/Fiji', 'Pacific/Funafuti', 'Pacific/Galapagos', 'Pacific/Gambier', 'Pacific/Guadalcanal', 'Pacific/Guam', 'Pacific/Honolulu', 'Pacific/Kanton', 'Pacific/Kiritimati', 'Pacific/Kosrae', 'Pacific/Kwajalein', 'Pacific/Majuro', 'Pacific/Marquesas', 'Pacific/Midway', 'Pacific/Nauru', 'Pacific/Niue', 'Pacific/Norfolk', 'Pacific/Noumea', 'Pacific/Pago_Pago', 'Pacific/Palau', 'Pacific/Pitcairn', 'Pacific/Pohnpei', 'Pacific/Port_Moresby', 'Pacific/Rarotonga', 'Pacific/Saipan', 'Pacific/Tahiti', 'Pacific/Tarawa', 'Pacific/Tongatapu', 'Pacific/Wake', 'Pacific/Wallis', 'UTC')", + "name": "scheduled_jobs_time_zone_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"timeout_ms\" BETWEEN 1000 AND 86400000", + "name": "scheduled_jobs_timeout_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"updated_at\" BETWEEN 0 AND 8640000000000000 AND \"updated_at\" >= \"created_at\"", + "name": "scheduled_jobs_updated_at_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, + { + "value": "\"version\" BETWEEN 1 AND 9007199254740991", + "name": "scheduled_jobs_version_check", + "entityType": "checks", + "table": "scheduled_jobs" + }, { "value": "\"applied_at\" BETWEEN 0 AND 8640000000000000", "name": "schema_migrations_applied_at_check", @@ -4939,6 +6950,42 @@ "name": "users_username_check", "entityType": "checks", "table": "users" + }, + { + "value": "\"capacity\" BETWEEN 1 AND 16", + "name": "worker_instances_capacity_check", + "entityType": "checks", + "table": "worker_instances" + }, + { + "value": "length(\"id\") = 36 AND instr(\"id\", char(0)) = 0 AND length(replace(\"id\", '-', '')) = 32 AND replace(\"id\", '-', '') NOT GLOB '*[^0-9a-f]*' AND substr(\"id\", 9, 1) = '-' AND substr(\"id\", 14, 1) = '-' AND substr(\"id\", 15, 1) = '7' AND substr(\"id\", 19, 1) = '-' AND substr(\"id\", 20, 1) GLOB '[89ab]' AND substr(\"id\", 24, 1) = '-'", + "name": "worker_instances_id_check", + "entityType": "checks", + "table": "worker_instances" + }, + { + "value": "\"pid\" BETWEEN 1 AND 2147483647", + "name": "worker_instances_pid_check", + "entityType": "checks", + "table": "worker_instances" + }, + { + "value": "length(\"release_id\") = 40 AND instr(\"release_id\", char(0)) = 0 AND \"release_id\" NOT GLOB '*[^0-9a-f]*'", + "name": "worker_instances_release_id_check", + "entityType": "checks", + "table": "worker_instances" + }, + { + "value": "(\"state\" = 'online' AND \"draining_at\" IS NULL AND \"stopped_at\" IS NULL) OR (\"state\" = 'draining' AND \"draining_at\" IS NOT NULL AND \"stopped_at\" IS NULL) OR (\"state\" = 'stopped' AND \"draining_at\" IS NOT NULL AND \"stopped_at\" IS NOT NULL)", + "name": "worker_instances_state_check", + "entityType": "checks", + "table": "worker_instances" + }, + { + "value": "\"started_at\" BETWEEN 0 AND 8640000000000000 AND \"heartbeat_at\" BETWEEN 0 AND 8640000000000000 AND \"heartbeat_at\" >= \"started_at\" AND (\"draining_at\" IS NULL OR (\"draining_at\" BETWEEN 0 AND 8640000000000000 AND \"draining_at\" >= \"started_at\")) AND (\"stopped_at\" IS NULL OR (\"stopped_at\" BETWEEN 0 AND 8640000000000000 AND \"stopped_at\" >= \"draining_at\"))", + "name": "worker_instances_time_check", + "entityType": "checks", + "table": "worker_instances" } ], "renames": [] diff --git a/greenfield/scripts/documentation/artifacts.test.ts b/greenfield/scripts/documentation/artifacts.test.ts index 000a88cc6..6f6459c82 100644 --- a/greenfield/scripts/documentation/artifacts.test.ts +++ b/greenfield/scripts/documentation/artifacts.test.ts @@ -107,6 +107,28 @@ describe("generated contract documentation", () => { type: "string", }, ], + [ + "jobs.runs", + "jobs.listRuns", + { + anyOf: [ + { + format: "uuid", + maxLength: 36, + minLength: 36, + pattern: + "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", + type: "string", + }, + { + maxLength: 80, + minLength: 1, + pattern: "^[a-z0-9][a-z0-9._-]*$", + type: "string", + }, + ], + }, + ], [ "monitoring.incidents", "incidents.list", @@ -122,6 +144,28 @@ describe("generated contract documentation", () => { "reports.list", { maxLength: 200, pattern: "\\S", type: "string" }, ], + [ + "schedules.records", + "schedules.list", + { + anyOf: [ + { + format: "uuid", + maxLength: 36, + minLength: 36, + pattern: + "^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$", + type: "string", + }, + { + maxLength: 80, + minLength: 1, + pattern: "^[a-z0-9][a-z0-9._-]*$", + type: "string", + }, + ], + }, + ], [ "tasks.records", "tasks.list", @@ -149,7 +193,10 @@ describe("generated contract documentation", () => { type: "object", }); } - expect(realtimeDocumentation?.match(/^\| `/gmu)).toHaveLength(5); + expect(realtimeDocumentation?.match(/^\| `/gmu)).toHaveLength(7); + expect(first.get("schemas/schedules.update.input.schema.json")).toContain( + "Five-field minute cron; live validation accepts JAN-DEC month and SUN-SAT weekday aliases, normalizes aliases and ASCII whitespace, and requires a future occurrence." + ); }); test("emits JSON Schema from the same Valibot transport schemas", () => { diff --git a/greenfield/scripts/documentation/jsonSchema.test.ts b/greenfield/scripts/documentation/jsonSchema.test.ts index 3209443b9..c96921797 100644 --- a/greenfield/scripts/documentation/jsonSchema.test.ts +++ b/greenfield/scripts/documentation/jsonSchema.test.ts @@ -23,6 +23,8 @@ import { listAutomationPrincipalsResultSchema, } from "../../src/contracts/automationSecurity.ts"; import { listIncidentsResultSchema } from "../../src/contracts/incidents.ts"; +import { scheduleCronExpressionSchema } from "../../src/contracts/jobModel.ts"; +import { jobRealtimeChangeSchemas } from "../../src/contracts/jobRealtime.ts"; import { completeMonitoringSnapshotInputSchema, monitoringJsonObjectSchema, @@ -101,6 +103,30 @@ describe("contract JSON Schema conversion", () => { ).toThrow('The "transform" action cannot be converted to JSON Schema.'); }); + test("documents accepted cron aliases before canonical normalization", () => { + const schema = convertContractSchema( + scheduleCronExpressionSchema, + "test.scheduleCronExpression", + "input" + ); + + expect(schema.description).toContain("JAN-DEC month"); + expect(schema.description).toContain("SUN-SAT weekday"); + }); + + test("documents runtime-only realtime entity identity equality", () => { + expect( + convertContractSchema( + jobRealtimeChangeSchemas[0], + "test.jobRealtimeChange", + "output" + ) + ).toMatchObject({ + $comment: + "Live Valibot validation additionally requires the realtime entity and compact payload IDs to match exactly.", + }); + }); + test("documents automation normalization and runtime-only cross-field checks", () => { expect( convertContractSchema( @@ -113,6 +139,8 @@ describe("contract JSON Schema conversion", () => { enum: [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", @@ -122,7 +150,7 @@ describe("contract JSON Schema conversion", () => { "tasks:write", ], }, - maxItems: 9, + maxItems: 11, type: "array", uniqueItems: true, }); diff --git a/greenfield/scripts/documentation/jsonSchema.ts b/greenfield/scripts/documentation/jsonSchema.ts index e89d885c6..706887aae 100644 --- a/greenfield/scripts/documentation/jsonSchema.ts +++ b/greenfield/scripts/documentation/jsonSchema.ts @@ -41,6 +41,31 @@ import { incidentPageCursorIsConsistent, newestIncidentOrderIsStable, } from "../../src/contracts/incidents.ts"; +import { + activeJobDisableIntentTimesAreConsistent, + jobPayloadFitsBudget, + jobResourceKeysAreCanonical, + jobRunEventIsConsistent, + jobRunEventMessageFitsBudget, + jobRunEventProgressFitsBudget, + jobRunResultFitsBudget, + jobRunSummaryIsConsistent, + jobWorkerSummaryIsConsistent, + normalizeScheduleCronExpression, + scheduleCronExpressionIsValid, + scheduleSummaryIsConsistent, + scheduleTimeZoneIsCanonical, +} from "../../src/contracts/jobModel.ts"; +import { jobRealtimeIdentityMatches } from "../../src/contracts/jobRealtime.ts"; +import { + activeJobResourceClassesAreCanonical, + jobQueueSummaryIsConsistent, + jobRunDetailIsConsistent, + jobRunPageCursorIsConsistent, + jobWorkerSummariesAreCanonical, + newestJobRunEventOrderIsStable, + newestJobRunOrderIsStable, +} from "../../src/contracts/jobs.ts"; import { activeIncidentSummaryTimesAreConsistent, activeIncidentTimesAreConsistent, @@ -65,6 +90,12 @@ import { reportPageCursorIsConsistent, upsertReportInputFitsBudget, } from "../../src/contracts/reports.ts"; +import { + scheduleOrderIsStable, + schedulePageCursorIsConsistent, + scheduleRunPageCursorIsConsistent, + scheduleUpdatePatchIsConsistent, +} from "../../src/contracts/schedules.ts"; import { isValidSecurityLabel, securityLabelMaximumLength, @@ -151,6 +182,102 @@ const runtimeCheckComments = new Map([ canonicalAgentStatuses, "Live Valibot validation additionally requires one canonically ordered status per configured agent ID.", ], + [ + jobPayloadFitsBudget, + "Live Valibot validation additionally limits the serialized job payload to its reviewed UTF-8 byte budget.", + ], + [ + jobRunResultFitsBudget, + "Live Valibot validation additionally limits the serialized job result to its reviewed UTF-8 byte budget.", + ], + [ + jobRunEventProgressFitsBudget, + "Live Valibot validation additionally limits serialized job progress to its reviewed UTF-8 byte budget.", + ], + [ + jobResourceKeysAreCanonical, + "Live Valibot validation additionally requires resource keys to be unique, strictly sorted, and within their aggregate UTF-8 byte budget.", + ], + [ + scheduleCronExpressionIsValid, + "Live Valibot validation additionally requires a valid five-field minute cron with a future occurrence.", + ], + [ + scheduleTimeZoneIsCanonical, + "Live Valibot validation additionally requires UTC or a canonical IANA time-zone identifier.", + ], + [ + jobRunSummaryIsConsistent, + "Live Valibot validation additionally requires run provenance, attempts, state, cancellation, and timestamps to agree.", + ], + [ + jobRunEventMessageFitsBudget, + "Live Valibot validation additionally limits the job-event message to its reviewed UTF-8 byte budget.", + ], + [ + jobRunEventIsConsistent, + "Live Valibot validation additionally requires job-event payload fields to agree with the event kind.", + ], + [ + jobWorkerSummaryIsConsistent, + "Live Valibot validation additionally requires worker capacity and lifecycle timestamps to agree.", + ], + [ + activeJobDisableIntentTimesAreConsistent, + "Live Valibot validation additionally requires disable-intent expiry after creation.", + ], + [ + scheduleSummaryIsConsistent, + "Live Valibot validation additionally binds schedule state to its cursor, disable intent, and embedded runs.", + ], + [ + newestJobRunOrderIsStable, + "Live Valibot validation additionally requires strict newest-first job-run ordering by queue timestamp and ID.", + ], + [ + activeJobResourceClassesAreCanonical, + "Live Valibot validation additionally requires active resource classes in canonical unique order.", + ], + [ + jobWorkerSummariesAreCanonical, + "Live Valibot validation additionally requires unique workers in canonical ID order.", + ], + [ + jobQueueSummaryIsConsistent, + "Live Valibot validation additionally binds queue-derived fields to their exact state counts.", + ], + [ + jobRunPageCursorIsConsistent, + "Live Valibot validation additionally requires a job-run cursor to identify the returned last row.", + ], + [ + newestJobRunEventOrderIsStable, + "Live Valibot validation additionally requires strict newest-first job-event sequence order.", + ], + [ + jobRunDetailIsConsistent, + "Live Valibot validation additionally binds job result, events, cursor, and run state.", + ], + [ + jobRealtimeIdentityMatches, + "Live Valibot validation additionally requires the realtime entity and compact payload IDs to match exactly.", + ], + [ + scheduleOrderIsStable, + "Live Valibot validation additionally requires strict ascending schedule ID order.", + ], + [ + schedulePageCursorIsConsistent, + "Live Valibot validation additionally requires a schedule cursor to identify the returned last row.", + ], + [ + scheduleUpdatePatchIsConsistent, + "Live Valibot validation additionally requires a non-empty schedule patch with an explicit disable transition.", + ], + [ + scheduleRunPageCursorIsConsistent, + "Live Valibot validation additionally requires a schedule-run cursor to identify the returned last row.", + ], [ monitoringJsonObjectFitsBudget, "Live Valibot validation additionally limits the serialized JSON object to its reviewed UTF-8 byte budget.", @@ -561,7 +688,8 @@ export function convertContractSchema( operation === sortApplicationCapabilities || operation === canonicalAgentDefinitions || operation === canonicalizeTaskStrings || - operation === freezeTaskStrings) + operation === freezeTaskStrings || + operation === normalizeScheduleCronExpression) ) { return jsonSchema; } diff --git a/greenfield/scripts/sourceBoundaries/externalAuthorityPolicy.ts b/greenfield/scripts/sourceBoundaries/externalAuthorityPolicy.ts index 20aedd576..ad22edd97 100644 --- a/greenfield/scripts/sourceBoundaries/externalAuthorityPolicy.ts +++ b/greenfield/scripts/sourceBoundaries/externalAuthorityPolicy.ts @@ -73,7 +73,10 @@ function isForbiddenBrowserPackage(specifier: string): boolean { } function isAllowedNeutralPackage(specifier: string): boolean { - return /^(?:date-fns(?:\/|$)|valibot(?:\/|$))/u.test(specifier); + return ( + /^(?:date-fns(?:\/|$)|valibot(?:\/|$))/u.test(specifier) || + /^(?:effect\/(?:Cron|Result))$/u.test(specifier) + ); } function importBindingSignature(sourceImport: SourceImport): string | undefined { diff --git a/greenfield/scripts/sourceBoundaries/policy.test.ts b/greenfield/scripts/sourceBoundaries/policy.test.ts index 2518a9b32..add50c625 100644 --- a/greenfield/scripts/sourceBoundaries/policy.test.ts +++ b/greenfield/scripts/sourceBoundaries/policy.test.ts @@ -48,7 +48,7 @@ describe("source-boundary policy", () => { expect( validateSourceImport( "src/app/worker.ts", - staticImport("../server/database/runtime/databaseRuntimeOwner.ts") + staticImport("../server/domains/jobs/workerRuntime.ts") ) ).toBeUndefined(); expect( @@ -100,6 +100,12 @@ describe("source-boundary policy", () => { staticImport("../../server/domains/security/authenticationLifecycle.ts") )?.message ).toContain("worker may not import server"); + expect( + validateSourceImport( + "src/app/worker.ts", + staticImport("../server/database/runtime/databaseRuntimeOwner.ts") + )?.message + ).toContain("worker-app may not import server"); expect( validateSourceImport("src/contracts/auth.ts", staticImport("./auth.test.ts")) ?.message @@ -191,6 +197,13 @@ describe("source-boundary policy", () => { expect( validateSourceImport("src/shared/dateTime.ts", staticImport("date-fns")) ).toBeUndefined(); + expect( + validateSourceImport("src/contracts/jobModel.ts", staticImport("effect/Cron")) + ).toBeUndefined(); + expect( + validateSourceImport("src/contracts/jobModel.ts", staticImport("effect")) + ?.message + ).toContain("environment-neutral packages"); expect( validateSourceImport("src/contracts/auth.ts", staticImport("node:fs")) ?.message diff --git a/greenfield/scripts/sourceBoundaries/sourceTopologyPolicy.ts b/greenfield/scripts/sourceBoundaries/sourceTopologyPolicy.ts index d49ddf800..feaf52746 100644 --- a/greenfield/scripts/sourceBoundaries/sourceTopologyPolicy.ts +++ b/greenfield/scripts/sourceBoundaries/sourceTopologyPolicy.ts @@ -39,8 +39,8 @@ const reviewedApplicationServerTargets: ReadonlyMap< [ "src/app/worker.ts", new Set([ + "src/server/domains/jobs/workerRuntime.ts", "src/server/platform/configuration/workerConfiguration.ts", - "src/server/database/runtime/databaseRuntimeOwner.ts", "src/server/platform/filesystem/projectLayout.ts", "src/server/platform/observability/projectFileLogSink.ts", "src/server/platform/observability/structuredLogger.ts", diff --git a/greenfield/src/app/dashboardServer.test.ts b/greenfield/src/app/dashboardServer.test.ts index fcf11a050..afe39c121 100644 --- a/greenfield/src/app/dashboardServer.test.ts +++ b/greenfield/src/app/dashboardServer.test.ts @@ -6,10 +6,13 @@ import path from "node:path"; import * as v from "valibot"; import { listAutomationPrincipalsResultSchema } from "../contracts/automationSecurity.ts"; +import { jobRunSummarySchema } from "../contracts/jobModel.ts"; +import { listJobRunsResultSchema } from "../contracts/jobs.ts"; import { monitoringSubmissionResultSchema, reportDetailSchema, } from "../contracts/monitoring.ts"; +import { listSchedulesResultSchema } from "../contracts/schedules.ts"; import { automationPrincipalCapabilities } from "../server/database/schema/automationPrincipalCapabilities.ts"; import { automationPrincipalCapabilityInsertSchema } from "../server/database/validation/automationPrincipalCapabilities.ts"; import { createWebAuthnRelyingPartyConfiguration } from "../server/domains/security/mfa/webauthn/relyingPartyConfiguration.ts"; @@ -246,4 +249,132 @@ describe("Dashboard security composition", () => { } } }); + + test("wires durable schedule reconciliation and idempotent enqueue through production HTTP", async () => { + const stateDirectory = await mkdtemp( + path.join(os.tmpdir(), "dashboard-server-jobs-composition-") + ); + await chmod(stateDirectory, 0o700); + const applicationRuntime = createDashboardApplicationRuntime({ + database: { + migrationsDirectory, + releaseId: "0".repeat(40), + startupMode: "initialize-empty", + stateDirectory, + }, + logger: createTestStructuredLogger(), + }); + let server: Awaited> | undefined; + + try { + await applicationRuntime.initialize(); + const database = await applicationRuntime.database.orm(); + const fixture = seedAuthenticationTestDatabase( + database, + authenticationTestNow + ); + server = await createDashboardServer({ + applicationRuntime, + browserOrigin: "https://dashboard.example", + gatewayUrl: "ws://127.0.0.1:1", + now: () => authenticationTestNow, + port: 0, + readiness: createReadinessController(), + totpSecretCipher: testTotpSecretCipher, + }); + const headers = { + cookie: `${dashboardSessionCookieName}=${fixture.session.token}`, + }; + const listInput = encodeURIComponent(JSON.stringify({ json: {} })); + const scheduleResponse = await fetch( + new URL(`/trpc/schedules.list?input=${listInput}`, server.url), + { headers } + ); + const scheduleBody = (await scheduleResponse.json()) as { + readonly error?: unknown; + readonly result?: { readonly data?: { readonly json?: unknown } }; + }; + + expect(scheduleResponse.status).toBe(200); + expect(scheduleBody.error).toBeUndefined(); + const schedules = v.parse( + listSchedulesResultSchema, + scheduleBody.result?.data?.json + ); + expect(schedules.schedules).toHaveLength(1); + expect(schedules.schedules[0]).toMatchObject({ + actionKey: "system.worker-smoke", + enabled: false, + id: "system.worker-smoke", + }); + + const idempotencyKey = "cHJvZHVjdGlvbi1odHRwLWNvbXBvc2l0aW9uLWtleS0x"; + const enqueue = () => + fetch(new URL("/trpc/schedules.run", server?.url), { + body: JSON.stringify({ + json: { + id: "system.worker-smoke", + idempotencyKey, + }, + }), + headers: { + ...headers, + "content-type": "application/json", + }, + method: "POST", + }); + const firstEnqueueResponse = await enqueue(); + const firstEnqueueBody = (await firstEnqueueResponse.json()) as { + readonly error?: unknown; + readonly result?: { readonly data?: { readonly json?: unknown } }; + }; + const secondEnqueueResponse = await enqueue(); + const secondEnqueueBody = (await secondEnqueueResponse.json()) as { + readonly error?: unknown; + readonly result?: { readonly data?: { readonly json?: unknown } }; + }; + + expect(firstEnqueueResponse.status).toBe(200); + expect(secondEnqueueResponse.status).toBe(200); + expect(firstEnqueueBody.error).toBeUndefined(); + expect(secondEnqueueBody.error).toBeUndefined(); + const firstRun = v.parse( + jobRunSummarySchema, + firstEnqueueBody.result?.data?.json + ); + const replayedRun = v.parse( + jobRunSummarySchema, + secondEnqueueBody.result?.data?.json + ); + expect(firstRun).toMatchObject({ + actionKey: "system.worker-smoke", + scheduledJobId: "system.worker-smoke", + state: "queued", + triggerType: "manual", + }); + expect(replayedRun.id).toBe(firstRun.id); + + const runResponse = await fetch( + new URL(`/trpc/jobs.listRuns?input=${listInput}`, server.url), + { headers } + ); + const runBody = (await runResponse.json()) as { + readonly error?: unknown; + readonly result?: { readonly data?: { readonly json?: unknown } }; + }; + expect(runResponse.status).toBe(200); + expect(runBody.error).toBeUndefined(); + const runs = v.parse(listJobRunsResultSchema, runBody.result?.data?.json); + expect(runs.runs.map(({ id }) => id)).toEqual([firstRun.id]); + expect(runs.summary.stateCounts.queued).toBe(1); + } finally { + try { + await (server === undefined + ? applicationRuntime.dispose() + : server.stop(true)); + } finally { + await rm(stateDirectory, { force: true, recursive: true }); + } + } + }); }); diff --git a/greenfield/src/app/dashboardServer.ts b/greenfield/src/app/dashboardServer.ts index 99f2775c5..bd36a238f 100644 --- a/greenfield/src/app/dashboardServer.ts +++ b/greenfield/src/app/dashboardServer.ts @@ -5,6 +5,11 @@ import { Redacted } from "effect"; import { createAgentRepository } from "../server/domains/agents/repository.ts"; import { createAgentService } from "../server/domains/agents/service.ts"; +import { createJobRepository } from "../server/domains/jobs/repository.ts"; +import { + createJobService, + reconcileJobSchedules, +} from "../server/domains/jobs/service.ts"; import { createMonitoringCatalogService } from "../server/domains/monitoring/catalogService.ts"; import { createMonitoringRepository } from "../server/domains/monitoring/repository.ts"; import { createMonitoringService } from "../server/domains/monitoring/service.ts"; @@ -86,6 +91,7 @@ export interface DashboardServerOptions extends Omit< | "hostname" | "mfaAccountLifecycle" | "mfaLoginLifecycle" + | "jobService" | "monitoringCatalogService" | "monitoringService" | "securityAuditLifecycle" @@ -274,6 +280,17 @@ export async function createDashboardServer( repository: createTaskRepository(database, databaseRuntime), wakeEventPump, }); + const jobRepository = createJobRepository(database, databaseRuntime); + await reconcileJobSchedules({ + ...(domainNow === undefined ? {} : { nowMs: () => domainNow().getTime() }), + repository: jobRepository, + wakeEventPump, + }); + const jobService = createJobService({ + ...(domainNow === undefined ? {} : { nowMs: () => domainNow().getTime() }), + repository: jobRepository, + wakeEventPump, + }); const monitoringRepository = createMonitoringRepository( database, databaseRuntime @@ -301,6 +318,7 @@ export async function createDashboardServer( hostname: "127.0.0.1", mfaAccountLifecycle, mfaLoginLifecycle, + jobService, monitoringCatalogService, monitoringService, port: options.port, diff --git a/greenfield/src/app/server.ts b/greenfield/src/app/server.ts index 54473f5fd..8b9f5ddb5 100644 --- a/greenfield/src/app/server.ts +++ b/greenfield/src/app/server.ts @@ -3,6 +3,7 @@ import * as v from "valibot"; import { healthLivenessPath, healthReadinessPath } from "../contracts/system.ts"; import type { AgentService } from "../server/domains/agents/service.ts"; +import type { JobService } from "../server/domains/jobs/service.ts"; import type { MonitoringCatalogService } from "../server/domains/monitoring/catalogService.ts"; import type { MonitoringService } from "../server/domains/monitoring/service.ts"; import type { AuthenticationLifecycleService } from "../server/domains/security/authenticationLifecycle.ts"; @@ -140,6 +141,7 @@ export interface ServerOptions { readonly hostname?: string; readonly mfaAccountLifecycle: MfaAccountLifecycleService; readonly mfaLoginLifecycle: MfaLoginLifecycleService; + readonly jobService: JobService["Service"]; readonly monitoringCatalogService: MonitoringCatalogService["Service"]; readonly monitoringService: MonitoringService["Service"]; readonly port: number; @@ -183,6 +185,7 @@ export async function createServer(options: ServerOptions): Promise = []; + const forceController = new AbortController(); + let resolveDisposalStarted: (() => void) | undefined; + const disposalStarted = new Promise((resolve) => { + resolveDisposalStarted = resolve; + }); + let resolveCompletion: (() => void) | undefined; + let rejectCompletion: ((error: unknown) => void) | undefined; + const completion = new Promise((resolve, reject) => { + resolveCompletion = resolve; + rejectCompletion = reject; + }); const destination = Object.freeze({ fallbackWrite() { events.push("log-fallback"); @@ -64,17 +81,46 @@ function processFixture(initializationFailure?: Error) { dispose() { events.push("signals-dispose"); }, - forceSignal: new AbortController().signal, - termination: Promise.resolve("SIGTERM" as const), + forceSignal: forceController.signal, + termination: + runtimeFailure !== undefined || runtimeStopsUnexpectedly + ? new Promise<"SIGTERM">(() => {}) + : Promise.resolve("SIGTERM" as const), }); const runtime: DashboardWorkerRuntime = Object.freeze({ - dispose() { + completion, + dispose(forceSignal?: AbortSignal) { events.push("runtime-dispose"); + forceSignals.push(forceSignal); + resolveDisposalStarted?.(); + if (waitForForceDuringDisposal) { + if (forceSignal === undefined) { + return Promise.reject( + new Error("Runtime cleanup did not receive the force signal") + ); + } + if (!forceSignal.aborted) { + return new Promise((resolve) => { + forceSignal.addEventListener("abort", () => resolve(), { + once: true, + }); + }); + } + } + resolveCompletion?.(); return Promise.resolve(); }, initialize() { events.push("runtime-initialize"); - if (initializationFailure) return Promise.reject(initializationFailure); + if (initializationFailure) { + rejectCompletion?.(initializationFailure); + return Promise.reject(initializationFailure); + } + if (runtimeFailure) { + queueMicrotask(() => rejectCompletion?.(runtimeFailure)); + } else if (runtimeStopsUnexpectedly) { + queueMicrotask(() => resolveCompletion?.()); + } return Promise.resolve(); }, }); @@ -103,7 +149,14 @@ function processFixture(initializationFailure?: Error) { return Promise.resolve(layout); }, } satisfies DashboardWorkerProcessDependencies); - return { dependencies, events, logLines }; + return { + dependencies, + disposalStarted, + events, + forceController, + forceSignals, + logLines, + }; } const processOptions = Object.freeze({ @@ -135,6 +188,9 @@ describe("Dashboard worker process", () => { expect( fixture.logLines.map((line) => (JSON.parse(line) as { event: string }).event) ).toEqual(["runtime.started", "runtime.stopped"]); + expect(fixture.forceSignals).toEqual([ + expect.objectContaining({ aborted: false }), + ]); }); test("disposes partial ownership and reports a redacted startup failure", () => { @@ -154,4 +210,63 @@ describe("Dashboard worker process", () => { expect(fatal.event).toBe("runtime.start_failed"); expect(JSON.stringify(fatal)).not.toContain("private worker failure"); }); + + test("disposes and fails when the durable coordinator exits unexpectedly", async () => { + const failure = new Error("private coordinator failure"); + const fixture = processFixture(undefined, failure); + + expect( + await runDashboardWorkerProcess(processOptions, fixture.dependencies).catch( + (error: unknown) => error + ) + ).toBe(failure); + + expect(fixture.events).toContain("runtime-dispose"); + const fatal = JSON.parse(fixture.logLines.at(-1) ?? "null") as { + event: string; + failure?: unknown; + }; + expect(fatal.event).toBe("runtime.start_failed"); + expect(JSON.stringify(fatal)).not.toContain("private coordinator failure"); + }); + + test("allows a second signal to force runtime-failure cleanup", async () => { + const failure = new Error("private coordinator failure"); + const fixture = processFixture(undefined, failure, false, true); + const execution = runDashboardWorkerProcess(processOptions, fixture.dependencies); + + await fixture.disposalStarted; + expect( + await Promise.race([ + execution.then( + () => "settled" as const, + () => "settled" as const + ), + Bun.sleep(10).then(() => "waiting" as const), + ]) + ).toBe("waiting"); + + fixture.forceController.abort( + new DOMException("Forced process shutdown requested", "AbortError") + ); + + expect(await execution.catch((error: unknown) => error)).toBe(failure); + expect(fixture.forceSignals).toEqual([fixture.forceController.signal]); + }); + + test("fails closed when the durable runtime resolves before a signal", async () => { + const fixture = processFixture(undefined, undefined, true); + + expect( + await runDashboardWorkerProcess(processOptions, fixture.dependencies).catch( + (error: unknown) => error + ) + ).toEqual(new Error("Dashboard worker runtime stopped unexpectedly")); + + expect(fixture.events).toContain("runtime-dispose"); + const fatal = JSON.parse(fixture.logLines.at(-1) ?? "null") as { + event: string; + }; + expect(fatal.event).toBe("runtime.start_failed"); + }); }); diff --git a/greenfield/src/app/worker.ts b/greenfield/src/app/worker.ts index 7fe4720bd..00b05ff08 100644 --- a/greenfield/src/app/worker.ts +++ b/greenfield/src/app/worker.ts @@ -1,7 +1,10 @@ import { realpath } from "node:fs/promises"; import path from "node:path"; -import { createDatabaseRuntimeOwner } from "../server/database/runtime/databaseRuntimeOwner.ts"; +import { + createDashboardWorkerRuntime, + createSystemJobWorkerSideEffects, +} from "../server/domains/jobs/workerRuntime.ts"; import { parseWorkerConfiguration, type WorkerConfiguration, @@ -62,11 +65,17 @@ const defaultDependencies = Object.freeze({ createLogDestination: (logsDirectory, processRole) => createProjectFileLogDestination(logsDirectory, processRole), createRuntime: (_configuration, layout, release) => - createDatabaseRuntimeOwner({ - migrationsDirectory: path.join(release.releaseRoot, "migrations"), + createDashboardWorkerRuntime({ + database: { + migrationsDirectory: path.join(release.releaseRoot, "migrations"), + releaseId: release.manifest.source.commitSha, + startupMode: "validate-only", + stateDirectory: layout.production.state.root, + }, + pid: process.pid, releaseId: release.manifest.source.commitSha, - startupMode: "validate-only", - stateDirectory: layout.production.state.root, + sideEffects: createSystemJobWorkerSideEffects(), + workerInstanceId: Bun.randomUUIDv7(), }), createTerminationController: createProcessTerminationController, loadRelease: (releasesDirectory, releaseRoot, processRole) => @@ -101,8 +110,7 @@ function normalizeWorkerProcessFailure(error: unknown): Error { } /** - * Runs the database-validating worker lifecycle until a process signal requests shutdown. - * Job capabilities are intentionally absent until their Phase 3 ports are composed. + * Runs durable schedule and job execution until a signal or coordinator defect wins. * @param options Typed environment source and exact immutable release root. * @param dependencies Injectable host/runtime boundaries. */ @@ -127,14 +135,28 @@ export async function runDashboardWorkerProcess( let failure: Error | undefined; try { runtime = dependencies.createRuntime(configuration, layout, release, logger); + const runtimeCompletion = runtime.completion.then( + () => ({ kind: "stopped" as const }), + (error: unknown) => ({ error, kind: "failed" as const }) + ); await runtime.initialize(); logger.info({ component: "runtime", event: "runtime.started", outcome: "success", }); - await termination.termination; - await runtime.dispose(); + const exit = await Promise.race([ + termination.termination.then(() => ({ kind: "signal" as const })), + runtimeCompletion, + ]); + if (exit.kind === "failed") { + throw exit.error; + } + if (exit.kind === "stopped") { + throw new Error("Dashboard worker runtime stopped unexpectedly"); + } + await runtime.dispose(termination.forceSignal); + await runtime.completion; logger.info({ component: "runtime", event: "runtime.stopped", @@ -144,7 +166,7 @@ export async function runDashboardWorkerProcess( failure = normalizeWorkerProcessFailure(error); if (runtime) { try { - await runtime.dispose(); + await runtime.dispose(termination.forceSignal); } catch { // Preserve the initiating process failure. } diff --git a/greenfield/src/browser/api/trpcClient.test.ts b/greenfield/src/browser/api/trpcClient.test.ts index 48bc62569..86230d537 100644 --- a/greenfield/src/browser/api/trpcClient.test.ts +++ b/greenfield/src/browser/api/trpcClient.test.ts @@ -99,6 +99,76 @@ describe("Dashboard browser tRPC client", () => { ]); }); + test("loads durable job and schedule contracts on demand", async () => { + const jobCalls: TransportCall[] = []; + const scheduleCalls: TransportCall[] = []; + const jobClient = createDashboardTrpcClient( + createRecordingTransport( + { + runs: [], + summary: { + activeResourceClasses: [], + control: { + claimingPaused: false, + updatedAtMs: 0, + version: 1, + }, + stateCounts: { + cancelled: 0, + failed: 0, + queued: 0, + running: 0, + succeeded: 0, + "timed-out": 0, + }, + workers: [], + }, + }, + jobCalls + ) + ); + const scheduleClient = createDashboardTrpcClient( + createRecordingTransport({ schedules: [] }, scheduleCalls) + ); + + expect(await jobClient.query("jobs.listRuns", { limit: 50 })).toEqual({ + runs: [], + summary: { + activeResourceClasses: [], + control: { + claimingPaused: false, + updatedAtMs: 0, + version: 1, + }, + stateCounts: { + cancelled: 0, + failed: 0, + queued: 0, + running: 0, + succeeded: 0, + "timed-out": 0, + }, + workers: [], + }, + }); + expect( + await scheduleClient.query("schedules.list", { + enabled: "all", + limit: 50, + }) + ).toEqual({ schedules: [] }); + expect(jobCalls).toEqual([ + { input: { limit: 50 }, kind: "query", path: "jobs.listRuns" }, + ]); + expect(scheduleCalls).toEqual([ + { + input: { enabled: "all", limit: 50 }, + kind: "query", + path: "schedules.list", + }, + ]); + }); + test("rejects invalid input before transport access", async () => { const calls: TransportCall[] = []; const client = createDashboardTrpcClient( diff --git a/greenfield/src/browser/api/trpcClient.ts b/greenfield/src/browser/api/trpcClient.ts index 384024f1e..0e4c02a8b 100644 --- a/greenfield/src/browser/api/trpcClient.ts +++ b/greenfield/src/browser/api/trpcClient.ts @@ -76,6 +76,10 @@ async function procedureContractsFor( const module = await import("../../contracts/incidents.ts"); return module.incidentProcedureContracts; } + case "jobs": { + const module = await import("../../contracts/jobs.ts"); + return module.jobProcedureContracts; + } case "notifications": { const module = await import("../../contracts/notifications.ts"); return module.notificationProcedureContracts; @@ -92,6 +96,10 @@ async function procedureContractsFor( const module = await import("../../contracts/reports.ts"); return module.reportProcedureContracts; } + case "schedules": { + const module = await import("../../contracts/schedules.ts"); + return module.scheduleProcedureContracts; + } case "automationSecurity": { const module = await import("../../contracts/automationSecurity.ts"); return module.automationSecurityProcedureContracts; diff --git a/greenfield/src/contracts/contractRegistry.ts b/greenfield/src/contracts/contractRegistry.ts index 95ca05e2e..901c37a74 100644 --- a/greenfield/src/contracts/contractRegistry.ts +++ b/greenfield/src/contracts/contractRegistry.ts @@ -5,6 +5,8 @@ import { authProcedureContracts } from "./auth.ts"; import { automationSecurityProcedureContracts } from "./automationSecurity.ts"; import { eventsStreamContract } from "./events.ts"; import { incidentProcedureContracts } from "./incidents.ts"; +import { jobRealtimeEventContracts } from "./jobRealtime.ts"; +import { jobProcedureContracts } from "./jobs.ts"; import { monitoringProcedureContracts } from "./monitoringIngestion.ts"; import { monitoringRealtimeEventContracts } from "./monitoringRealtime.ts"; import { notificationProcedureContracts } from "./notifications.ts"; @@ -15,6 +17,7 @@ import { type RealtimeEventContract, } from "./registry.ts"; import { reportProcedureContracts } from "./reports.ts"; +import { scheduleProcedureContracts } from "./schedules.ts"; import { securityAuditProcedureContracts } from "./securityAudit.ts"; import { systemProcedureContracts, systemRawHttpContracts } from "./system.ts"; import { taskRealtimeEventContract } from "./taskRealtime.ts"; @@ -28,9 +31,11 @@ const registeredProcedureContracts = [ ...automationSecurityProcedureContracts, eventsStreamContract, ...incidentProcedureContracts, + ...jobProcedureContracts, ...monitoringProcedureContracts, ...notificationProcedureContracts, ...reportProcedureContracts, + ...scheduleProcedureContracts, ...securityAuditProcedureContracts, ...systemProcedureContracts, ...taskProcedureContracts, @@ -74,6 +79,7 @@ export const rawHttpContracts: readonly RawHttpContract[] = [...systemRawHttpCon /** Implemented realtime topics used by runtime wiring and docs. */ export const realtimeEventContracts: readonly RealtimeEventContract[] = Object.freeze([ agentRealtimeEventContract, + ...jobRealtimeEventContracts, ...monitoringRealtimeEventContracts, taskRealtimeEventContract, ]); diff --git a/greenfield/src/contracts/events.test.ts b/greenfield/src/contracts/events.test.ts index 59cc4d094..d0640e7d1 100644 --- a/greenfield/src/contracts/events.test.ts +++ b/greenfield/src/contracts/events.test.ts @@ -5,6 +5,7 @@ import * as v from "valibot"; import { eventsStreamContract, realtimeStreamCapabilities, + realtimeStreamDataSchema, realtimeStreamInputSchema, realtimeStreamOutputSchema, realtimeTopicDefinitions, @@ -18,6 +19,7 @@ describe("realtime transport contracts", () => { test("documents only capabilities required by registered topics", () => { expect(realtimeStreamCapabilities).toEqual([ "agents:read", + "jobs:read", "notifications:read", "reports:read", "tasks:read", @@ -89,6 +91,42 @@ describe("realtime transport contracts", () => { ).toMatchObject({ id: "1" }); }); + test("rejects mismatched durable job entity and payload identities", () => { + const runId = "018f6f50-6a9e-7b88-8000-000000000001"; + const mismatchedChanges = [ + { + entityId: runId, + entityType: "job-run", + occurredAtMs: 1000, + operation: "updated", + payload: { id: "system.worker-smoke" }, + topic: "jobs.runs", + }, + { + entityId: "queue", + entityType: "job-queue", + occurredAtMs: 1000, + operation: "snapshot-required", + payload: { id: runId }, + topic: "jobs.runs", + }, + { + entityId: "system.worker-smoke", + entityType: "schedule", + occurredAtMs: 1000, + operation: "updated", + payload: { id: "queue" }, + topic: "schedules.records", + }, + ]; + + for (const event of mismatchedChanges) { + expect( + v.safeParse(realtimeStreamDataSchema, { event, kind: "change" }).success + ).toBeFalse(); + } + }); + test("shares exact producer routing policies with the client contract", () => { expect( v.parse(monitoringRealtimeRoutingSchema, { diff --git a/greenfield/src/contracts/events.ts b/greenfield/src/contracts/events.ts index b590e86aa..fb4a0dc93 100644 --- a/greenfield/src/contracts/events.ts +++ b/greenfield/src/contracts/events.ts @@ -9,6 +9,11 @@ import { agentRealtimeTopic, agentRealtimeTopicDefinition, } from "./agentRealtime.ts"; +import { + jobRealtimeChangeSchemas, + jobRealtimeTopicDefinitions, + jobRealtimeTopics, +} from "./jobRealtime.ts"; import { monitoringRealtimeChangeSchemas, monitoringRealtimeTopicDefinitions, @@ -26,6 +31,7 @@ import { /** All topic definitions currently accepted by the realtime transport. */ export const realtimeTopicDefinitions = Object.freeze([ agentRealtimeTopicDefinition, + ...jobRealtimeTopicDefinitions, ...monitoringRealtimeTopicDefinitions, taskRealtimeTopicDefinition, ] as const); @@ -42,6 +48,7 @@ export function findRealtimeTopicDefinition(topic: string) { /** Exact unique capability vocabulary used by registered realtime topics. */ export const realtimeStreamCapabilities = Object.freeze([ "agents:read", + "jobs:read", "notifications:read", "reports:read", "tasks:read", @@ -50,6 +57,8 @@ export const realtimeStreamCapabilities = Object.freeze([ /** Exact registered topic vocabulary accepted by the tracked SSE contract. */ export const realtimeStreamTopics = Object.freeze([ agentRealtimeTopic, + jobRealtimeTopics.runs, + jobRealtimeTopics.schedules, monitoringRealtimeTopics.incidents, monitoringRealtimeTopics.notifications, monitoringRealtimeTopics.reports, @@ -86,8 +95,9 @@ export const realtimeStreamInputSchema = v.strictObject({ /** Data inside one tRPC tracked SSE envelope. */ export const realtimeStreamDataSchema = v.variant("kind", [ v.strictObject({ - event: v.variant("topic", [ + event: v.union([ agentRealtimeChangeSchema, + ...jobRealtimeChangeSchemas, ...monitoringRealtimeChangeSchemas, taskRealtimeChangeSchema, ]), diff --git a/greenfield/src/contracts/jobModel.test.ts b/greenfield/src/contracts/jobModel.test.ts new file mode 100644 index 000000000..3280f067b --- /dev/null +++ b/greenfield/src/contracts/jobModel.test.ts @@ -0,0 +1,330 @@ +import { describe, expect, test } from "bun:test"; + +import * as v from "valibot"; + +import { + jobResourceKeysSchema, + jobRunEventMaximum, + jobRunEventProgressMaximumBytes, + jobRunEventProgressSchema, + jobRunEventSchema, + jobRunPayloadEventMaximum, + jobRunSummarySchema, + jobWorkerSummarySchema, + normalizeScheduleCronExpression, + scheduleConfigurationSchema, + scheduleCronExpressionSchema, + scheduleSummarySchema, + scheduleTimeZoneSchema, +} from "./jobModel.ts"; +import { canonicalScheduleTimeZones } from "./scheduleTimeZones.ts"; + +const runId = "018f6f50-6a9e-7b88-8000-000000000001"; +const workerId = "018f6f50-6a9e-7b88-8000-000000000002"; +const scheduleId = "system.worker-smoke"; + +function queuedRun() { + return { + actionKey: "system.worker-smoke", + attemptCount: 0, + attemptLimit: 3, + availableAtMs: 1000, + cancellationPolicy: "cooperative" as const, + displayName: "Worker smoke", + eventCount: 1, + id: runId, + priority: 0, + queuedAtMs: 1000, + resourceClass: "light" as const, + resourceKeys: ["database"], + retrySafe: true, + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + state: "queued" as const, + stateVersion: 1, + timeoutMs: 30_000, + triggerType: "manual" as const, + updatedAtMs: 1000, + }; +} + +describe("durable job models", () => { + test("accepts lifecycle-consistent run projections without internal execution data", () => { + expect(v.parse(jobRunSummarySchema, queuedRun()).state).toBe("queued"); + + const succeeded = { + ...queuedRun(), + attemptCount: 1, + eventCount: 3, + finishedAtMs: 3000, + firstStartedAtMs: 2000, + lastAttemptStartedAtMs: 2000, + state: "succeeded", + stateVersion: 3, + updatedAtMs: 3000, + }; + expect(v.parse(jobRunSummarySchema, succeeded).state).toBe("succeeded"); + + expect( + v.parse(jobRunSummarySchema, { + ...queuedRun(), + attemptCount: 1, + availableAtMs: 60_000, + eventCount: 4, + firstStartedAtMs: 2000, + lastAttemptStartedAtMs: 2000, + stateVersion: 3, + updatedAtMs: 3000, + }).availableAtMs + ).toBe(60_000); + + for (const extra of [ + { leaseToken: "018f6f50-6a9e-7b88-8000-000000000099" }, + { payload: { secret: true } }, + { workerInstanceId: workerId }, + ]) { + expect( + v.safeParse(jobRunSummarySchema, { ...queuedRun(), ...extra }).success + ).toBeFalse(); + } + }); + + test("rejects inconsistent schedule provenance, attempts, terminal state, and time", () => { + const invalidRuns = [ + { ...queuedRun(), scheduledJobVersion: undefined }, + { + ...queuedRun(), + scheduledJobId: undefined, + scheduledJobVersion: undefined, + }, + { ...queuedRun(), triggerType: "startup" }, + { ...queuedRun(), scheduledForAtMs: 500, triggerType: "manual" }, + { + ...queuedRun(), + scheduledForAtMs: 500, + scheduledJobId: undefined, + scheduledJobVersion: undefined, + triggerType: "schedule", + }, + { ...queuedRun(), attemptCount: 1 }, + { ...queuedRun(), attemptCount: 4 }, + { ...queuedRun(), state: "failed" }, + { + ...queuedRun(), + finishedAtMs: 900, + state: "cancelled", + terminalCode: "cancelled", + terminalMessage: "Cancelled", + }, + { ...queuedRun(), cancelRequestedAtMs: 1000, cancellationPolicy: "never" }, + ]; + + for (const run of invalidRuns) { + expect(v.safeParse(jobRunSummarySchema, run).success).toBeFalse(); + } + }); + + test("bounds and canonicalizes resources and durable event payloads", () => { + expect(v.parse(jobResourceKeysSchema, ["database", "worker"])).toEqual([ + "database", + "worker", + ]); + for (const keys of [ + ["worker", "database"], + ["database", "database"], + ["Database"], + ]) { + expect(v.safeParse(jobResourceKeysSchema, keys).success).toBeFalse(); + } + + expect(jobRunEventMaximum - jobRunPayloadEventMaximum).toBe(33); + expect( + v.safeParse(jobRunEventProgressSchema, { + value: "x".repeat(jobRunEventProgressMaximumBytes), + }).success + ).toBeFalse(); + expect( + v.parse(jobRunEventSchema, { + attempt: 1, + kind: "progress", + occurredAtMs: 2000, + progress: { percent: 50 }, + sequence: 2, + workerInstanceId: workerId, + }).sequence + ).toBe(2); + for (const event of [ + { + attempt: 1, + kind: "stdout", + occurredAtMs: 2000, + sequence: 2, + }, + { + attempt: 1, + kind: "claimed", + occurredAtMs: 2000, + progress: { unexpected: true }, + sequence: 2, + }, + ]) { + expect(v.safeParse(jobRunEventSchema, event).success).toBeFalse(); + } + }); + + test("normalizes five-field cron aliases and ASCII whitespace before validation", () => { + expect(normalizeScheduleCronExpression(" 0\t9 * JAN MON-FRI ")).toBe( + "0 9 * 1 1-5" + ); + expect(v.parse(scheduleCronExpressionSchema, "0\t9 * JAN MON-FRI")).toBe( + "0 9 * 1 1-5" + ); + expect( + v.parse(scheduleConfigurationSchema, { + expression: "*/5 * * * *", + kind: "cron", + timeZone: "Europe/Oslo", + }) + ).toEqual({ + expression: "*/5 * * * *", + kind: "cron", + timeZone: "Europe/Oslo", + }); + + for (const expression of [ + "0 */5 * * * *", + "* * * *", + "61 * * * *", + "0 0 30 2 *", + "*\u00A0* * * *", + "0 9 * MON *", + "0 9 * * JAN", + ]) { + expect( + v.safeParse(scheduleCronExpressionSchema, expression).success + ).toBeFalse(); + } + }); + + test("accepts explicit UTC and canonical IANA zones but rejects aliases and offsets", () => { + expect(Object.isFrozen(canonicalScheduleTimeZones)).toBeTrue(); + expect(canonicalScheduleTimeZones).toEqual( + [...canonicalScheduleTimeZones].toSorted() + ); + expect(new Set(canonicalScheduleTimeZones).size).toBe( + canonicalScheduleTimeZones.length + ); + for (const timeZone of ["UTC", "Europe/Oslo", "America/New_York"]) { + expect(canonicalScheduleTimeZones).toContain(timeZone); + expect(v.parse(scheduleTimeZoneSchema, timeZone)).toBe(timeZone); + } + for (const timeZone of ["US/Eastern", "GMT", "+01:00", "local"]) { + expect(canonicalScheduleTimeZones).not.toContain(timeZone); + expect(v.safeParse(scheduleTimeZoneSchema, timeZone).success).toBeFalse(); + } + }); + + test("validates worker and schedule projections across state boundaries", () => { + expect( + v.parse(jobWorkerSummarySchema, { + activeRunCount: 1, + capacity: 2, + heartbeatAtMs: 2000, + id: workerId, + releaseId: "a".repeat(40), + startedAtMs: 1000, + state: "online", + }).state + ).toBe("online"); + expect( + v.parse(jobWorkerSummarySchema, { + activeRunCount: 1, + capacity: 2, + drainingAtMs: 2500, + heartbeatAtMs: 2600, + id: workerId, + releaseId: "a".repeat(40), + startedAtMs: 1000, + state: "draining", + }).state + ).toBe("draining"); + expect( + v.parse(jobWorkerSummarySchema, { + activeRunCount: 0, + capacity: 2, + drainingAtMs: 2500, + heartbeatAtMs: 2600, + id: workerId, + releaseId: "a".repeat(40), + startedAtMs: 1000, + state: "stopped", + stoppedAtMs: 3000, + }).state + ).toBe("stopped"); + expect( + v.safeParse(jobWorkerSummarySchema, { + activeRunCount: 3, + capacity: 2, + heartbeatAtMs: 2000, + id: workerId, + releaseId: "a".repeat(40), + startedAtMs: 1000, + state: "online", + }).success + ).toBeFalse(); + expect( + v.safeParse(jobWorkerSummarySchema, { + activeRunCount: 1, + capacity: 2, + drainingAtMs: 2500, + heartbeatAtMs: 2000, + id: workerId, + releaseId: "a".repeat(40), + startedAtMs: 1000, + state: "stopped", + stoppedAtMs: 3000, + }).success + ).toBeFalse(); + expect( + v.safeParse(jobWorkerSummarySchema, { + activeRunCount: 0, + capacity: 2, + heartbeatAtMs: 2000, + id: workerId, + releaseId: "a".repeat(40), + startedAtMs: 1000, + state: "stopped", + stoppedAtMs: 3000, + }).success + ).toBeFalse(); + + const schedule = { + actionKey: "system.worker-smoke", + activeRun: queuedRun(), + attemptLimit: 3, + cancellationPolicy: "cooperative" as const, + createdAtMs: 500, + description: "Checks the worker without host mutation.", + enabled: true, + id: scheduleId, + latestRun: queuedRun(), + name: "Worker smoke", + nextRunAtMs: 60_000, + priority: 0, + resourceClass: "light" as const, + resourceKeys: ["database"], + retrySafe: true, + schedule: { intervalMs: 60_000, kind: "interval" as const }, + timeoutMs: 30_000, + updatedAtMs: 1000, + version: 1, + }; + expect(v.parse(scheduleSummarySchema, schedule).id).toBe(scheduleId); + expect( + v.safeParse(scheduleSummarySchema, { + ...schedule, + activeRun: { ...queuedRun(), scheduledJobId: "other" }, + }).success + ).toBeFalse(); + }); +}); diff --git a/greenfield/src/contracts/jobModel.ts b/greenfield/src/contracts/jobModel.ts new file mode 100644 index 000000000..41b20ffa6 --- /dev/null +++ b/greenfield/src/contracts/jobModel.ts @@ -0,0 +1,765 @@ +import * as Cron from "effect/Cron"; +import * as Result from "effect/Result"; +import * as v from "valibot"; + +import { timestampMillisecondsSchema } from "../shared/dateTime.ts"; +import { utf8ByteLength } from "../shared/encoding.ts"; +import { jsonObjectSchema, type JsonObject } from "../shared/json.ts"; +import { + boundedControlSafeTextSchema, + compareStrings, + fullCommitShaSchema, + hasUniqueArrayItems, + lowercaseUuidV7Schema, + nonnegativeSafeIntegerSchema, + positiveSafeIntegerSchema, +} from "../shared/validation.ts"; +import { canonicalScheduleTimeZones } from "./scheduleTimeZones.ts"; +import { isCanonicalWebAuthnBase64Url } from "./webauthn.ts"; + +/** Canonical durable job-run states. */ +export const jobRunStates = [ + "cancelled", + "failed", + "queued", + "running", + "succeeded", + "timed-out", +] as const; + +/** Provenance recorded for every durable run. */ +export const jobTriggerTypes = ["manual", "schedule", "startup", "system"] as const; + +/** Reviewed worker resource classes, ordered canonically for transport output. */ +export const jobResourceClasses = [ + "exclusive", + "host-heavy", + "interactive", + "light", + "network", +] as const; + +/** Cancellation behavior captured in each immutable execution snapshot. */ +export const jobCancellationPolicies = ["cooperative", "never", "queued-only"] as const; + +/** Durable bounded event vocabulary for one job run. */ +export const jobRunEventKinds = [ + "cancel-requested", + "cancelled", + "claimed", + "failed", + "lease-expired", + "output-truncated", + "progress", + "queued", + "retry-scheduled", + "stderr", + "stdout", + "succeeded", + "timed-out", +] as const; + +/** Worker lifecycle states visible to queue readers. */ +export const jobWorkerStates = ["draining", "online", "stopped"] as const; + +/** Dashboard-local schedule variants implemented in Phase 3. */ +export const scheduleKinds = ["cron", "daily", "interval"] as const; + +export const jobActionKeyMaximumLength = 128; +export const jobDisplayNameMaximumLength = 160; +export const jobDescriptionMaximumLength = 1000; +export const jobPayloadMaximumBytes = 64 * 1024; +export const jobResourceKeyMaximumLength = 128; +export const jobResourceKeyMaximum = 32; +export const jobResourceKeysMaximumBytes = 4 * 1024; +export const jobRunResultMaximumBytes = 64 * 1024; +export const jobRunTerminalCodeMaximumLength = 128; +export const jobRunTerminalMessageMaximumLength = 2000; +export const jobRunAttemptMaximum = 10; +export const jobRunEventMaximum = 1000; +/** Payload slots left after reserving every worst-case structural lifecycle event. */ +export const jobRunPayloadEventMaximum = 967; +export const jobRunOutputMaximumBytes = 1024 * 1024; +export const jobRunEventMessageMaximumLength = 4096; +export const jobRunEventMessageMaximumBytes = 4096; +/** Payload bytes left after reserving one bounded message for every attempt. */ +export const jobRunPayloadEventMaximumBytes = + jobRunOutputMaximumBytes - jobRunAttemptMaximum * jobRunEventMessageMaximumBytes; +export const jobRunEventProgressMaximumBytes = 16 * 1024; +export const jobWorkerCapacityMaximum = 16; +export const jobWorkerFreshnessMs = 30_000; +export const jobWorkerSummaryMaximum = 32; +export const jobIdempotencyKeyMinimumLength = 32; +export const jobIdempotencyKeyMaximumLength = 128; +export const scheduleIdMaximumLength = 80; +export const scheduleCronExpressionMaximumLength = 200; +export const scheduleTimeZoneMaximumLength = 64; +export const scheduleIntervalMinimumMilliseconds = 60_000; +export const scheduleIntervalMaximumMilliseconds = 31_536_000_000; +export const jobTimeoutMinimumMilliseconds = 1000; +export const jobTimeoutMaximumMilliseconds = 86_400_000; + +export type JobCancellationPolicy = (typeof jobCancellationPolicies)[number]; +export type JobResourceClass = (typeof jobResourceClasses)[number]; +export type JobRunEventKind = (typeof jobRunEventKinds)[number]; +export type JobRunState = (typeof jobRunStates)[number]; +export type JobTriggerType = (typeof jobTriggerTypes)[number]; +export type JobWorkerState = (typeof jobWorkerStates)[number]; +export type ScheduleKind = (typeof scheduleKinds)[number]; + +export const jobRunStateSchema = v.picklist(jobRunStates, "Job run state is invalid"); +export const jobTriggerTypeSchema = v.picklist( + jobTriggerTypes, + "Job trigger type is invalid" +); +export const jobResourceClassSchema = v.picklist( + jobResourceClasses, + "Job resource class is invalid" +); +export const jobCancellationPolicySchema = v.picklist( + jobCancellationPolicies, + "Job cancellation policy is invalid" +); +export const jobRunEventKindSchema = v.picklist( + jobRunEventKinds, + "Job run event kind is invalid" +); +export const jobWorkerStateSchema = v.picklist( + jobWorkerStates, + "Job worker state is invalid" +); +export const scheduleKindSchema = v.picklist(scheduleKinds, "Schedule kind is invalid"); + +export const jobTimestampSchema = timestampMillisecondsSchema("Job timestamp is invalid"); +export const jobRunIdSchema = lowercaseUuidV7Schema("Job run id is invalid"); +export const jobRunEventSequenceSchema = v.pipe( + positiveSafeIntegerSchema("Job run event sequence is invalid"), + v.maxValue(jobRunEventMaximum, "Job run event sequence is outside its budget") +); +export const jobWorkerInstanceIdSchema = lowercaseUuidV7Schema( + "Job worker instance id is invalid" +); +export const jobVersionSchema = positiveSafeIntegerSchema("Job version is invalid"); + +/** Canonical Dashboard-owned schedule identity. */ +export const scheduleIdSchema = v.pipe( + v.string("Schedule id is invalid"), + v.minLength(1, "Schedule id is invalid"), + v.maxLength(scheduleIdMaximumLength, "Schedule id is invalid"), + v.regex(/^[a-z0-9][a-z0-9._-]*$/u, "Schedule id is invalid") +); + +/** Canonical action-registry identity captured by schedules and runs. */ +export const jobActionKeySchema = v.pipe( + v.string("Job action key is invalid"), + v.minLength(1, "Job action key is invalid"), + v.maxLength(jobActionKeyMaximumLength, "Job action key is invalid"), + v.regex(/^[a-z0-9][a-z0-9._-]*$/u, "Job action key is invalid") +); + +/** Canonical resource key used for cross-worker exclusivity. */ +export const jobResourceKeySchema = v.pipe( + v.string("Job resource key is invalid"), + v.minLength(1, "Job resource key is invalid"), + v.maxLength(jobResourceKeyMaximumLength, "Job resource key is invalid"), + v.regex(/^[a-z0-9][a-z0-9._-]*$/u, "Job resource key is invalid") +); + +/** Client-generated lost-response-safe key, scoped to the authenticated caller. */ +export const jobIdempotencyKeySchema = v.pipe( + v.string("Job idempotency key is invalid"), + v.minLength(jobIdempotencyKeyMinimumLength, "Job idempotency key is invalid"), + v.maxLength(jobIdempotencyKeyMaximumLength, "Job idempotency key is invalid"), + v.regex(/^[A-Za-z0-9_-]+$/u, "Job idempotency key is invalid"), + v.check(isCanonicalWebAuthnBase64Url, "Job idempotency key is invalid") +); + +export const jobDisplayNameSchema = boundedControlSafeTextSchema( + jobDisplayNameMaximumLength, + "Job display name is invalid" +); +export const jobDescriptionSchema = boundedControlSafeTextSchema( + jobDescriptionMaximumLength, + "Job description is invalid" +); +export const jobRunTerminalCodeSchema = v.pipe( + v.string("Job terminal code is invalid"), + v.minLength(1, "Job terminal code is invalid"), + v.maxLength(jobRunTerminalCodeMaximumLength, "Job terminal code is invalid"), + v.regex(/^[a-z0-9][a-z0-9._/-]*$/u, "Job terminal code is invalid") +); +export const jobRunTerminalMessageSchema = boundedControlSafeTextSchema( + jobRunTerminalMessageMaximumLength, + "Job terminal message is invalid" +); + +function encodedJsonBytes(value: unknown): number { + return utf8ByteLength(JSON.stringify(value)); +} + +function jsonObjectFitsBudget(value: JsonObject, maximumBytes: number): boolean { + return encodedJsonBytes(value) <= maximumBytes; +} + +/** + * @param value Candidate action payload. + * @returns Whether it fits its byte budget. + */ +export function jobPayloadFitsBudget(value: JsonObject): boolean { + return jsonObjectFitsBudget(value, jobPayloadMaximumBytes); +} + +/** + * @param value Candidate public result. + * @returns Whether it fits its byte budget. + */ +export function jobRunResultFitsBudget(value: JsonObject): boolean { + return jsonObjectFitsBudget(value, jobRunResultMaximumBytes); +} + +/** + * @param value Candidate event progress. + * @returns Whether it fits its byte budget. + */ +export function jobRunEventProgressFitsBudget(value: JsonObject): boolean { + return jsonObjectFitsBudget(value, jobRunEventProgressMaximumBytes); +} + +/** Immutable action input retained server-side but never exposed by read models. */ +export const jobPayloadSchema = v.pipe( + jsonObjectSchema, + v.check( + jobPayloadFitsBudget, + `Job payload exceeds ${jobPayloadMaximumBytes} encoded bytes` + ) +); + +/** Redacted structured action result that is safe to expose to job readers. */ +export const jobRunResultSchema = v.pipe( + jsonObjectSchema, + v.check( + jobRunResultFitsBudget, + `Job result exceeds ${jobRunResultMaximumBytes} encoded bytes` + ) +); + +/** Bounded structured progress attached to one durable event. */ +export const jobRunEventProgressSchema = v.pipe( + jsonObjectSchema, + v.check( + jobRunEventProgressFitsBudget, + `Job event progress exceeds ${jobRunEventProgressMaximumBytes} encoded bytes` + ) +); + +/** + * @param keys Resource keys to inspect. + * @returns Whether resource keys are unique and in canonical code-unit order. + */ +export function jobResourceKeysAreCanonical(keys: string[]): boolean { + return ( + hasUniqueArrayItems(keys) && + keys.every((key, index) => { + const previous = keys[index - 1]; + return previous === undefined || compareStrings(previous, key) < 0; + }) && + encodedJsonBytes(keys) <= jobResourceKeysMaximumBytes + ); +} + +/** Canonical sorted resource set captured in schedules and immutable run snapshots. */ +export const jobResourceKeysSchema = v.pipe( + v.array(jobResourceKeySchema, "Job resource keys are invalid"), + v.maxLength(jobResourceKeyMaximum, "Job resource keys are outside their budget"), + v.check(jobResourceKeysAreCanonical, "Job resource keys are not canonical") +); + +export const jobPrioritySchema = v.pipe( + v.number("Job priority is invalid"), + v.safeInteger("Job priority is invalid"), + v.minValue(-100, "Job priority is invalid"), + v.maxValue(100, "Job priority is invalid") +); +export const jobTimeoutSchema = v.pipe( + positiveSafeIntegerSchema("Job timeout is invalid"), + v.minValue(jobTimeoutMinimumMilliseconds, "Job timeout is invalid"), + v.maxValue(jobTimeoutMaximumMilliseconds, "Job timeout is invalid") +); +export const jobAttemptLimitSchema = v.pipe( + positiveSafeIntegerSchema("Job attempt limit is invalid"), + v.maxValue(jobRunAttemptMaximum, "Job attempt limit is invalid") +); +export const jobAttemptCountSchema = v.pipe( + nonnegativeSafeIntegerSchema("Job attempt count is invalid"), + v.maxValue(jobRunAttemptMaximum, "Job attempt count is invalid") +); + +const cronMonthAliases: Readonly> = Object.freeze({ + apr: "4", + aug: "8", + dec: "12", + feb: "2", + jan: "1", + jul: "7", + jun: "6", + mar: "3", + may: "5", + nov: "11", + oct: "10", + sep: "9", +}); +const cronWeekdayAliases: Readonly> = Object.freeze({ + fri: "5", + mon: "1", + sat: "6", + sun: "0", + thu: "4", + tue: "2", + wed: "3", +}); + +function collapseScheduleCronAsciiWhitespace(value: string): string { + let collapsed = ""; + let pendingSpace = false; + for (const character of value) { + const isAsciiWhitespace = + character === " " || + character === "\t" || + character === "\n" || + character === "\v" || + character === "\f" || + character === "\r"; + if (isAsciiWhitespace) { + pendingSpace = collapsed.length > 0; + continue; + } + if (pendingSpace) collapsed += " "; + collapsed += character; + pendingSpace = false; + } + return collapsed; +} + +/** + * Normalizes permitted ASCII whitespace and month/weekday aliases before storage. + * @param value Candidate five-field cron expression. + * @returns Canonically spaced expression with numeric aliases. + */ +export function normalizeScheduleCronExpression(value: string): string { + const collapsed = collapseScheduleCronAsciiWhitespace(value).toLowerCase(); + return collapsed + .split(" ") + .map((field, index) => { + if (index === 3) { + return field.replaceAll( + /\b(?:apr|aug|dec|feb|jan|jul|jun|mar|may|nov|oct|sep)\b/gu, + (alias) => cronMonthAliases[alias] ?? alias + ); + } + if (index === 4) { + return field.replaceAll( + /\b(?:fri|mon|sat|sun|thu|tue|wed)\b/gu, + (alias) => cronWeekdayAliases[alias] ?? alias + ); + } + return field; + }) + .join(" "); +} + +/** + * @param value Normalized cron expression to inspect. + * @returns Whether it is a valid five-field minute cron with a future occurrence. + */ +export function scheduleCronExpressionIsValid(value: string): boolean { + if (!/^[-0-9*,/ ]+$/u.test(value) || value.split(" ").length !== 5) { + return false; + } + const parsed = Cron.parse(value, "UTC"); + if (Result.isFailure(parsed)) return false; + try { + const origin = new Date(0); + const next = Cron.next(parsed.success, origin); + return next.getTime() - origin.getTime() >= scheduleIntervalMinimumMilliseconds; + } catch { + return false; + } +} + +/** Canonical five-field Dashboard-local cron expression. */ +export const scheduleCronExpressionSchema = v.pipe( + v.string("Schedule cron expression is invalid"), + v.maxLength( + scheduleCronExpressionMaximumLength * 2, + "Schedule cron expression is invalid" + ), + v.description( + "Five-field minute cron; live validation accepts JAN-DEC month and SUN-SAT weekday aliases, normalizes aliases and ASCII whitespace, and requires a future occurrence." + ), + v.transform(normalizeScheduleCronExpression), + v.minLength(9, "Schedule cron expression is invalid"), + v.maxLength( + scheduleCronExpressionMaximumLength, + "Schedule cron expression is invalid" + ), + v.check(scheduleCronExpressionIsValid, "Schedule cron expression is invalid") +); + +const canonicalScheduleTimeZoneSet = new Set(canonicalScheduleTimeZones); + +/** + * @param value Candidate time-zone identifier. + * @returns Whether it is an explicit canonical IANA identifier or `UTC`. + */ +export function scheduleTimeZoneIsCanonical(value: string): boolean { + return canonicalScheduleTimeZoneSet.has(value); +} + +export const scheduleTimeZoneSchema = v.pipe( + v.string("Schedule time zone is invalid"), + v.minLength(1, "Schedule time zone is invalid"), + v.maxLength(scheduleTimeZoneMaximumLength, "Schedule time zone is invalid"), + v.check(scheduleTimeZoneIsCanonical, "Schedule time zone is invalid") +); + +export const scheduleTimeOfDaySchema = v.pipe( + v.string("Schedule time of day is invalid"), + v.regex(/^(?:[01]\d|2[0-3]):[0-5]\d$/u, "Schedule time of day is invalid") +); + +const cronScheduleSchema = v.strictObject({ + expression: scheduleCronExpressionSchema, + kind: v.literal("cron"), + timeZone: scheduleTimeZoneSchema, +}); +const dailyScheduleSchema = v.strictObject({ + kind: v.literal("daily"), + timeOfDay: scheduleTimeOfDaySchema, + timeZone: scheduleTimeZoneSchema, +}); +const intervalScheduleSchema = v.strictObject({ + intervalMs: v.pipe( + positiveSafeIntegerSchema("Schedule interval is invalid"), + v.minValue(scheduleIntervalMinimumMilliseconds, "Schedule interval is invalid"), + v.maxValue(scheduleIntervalMaximumMilliseconds, "Schedule interval is invalid") + ), + kind: v.literal("interval"), +}); + +/** Complete mutually exclusive Dashboard-local schedule configuration. */ +export const scheduleConfigurationSchema = v.variant("kind", [ + cronScheduleSchema, + dailyScheduleSchema, + intervalScheduleSchema, +]); + +const jobRunSummaryObjectSchema = v.strictObject({ + actionKey: jobActionKeySchema, + attemptCount: jobAttemptCountSchema, + attemptLimit: jobAttemptLimitSchema, + availableAtMs: jobTimestampSchema, + cancellationPolicy: jobCancellationPolicySchema, + cancelRequestedAtMs: v.optional(jobTimestampSchema), + displayName: jobDisplayNameSchema, + eventCount: v.pipe( + nonnegativeSafeIntegerSchema("Job event count is invalid"), + v.maxValue(jobRunEventMaximum, "Job event count is outside its budget") + ), + finishedAtMs: v.optional(jobTimestampSchema), + firstStartedAtMs: v.optional(jobTimestampSchema), + id: jobRunIdSchema, + lastAttemptStartedAtMs: v.optional(jobTimestampSchema), + priority: jobPrioritySchema, + queuedAtMs: jobTimestampSchema, + resourceClass: jobResourceClassSchema, + resourceKeys: jobResourceKeysSchema, + retrySafe: v.boolean("Job retry-safe policy is invalid"), + scheduledForAtMs: v.optional(jobTimestampSchema), + scheduledJobId: v.optional(scheduleIdSchema), + scheduledJobVersion: v.optional(jobVersionSchema), + state: jobRunStateSchema, + stateVersion: jobVersionSchema, + terminalCode: v.optional(jobRunTerminalCodeSchema), + terminalMessage: v.optional(jobRunTerminalMessageSchema), + timeoutMs: jobTimeoutSchema, + triggerType: jobTriggerTypeSchema, + updatedAtMs: jobTimestampSchema, +}); + +export type JobRunSummary = v.InferOutput; + +/** + * @param run Public run projection to inspect. + * @returns Whether it preserves lifecycle and timestamp invariants. + */ +export function jobRunSummaryIsConsistent(run: JobRunSummary): boolean { + const hasScheduleIdentity = run.scheduledJobId !== undefined; + if (hasScheduleIdentity !== (run.scheduledJobVersion !== undefined)) return false; + if (["manual", "schedule"].includes(run.triggerType) !== hasScheduleIdentity) { + return false; + } + if ( + (run.triggerType === "schedule") !== (run.scheduledForAtMs !== undefined) || + (run.triggerType === "schedule" && !hasScheduleIdentity) + ) { + return false; + } + if (run.attemptCount > run.attemptLimit) return false; + const hasStarted = run.firstStartedAtMs !== undefined; + if ( + hasStarted !== (run.lastAttemptStartedAtMs !== undefined) || + hasStarted !== run.attemptCount > 0 + ) { + return false; + } + + const terminal = ["cancelled", "failed", "succeeded", "timed-out"].includes( + run.state + ); + const hasTerminalError = run.terminalCode !== undefined; + if ( + terminal !== (run.finishedAtMs !== undefined) || + hasTerminalError !== (run.terminalMessage !== undefined) || + (run.state === "succeeded" && hasTerminalError) || + (["cancelled", "failed", "timed-out"].includes(run.state) && !hasTerminalError) || + (["queued", "running"].includes(run.state) && hasTerminalError) + ) { + return false; + } + if ( + ["failed", "running", "succeeded", "timed-out"].includes(run.state) && + run.attemptCount === 0 + ) { + return false; + } + if (run.cancellationPolicy === "never" && run.cancelRequestedAtMs !== undefined) { + return false; + } + + const orderedTimestamps = [ + run.firstStartedAtMs, + run.lastAttemptStartedAtMs, + run.cancelRequestedAtMs, + run.finishedAtMs, + ].filter((timestamp): timestamp is number => timestamp !== undefined); + return ( + run.availableAtMs >= run.queuedAtMs && + run.updatedAtMs >= run.queuedAtMs && + orderedTimestamps.every( + (timestamp) => timestamp >= run.queuedAtMs && timestamp <= run.updatedAtMs + ) && + (run.firstStartedAtMs === undefined || + run.lastAttemptStartedAtMs === undefined || + run.lastAttemptStartedAtMs >= run.firstStartedAtMs) && + (run.finishedAtMs === undefined || + run.lastAttemptStartedAtMs === undefined || + run.finishedAtMs >= run.lastAttemptStartedAtMs) + ); +} + +/** Public run projection without raw payload, worker lease, or fencing data. */ +export const jobRunSummarySchema = v.pipe( + jobRunSummaryObjectSchema, + v.check(jobRunSummaryIsConsistent, "Job run summary is inconsistent") +); + +const jobRunEventMessageSchema = v.pipe( + boundedControlSafeTextSchema( + jobRunEventMessageMaximumLength, + "Job run event message is invalid" + ), + v.check( + jobRunEventMessageFitsBudget, + "Job run event message is outside its byte budget" + ) +); + +/** + * @param message Candidate event message. + * @returns Whether it fits its byte budget. + */ +export function jobRunEventMessageFitsBudget(message: string): boolean { + return utf8ByteLength(message) <= jobRunEventMessageMaximumBytes; +} + +const jobRunEventObjectSchema = v.strictObject({ + attempt: jobAttemptCountSchema, + kind: jobRunEventKindSchema, + message: v.optional(jobRunEventMessageSchema), + occurredAtMs: jobTimestampSchema, + progress: v.optional(jobRunEventProgressSchema), + sequence: jobRunEventSequenceSchema, + workerInstanceId: v.optional(jobWorkerInstanceIdSchema), +}); + +export type JobRunEvent = v.InferOutput; + +/** + * @param event Durable run event to inspect. + * @returns Whether its kind agrees with required bounded payload fields. + */ +export function jobRunEventIsConsistent(event: JobRunEvent): boolean { + if (event.kind === "progress") return event.progress !== undefined; + if (event.kind === "stderr" || event.kind === "stdout") { + return event.message !== undefined && event.progress === undefined; + } + return event.progress === undefined; +} + +/** One durable bounded progress or lifecycle event. */ +export const jobRunEventSchema = v.pipe( + jobRunEventObjectSchema, + v.check(jobRunEventIsConsistent, "Job run event payload is inconsistent") +); + +const jobWorkerSummaryObjectSchema = v.strictObject({ + activeRunCount: v.pipe( + nonnegativeSafeIntegerSchema("Worker active-run count is invalid"), + v.maxValue(jobWorkerCapacityMaximum, "Worker active-run count is invalid") + ), + capacity: v.pipe( + positiveSafeIntegerSchema("Worker capacity is invalid"), + v.maxValue(jobWorkerCapacityMaximum, "Worker capacity is invalid") + ), + drainingAtMs: v.optional(jobTimestampSchema), + heartbeatAtMs: jobTimestampSchema, + id: jobWorkerInstanceIdSchema, + releaseId: fullCommitShaSchema("Worker release id is invalid"), + startedAtMs: jobTimestampSchema, + state: jobWorkerStateSchema, + stoppedAtMs: v.optional(jobTimestampSchema), +}); + +type JobWorkerSummaryValue = v.InferOutput; + +/** + * @param worker Public worker summary to inspect. + * @returns Whether worker state and lifecycle timestamps agree. + */ +export function jobWorkerSummaryIsConsistent(worker: JobWorkerSummaryValue): boolean { + if (worker.activeRunCount > worker.capacity) return false; + if (worker.state === "stopped" && worker.activeRunCount !== 0) return false; + if (worker.heartbeatAtMs < worker.startedAtMs) return false; + if ( + (worker.state === "online" && + (worker.drainingAtMs !== undefined || worker.stoppedAtMs !== undefined)) || + (worker.state === "draining" && + (worker.drainingAtMs === undefined || worker.stoppedAtMs !== undefined)) || + (worker.state === "stopped" && + (worker.drainingAtMs === undefined || worker.stoppedAtMs === undefined)) + ) { + return false; + } + return ( + (worker.drainingAtMs === undefined || + worker.drainingAtMs >= worker.startedAtMs) && + (worker.stoppedAtMs === undefined || + worker.stoppedAtMs >= (worker.drainingAtMs ?? worker.startedAtMs)) + ); +} + +export const jobWorkerSummarySchema = v.pipe( + jobWorkerSummaryObjectSchema, + v.check(jobWorkerSummaryIsConsistent, "Worker summary is inconsistent") +); + +/** Versioned singleton state controlling cross-process claims. */ +export const jobWorkerControlSchema = v.strictObject({ + claimingPaused: v.boolean("Worker claiming state is invalid"), + updatedAtMs: jobTimestampSchema, + version: jobVersionSchema, +}); + +/** Active operator disable intent attached to one schedule. */ +const activeJobDisableIntentObjectSchema = v.strictObject({ + createdAtMs: jobTimestampSchema, + expiresAtMs: v.optional(jobTimestampSchema), + id: lowercaseUuidV7Schema("Job disable intent id is invalid"), + reason: boundedControlSafeTextSchema( + jobDescriptionMaximumLength, + "Job disable reason is invalid" + ), +}); + +type ActiveJobDisableIntentValue = v.InferOutput< + typeof activeJobDisableIntentObjectSchema +>; + +/** + * @param intent Active disable intent to inspect. + * @returns Whether an optional expiry is strictly after creation. + */ +export function activeJobDisableIntentTimesAreConsistent( + intent: ActiveJobDisableIntentValue +): boolean { + return intent.expiresAtMs === undefined || intent.expiresAtMs > intent.createdAtMs; +} + +export const activeJobDisableIntentSchema = v.pipe( + activeJobDisableIntentObjectSchema, + v.check( + activeJobDisableIntentTimesAreConsistent, + "Job disable intent timestamps are inconsistent" + ) +); + +const scheduleSummaryObjectSchema = v.strictObject({ + actionKey: jobActionKeySchema, + activeDisableIntent: v.optional(activeJobDisableIntentSchema), + activeRun: v.optional(jobRunSummarySchema), + attemptLimit: jobAttemptLimitSchema, + cancellationPolicy: jobCancellationPolicySchema, + createdAtMs: jobTimestampSchema, + description: jobDescriptionSchema, + enabled: v.boolean("Schedule enabled state is invalid"), + id: scheduleIdSchema, + latestRun: v.optional(jobRunSummarySchema), + name: jobDisplayNameSchema, + nextRunAtMs: v.optional(jobTimestampSchema), + priority: jobPrioritySchema, + resourceClass: jobResourceClassSchema, + resourceKeys: jobResourceKeysSchema, + retrySafe: v.boolean("Schedule retry-safe policy is invalid"), + schedule: scheduleConfigurationSchema, + timeoutMs: jobTimeoutSchema, + updatedAtMs: jobTimestampSchema, + version: jobVersionSchema, +}); + +export type ScheduleSummary = v.InferOutput; + +/** + * @param schedule Public schedule summary to inspect. + * @returns Whether state, timestamps, and embedded run references agree. + */ +export function scheduleSummaryIsConsistent(schedule: ScheduleSummary): boolean { + if (schedule.enabled !== (schedule.nextRunAtMs !== undefined)) return false; + if (schedule.enabled && schedule.activeDisableIntent !== undefined) return false; + if (schedule.updatedAtMs < schedule.createdAtMs) return false; + for (const run of [schedule.activeRun, schedule.latestRun]) { + if (run !== undefined && run.scheduledJobId !== schedule.id) return false; + } + if ( + schedule.activeRun !== undefined && + !["queued", "running"].includes(schedule.activeRun.state) + ) { + return false; + } + return ( + schedule.activeRun === undefined || + schedule.latestRun === undefined || + schedule.activeRun.id === schedule.latestRun.id + ); +} + +/** Public schedule projection without its raw action payload. */ +export const scheduleSummarySchema = v.pipe( + scheduleSummaryObjectSchema, + v.check(scheduleSummaryIsConsistent, "Schedule summary is inconsistent") +); + +export type ActiveJobDisableIntent = v.InferOutput; +export type JobRunResult = v.InferOutput; +export type JobWorkerControl = v.InferOutput; +export type JobWorkerSummary = v.InferOutput; +export type ScheduleConfiguration = v.InferOutput; diff --git a/greenfield/src/contracts/jobProcedurePolicies.ts b/greenfield/src/contracts/jobProcedurePolicies.ts new file mode 100644 index 000000000..695a1a839 --- /dev/null +++ b/greenfield/src/contracts/jobProcedurePolicies.ts @@ -0,0 +1,28 @@ +/** Shared authenticated read policy for durable jobs and schedules. */ +export const jobReadAccess = { + capabilities: ["jobs:read"], + capabilityPolicy: "all", + kind: "authenticated", +} as const; + +/** Shared browser-session mutation policy for durable jobs and schedules. */ +export const jobSessionWriteAccess = { + capabilities: ["jobs:write"], + capabilityPolicy: "all", + kind: "authenticated", + principalKinds: ["session"], +} as const; + +/** Shared transport policy for durable job-domain queries. */ +export const jobQueryTransport = { + batching: "adapter-default", + handler: "default", + requestBody: "default", +} as const; + +/** Shared transport policy for durable job-domain mutations. */ +export const jobMutationTransport = { + batching: "forbidden", + handler: "default", + requestBody: "default", +} as const; diff --git a/greenfield/src/contracts/jobRealtime.test.ts b/greenfield/src/contracts/jobRealtime.test.ts new file mode 100644 index 000000000..7a33a22de --- /dev/null +++ b/greenfield/src/contracts/jobRealtime.test.ts @@ -0,0 +1,146 @@ +import { describe, expect, test } from "bun:test"; + +import * as v from "valibot"; + +import { + findJobRealtimeTopicDefinition, + jobChangePayloadSchema, + jobRealtimeChangeSchemas, + jobRealtimeEventContracts, + jobRealtimeRoutingSchema, + jobRealtimeTopicDefinitions, + jobRealtimeTopics, +} from "./jobRealtime.ts"; + +const runId = "018f6f50-6a9e-7b88-8000-000000000001"; + +describe("durable job realtime contracts", () => { + test("registers exact read-authorized topics and seven-day snapshots", () => { + expect(jobRealtimeTopics).toEqual({ + runs: "jobs.runs", + schedules: "schedules.records", + }); + expect( + jobRealtimeTopicDefinitions.map(({ capability, topic }) => ({ + capability, + topic, + })) + ).toEqual([ + { capability: "jobs:read", topic: "jobs.runs" }, + { capability: "jobs:read", topic: "schedules.records" }, + ]); + expect( + jobRealtimeEventContracts.map(({ retention, snapshotProcedure, topic }) => ({ + retention, + snapshotProcedure, + topic, + })) + ).toEqual([ + { + retention: "7 days", + snapshotProcedure: "jobs.listRuns", + topic: "jobs.runs", + }, + { + retention: "7 days", + snapshotProcedure: "schedules.list", + topic: "schedules.records", + }, + ]); + expect(findJobRealtimeTopicDefinition("jobs.runs")?.capability).toBe("jobs:read"); + expect(findJobRealtimeTopicDefinition("jobs.unknown")).toBeUndefined(); + }); + + test("keeps run, queue-summary, and schedule producer routes distinct", () => { + for (const route of [ + { + entityType: "job-run", + operation: "created", + topic: "jobs.runs", + }, + { + entityType: "job-queue", + operation: "snapshot-required", + topic: "jobs.runs", + }, + { + entityType: "schedule", + operation: "updated", + topic: "schedules.records", + }, + ]) { + expect(v.safeParse(jobRealtimeRoutingSchema, route).success).toBeTrue(); + } + + for (const route of [ + { + entityType: "job-queue", + operation: "updated", + topic: "jobs.runs", + }, + { + entityType: "schedule", + operation: "snapshot-required", + topic: "schedules.records", + }, + { + entityType: "job-run", + operation: "created", + topic: "schedules.records", + }, + ]) { + expect(v.safeParse(jobRealtimeRoutingSchema, route).success).toBeFalse(); + } + }); + + test("accepts compact IDs and validates client deliveries with exact entity shapes", () => { + expect(v.parse(jobChangePayloadSchema, { id: runId })).toEqual({ id: runId }); + expect(v.parse(jobChangePayloadSchema, { id: "queue" })).toEqual({ + id: "queue", + }); + expect( + v.safeParse(jobChangePayloadSchema, { id: "Invalid ID" }).success + ).toBeFalse(); + + const changes = [ + { + entityId: runId, + entityType: "job-run", + occurredAtMs: 1000, + operation: "updated", + payload: { id: runId }, + topic: "jobs.runs", + }, + { + entityId: "queue", + entityType: "job-queue", + occurredAtMs: 1000, + operation: "snapshot-required", + payload: { id: "queue" }, + topic: "jobs.runs", + }, + { + entityId: "system.worker-smoke", + entityType: "schedule", + occurredAtMs: 1000, + operation: "updated", + payload: { id: "system.worker-smoke" }, + topic: "schedules.records", + }, + ]; + + for (const [index, schema] of jobRealtimeChangeSchemas.entries()) { + expect(v.safeParse(schema, changes[index]).success).toBeTrue(); + } + + const mismatchedPayloadIds = ["system.worker-smoke", runId, runId]; + for (const [index, schema] of jobRealtimeChangeSchemas.entries()) { + expect( + v.safeParse(schema, { + ...changes[index], + payload: { id: mismatchedPayloadIds[index] }, + }).success + ).toBeFalse(); + } + }); +}); diff --git a/greenfield/src/contracts/jobRealtime.ts b/greenfield/src/contracts/jobRealtime.ts new file mode 100644 index 000000000..a061acb3e --- /dev/null +++ b/greenfield/src/contracts/jobRealtime.ts @@ -0,0 +1,158 @@ +import * as v from "valibot"; + +import { timestampMillisecondsSchema } from "../shared/dateTime.ts"; +import { jobRunIdSchema, scheduleIdSchema } from "./jobModel.ts"; +import { realtimeEventRetentionLabel, type RealtimeTopicDefinition } from "./realtime.ts"; +import type { RealtimeEventContract } from "./registry.ts"; + +/** Durable job and schedule invalidation topics. */ +export const jobRealtimeTopics = Object.freeze({ + runs: "jobs.runs", + schedules: "schedules.records", +}); + +const jobRunEntityType = "job-run"; +const jobQueueEntityType = "job-queue"; +const scheduleEntityType = "schedule"; +const jobRunOperations = ["created", "updated"] as const; +const jobQueueOperations = ["snapshot-required"] as const; +const scheduleOperations = ["created", "updated"] as const; + +const jobRunRoutingEntries = { + entityType: v.literal(jobRunEntityType), + operation: v.picklist(jobRunOperations), + topic: v.literal(jobRealtimeTopics.runs), +}; +const jobQueueRoutingEntries = { + entityType: v.literal(jobQueueEntityType), + operation: v.picklist(jobQueueOperations), + topic: v.literal(jobRealtimeTopics.runs), +}; +const scheduleRoutingEntries = { + entityType: v.literal(scheduleEntityType), + operation: v.picklist(scheduleOperations), + topic: v.literal(jobRealtimeTopics.schedules), +}; + +/** Producer routing that keeps run, queue-summary, and schedule events distinct. */ +export const jobRealtimeRoutingSchema = v.variant("entityType", [ + v.strictObject(jobRunRoutingEntries), + v.strictObject(jobQueueRoutingEntries), + v.strictObject(scheduleRoutingEntries), +]); + +/** Compact invalidation payload shared by job and schedule consumers. */ +export const jobChangePayloadSchema = v.strictObject({ + id: v.union([jobRunIdSchema, scheduleIdSchema]), +}); + +/** Topic-specific capability, entity, operation, and payload policies. */ +export const jobRealtimeTopicDefinitions = [ + { + capability: "jobs:read", + entityTypes: [jobQueueEntityType, jobRunEntityType], + operations: [...jobQueueOperations, ...jobRunOperations], + payload: jobChangePayloadSchema, + topic: jobRealtimeTopics.runs, + }, + { + capability: "jobs:read", + entityTypes: [scheduleEntityType], + operations: scheduleOperations, + payload: jobChangePayloadSchema, + topic: jobRealtimeTopics.schedules, + }, +] as const satisfies readonly RealtimeTopicDefinition[]; + +/** Standalone durable-job invalidations and their authoritative snapshots. */ +export const jobRealtimeEventContracts = [ + { + payload: jobChangePayloadSchema, + payloadSchemaId: "jobs.runs.realtime.payload", + retention: realtimeEventRetentionLabel, + snapshotProcedure: "jobs.listRuns", + summary: "Invalidates durable run rows and exact queue state.", + topic: jobRealtimeTopics.runs, + }, + { + payload: jobChangePayloadSchema, + payloadSchemaId: "schedules.records.realtime.payload", + retention: realtimeEventRetentionLabel, + snapshotProcedure: "schedules.list", + summary: "Invalidates the code-owned Dashboard schedule directory.", + topic: jobRealtimeTopics.schedules, + }, +] as const satisfies readonly RealtimeEventContract[]; + +/** + * Finds one exact durable-job topic policy. + * @param topic Candidate durable topic. + * @returns Its registered definition, when present. + */ +export function findJobRealtimeTopicDefinition(topic: string) { + return jobRealtimeTopicDefinitions.find((definition) => definition.topic === topic); +} + +const jobRealtimeTimestampSchema = timestampMillisecondsSchema( + "Job realtime timestamp is invalid" +); + +/** + * @param event Compact durable-job invalidation envelope. + * @returns Whether its routing and payload identities name the same entity. + */ +export function jobRealtimeIdentityMatches(event: { + readonly entityId: string; + readonly payload: { readonly id: string }; +}): boolean { + return event.payload.id === event.entityId; +} + +const jobRunRealtimeChangeObjectSchema = v.strictObject({ + entityId: jobRunIdSchema, + entityType: jobRunRoutingEntries.entityType, + occurredAtMs: jobRealtimeTimestampSchema, + operation: jobRunRoutingEntries.operation, + payload: jobChangePayloadSchema, + topic: jobRunRoutingEntries.topic, +}); +const jobQueueRealtimeChangeObjectSchema = v.strictObject({ + entityId: scheduleIdSchema, + entityType: jobQueueRoutingEntries.entityType, + occurredAtMs: jobRealtimeTimestampSchema, + operation: jobQueueRoutingEntries.operation, + payload: jobChangePayloadSchema, + topic: jobQueueRoutingEntries.topic, +}); +const scheduleRealtimeChangeObjectSchema = v.strictObject({ + entityId: scheduleIdSchema, + entityType: scheduleRoutingEntries.entityType, + occurredAtMs: jobRealtimeTimestampSchema, + operation: scheduleRoutingEntries.operation, + payload: jobChangePayloadSchema, + topic: scheduleRoutingEntries.topic, +}); + +const jobRealtimeIdentityMessage = "Job realtime entity identity is inconsistent"; + +function withMatchingJobRealtimeIdentity< + TSchema extends v.GenericSchema< + unknown, + { readonly entityId: string; readonly payload: { readonly id: string } } + >, +>(schema: TSchema) { + return v.pipe( + schema, + v.check, typeof jobRealtimeIdentityMessage>( + jobRealtimeIdentityMatches, + jobRealtimeIdentityMessage + ) + ); +} + +/** Topic-specific client change schemas built from the producer routing policy. */ +export const jobRealtimeChangeSchemas = [ + withMatchingJobRealtimeIdentity(jobRunRealtimeChangeObjectSchema), + withMatchingJobRealtimeIdentity(jobQueueRealtimeChangeObjectSchema), + withMatchingJobRealtimeIdentity(scheduleRealtimeChangeObjectSchema), +] as const; diff --git a/greenfield/src/contracts/jobs.test.ts b/greenfield/src/contracts/jobs.test.ts new file mode 100644 index 000000000..82a7df14c --- /dev/null +++ b/greenfield/src/contracts/jobs.test.ts @@ -0,0 +1,260 @@ +import { describe, expect, test } from "bun:test"; + +import * as v from "valibot"; + +import { + getJobRunInputSchema, + jobProcedureContracts, + jobRunDetailSchema, + jobRunPageMaximum, + listJobRunsInputSchema, + listJobRunsResultSchema, +} from "./jobs.ts"; + +const firstRunId = "018f6f50-6a9e-7b88-8000-000000000002"; +const secondRunId = "018f6f50-6a9e-7b88-8000-000000000001"; + +function queuedRun(id: string, queuedAtMs: number) { + return { + actionKey: "system.worker-smoke", + attemptCount: 0, + attemptLimit: 3, + availableAtMs: queuedAtMs, + cancellationPolicy: "cooperative" as const, + displayName: "Worker smoke", + eventCount: 1, + id, + priority: 0, + queuedAtMs, + resourceClass: "light" as const, + resourceKeys: ["database"], + retrySafe: true, + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + state: "queued" as const, + stateVersion: 1, + timeoutMs: 30_000, + triggerType: "manual" as const, + updatedAtMs: queuedAtMs, + }; +} + +function queueSummary() { + return { + activeResourceClasses: [], + control: { claimingPaused: false, updatedAtMs: 500, version: 1 }, + oldestQueuedAtMs: 1000, + stateCounts: { + cancelled: 0, + failed: 0, + queued: 2, + running: 0, + succeeded: 0, + "timed-out": 0, + }, + workers: [], + }; +} + +describe("job procedure contracts", () => { + test("locks the four procedures to read or session-only write access", () => { + expect( + jobProcedureContracts.map(({ access, kind, name, transport }) => ({ + access, + batching: transport.batching, + kind, + name, + })) + ).toEqual([ + { + access: { + capabilities: ["jobs:read"], + capabilityPolicy: "all", + kind: "authenticated", + }, + batching: "adapter-default", + kind: "query", + name: "jobs.listRuns", + }, + { + access: { + capabilities: ["jobs:read"], + capabilityPolicy: "all", + kind: "authenticated", + }, + batching: "adapter-default", + kind: "query", + name: "jobs.getRun", + }, + { + access: { + capabilities: ["jobs:write"], + capabilityPolicy: "all", + kind: "authenticated", + principalKinds: ["session"], + }, + batching: "forbidden", + kind: "mutation", + name: "jobs.cancelRun", + }, + { + access: { + capabilities: ["jobs:write"], + capabilityPolicy: "all", + kind: "authenticated", + principalKinds: ["session"], + }, + batching: "forbidden", + kind: "mutation", + name: "jobs.setClaimingPaused", + }, + ]); + }); + + test("defaults and bounds stable run and event requests", () => { + expect(v.parse(listJobRunsInputSchema, {})).toEqual({ limit: 50 }); + expect( + v.parse(listJobRunsInputSchema, { + cursor: { id: firstRunId, queuedAtMs: 2000 }, + filters: { + resourceClasses: ["light"], + scheduleId: "system.worker-smoke", + states: ["queued", "running"], + triggerTypes: ["manual"], + }, + limit: jobRunPageMaximum, + }).limit + ).toBe(jobRunPageMaximum); + + for (const input of [ + { limit: 0 }, + { limit: jobRunPageMaximum + 1 }, + { filters: { states: ["queued", "queued"] } }, + { filters: { resourceClasses: [] } }, + ]) { + expect(v.safeParse(listJobRunsInputSchema, input).success).toBeFalse(); + } + + expect(v.parse(getJobRunInputSchema, { id: firstRunId })).toEqual({ + eventLimit: 50, + id: firstRunId, + }); + }); + + test("requires newest-first rows and an exact continuation cursor", () => { + const runs = [queuedRun(firstRunId, 2000), queuedRun(secondRunId, 1000)]; + expect( + v + .parse(listJobRunsResultSchema, { + nextCursor: { id: secondRunId, queuedAtMs: 1000 }, + runs, + summary: queueSummary(), + }) + .runs.map(({ id }) => id) + ).toEqual([firstRunId, secondRunId]); + + for (const result of [ + { runs: runs.toReversed(), summary: queueSummary() }, + { + nextCursor: { id: firstRunId, queuedAtMs: 2000 }, + runs, + summary: queueSummary(), + }, + { + runs, + summary: { + ...queueSummary(), + oldestQueuedAtMs: undefined, + }, + }, + ]) { + expect(v.safeParse(listJobRunsResultSchema, result).success).toBeFalse(); + } + }); + + test("validates redacted successful detail and bounded newest-first events", () => { + const run = { + ...queuedRun(firstRunId, 1000), + attemptCount: 1, + eventCount: 3, + finishedAtMs: 3000, + firstStartedAtMs: 2000, + lastAttemptStartedAtMs: 2000, + state: "succeeded" as const, + stateVersion: 3, + updatedAtMs: 3000, + }; + const events = [ + { + attempt: 1, + kind: "succeeded" as const, + occurredAtMs: 3000, + sequence: 3, + }, + { + attempt: 1, + kind: "claimed" as const, + occurredAtMs: 2000, + sequence: 2, + }, + ]; + expect( + v.parse(jobRunDetailSchema, { + events, + nextEventCursor: { sequence: 2 }, + result: { status: "ok" }, + run, + }).result + ).toEqual({ status: "ok" }); + + for (const detail of [ + { events, run }, + { events: events.toReversed(), result: { status: "ok" }, run }, + { + events, + nextEventCursor: { sequence: 3 }, + result: { status: "ok" }, + run, + }, + { + events: [{ ...events[0], sequence: 4 }], + result: { status: "ok" }, + run, + }, + { + events, + payload: { hidden: true }, + result: { status: "ok" }, + run, + }, + ]) { + expect(v.safeParse(jobRunDetailSchema, detail).success).toBeFalse(); + } + }); + + test("declares stable sorted error sets", () => { + expect( + jobProcedureContracts.map(({ errors, name }) => ({ errors, name })) + ).toEqual([ + { errors: ["FORBIDDEN", "UNAUTHORIZED"], name: "jobs.listRuns" }, + { + errors: ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + name: "jobs.getRun", + }, + { + errors: [ + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + name: "jobs.cancelRun", + }, + { + errors: ["CONFLICT", "FORBIDDEN", "SERVICE_UNAVAILABLE", "UNAUTHORIZED"], + name: "jobs.setClaimingPaused", + }, + ]); + }); +}); diff --git a/greenfield/src/contracts/jobs.ts b/greenfield/src/contracts/jobs.ts new file mode 100644 index 000000000..cc3b5fcf6 --- /dev/null +++ b/greenfield/src/contracts/jobs.ts @@ -0,0 +1,365 @@ +import * as v from "valibot"; + +import { + compareStrings, + hasUniqueArrayItems, + nonnegativeSafeIntegerSchema, +} from "../shared/validation.ts"; +import { enumFilterSchema } from "./filterSchemas.ts"; +import { + type JobRunEvent, + type JobRunSummary, + type JobResourceClass, + type JobWorkerSummary, + jobResourceClasses, + jobResourceClassSchema, + jobRunEventSchema, + jobRunEventSequenceSchema, + jobRunIdSchema, + jobRunResultSchema, + jobRunStates, + jobRunSummarySchema, + jobTimestampSchema, + jobTriggerTypes, + jobVersionSchema, + jobWorkerControlSchema, + jobWorkerSummaryMaximum, + jobWorkerSummarySchema, + scheduleIdSchema, +} from "./jobModel.ts"; +import { + jobMutationTransport, + jobQueryTransport, + jobReadAccess, + jobSessionWriteAccess, +} from "./jobProcedurePolicies.ts"; +import type { ProcedureContract } from "./registry.ts"; + +/** Default durable runs returned by one request. */ +export const jobRunPageDefault = 50; +/** Hard durable run-row budget for one response. */ +export const jobRunPageMaximum = 100; +/** Default run events returned by one exact-detail request. */ +export const jobRunEventPageDefault = 50; +/** Hard run-event budget for one detail response. */ +export const jobRunEventPageMaximum = 100; + +const jobRunFilterMaximum = 16; +const jobRunLimitSchema = v.pipe( + v.number("Job run page limit is invalid"), + v.safeInteger("Job run page limit is invalid"), + v.minValue(1, "Job run page limit is invalid"), + v.maxValue(jobRunPageMaximum, "Job run page limit is outside its budget") +); +const jobRunEventLimitSchema = v.pipe( + v.number("Job event page limit is invalid"), + v.safeInteger("Job event page limit is invalid"), + v.minValue(1, "Job event page limit is invalid"), + v.maxValue(jobRunEventPageMaximum, "Job event page limit is outside its budget") +); + +/** Stable newest-first cursor for global and schedule-scoped run history. */ +export const jobRunCursorSchema = v.strictObject({ + id: jobRunIdSchema, + queuedAtMs: jobTimestampSchema, +}); + +/** Stable newest-first cursor for one run's event history. */ +export const jobRunEventCursorSchema = v.strictObject({ + sequence: jobRunEventSequenceSchema, +}); + +/** Bounded filters supported by the durable run inventory. */ +export const jobRunFiltersSchema = v.strictObject({ + resourceClasses: v.optional( + enumFilterSchema(jobResourceClasses, "Job resource class", jobRunFilterMaximum) + ), + scheduleId: v.optional(scheduleIdSchema), + states: v.optional( + enumFilterSchema(jobRunStates, "Job run state", jobRunFilterMaximum) + ), + triggerTypes: v.optional( + enumFilterSchema(jobTriggerTypes, "Job trigger type", jobRunFilterMaximum) + ), +}); + +/** One stable keyset-paginated global run request. */ +export const listJobRunsInputSchema = v.strictObject({ + cursor: v.optional(jobRunCursorSchema), + filters: v.optional(jobRunFiltersSchema), + limit: v.optional(jobRunLimitSchema, jobRunPageDefault), +}); + +/** + * @param runs Run summaries to inspect. + * @returns Whether they use strict newest-first cursor order. + */ +export function newestJobRunOrderIsStable(runs: JobRunSummary[]): boolean { + return runs.every((run, index) => { + const previous = runs[index - 1]; + return ( + previous === undefined || + run.queuedAtMs < previous.queuedAtMs || + (run.queuedAtMs === previous.queuedAtMs && run.id < previous.id) + ); + }); +} + +/** Bounded stable run rows reused by global and schedule-scoped history. */ +export const jobRunPageSchema = v.pipe( + v.array(jobRunSummarySchema, "Job run page is invalid"), + v.maxLength(jobRunPageMaximum, "Job run page is outside its budget"), + v.check(newestJobRunOrderIsStable, "Job run page order is invalid") +); + +const jobRunStateCountsSchema = v.strictObject({ + cancelled: nonnegativeSafeIntegerSchema("Cancelled job count is invalid"), + failed: nonnegativeSafeIntegerSchema("Failed job count is invalid"), + queued: nonnegativeSafeIntegerSchema("Queued job count is invalid"), + running: nonnegativeSafeIntegerSchema("Running job count is invalid"), + succeeded: nonnegativeSafeIntegerSchema("Succeeded job count is invalid"), + "timed-out": nonnegativeSafeIntegerSchema("Timed-out job count is invalid"), +}); + +/** + * @param resourceClasses Active resource classes to inspect. + * @returns Whether the set is unique and in canonical order. + */ +export function activeJobResourceClassesAreCanonical( + resourceClasses: JobResourceClass[] +): boolean { + return ( + hasUniqueArrayItems(resourceClasses) && + resourceClasses.every((resourceClass, index) => { + const previous = resourceClasses[index - 1]; + return previous === undefined || compareStrings(previous, resourceClass) < 0; + }) + ); +} + +const activeJobResourceClassesSchema = v.pipe( + v.array(jobResourceClassSchema, "Active job resource classes are invalid"), + v.maxLength( + jobResourceClasses.length, + "Active job resource classes are outside their budget" + ), + v.check( + activeJobResourceClassesAreCanonical, + "Active job resource classes are not canonical" + ) +); + +/** + * @param workers Worker summaries to inspect. + * @returns Whether they have unique IDs in canonical order. + */ +export function jobWorkerSummariesAreCanonical(workers: JobWorkerSummary[]): boolean { + return ( + hasUniqueArrayItems(workers.map(({ id }) => id)) && + workers.every((worker, index) => { + const previous = workers[index - 1]; + return previous === undefined || compareStrings(previous.id, worker.id) < 0; + }) + ); +} + +const jobWorkerSummariesSchema = v.pipe( + v.array(jobWorkerSummarySchema, "Job worker summaries are invalid"), + v.maxLength(jobWorkerSummaryMaximum, "Job worker summaries are outside their budget"), + v.check(jobWorkerSummariesAreCanonical, "Job worker summaries are not canonical") +); + +const jobQueueSummaryObjectSchema = v.strictObject({ + activeResourceClasses: activeJobResourceClassesSchema, + control: jobWorkerControlSchema, + oldestQueuedAtMs: v.optional(jobTimestampSchema), + stateCounts: jobRunStateCountsSchema, + workers: jobWorkerSummariesSchema, +}); + +type JobQueueSummaryValue = v.InferOutput; + +/** + * @param summary Exact queue summary to inspect. + * @returns Whether counts agree with optional derived fields. + */ +export function jobQueueSummaryIsConsistent(summary: JobQueueSummaryValue): boolean { + return ( + summary.stateCounts.queued > 0 === (summary.oldestQueuedAtMs !== undefined) && + summary.stateCounts.running > 0 === summary.activeResourceClasses.length > 0 + ); +} + +/** Exact bounded worker and queue projection returned with global run history. */ +export const jobQueueSummarySchema = v.pipe( + jobQueueSummaryObjectSchema, + v.check(jobQueueSummaryIsConsistent, "Job queue summary is inconsistent") +); + +const listJobRunsResultObjectSchema = v.strictObject({ + nextCursor: v.optional(jobRunCursorSchema), + runs: jobRunPageSchema, + summary: jobQueueSummarySchema, +}); + +type ListJobRunsResultValue = v.InferOutput; + +/** + * @param result Run page and cursor to inspect. + * @returns Whether an optional cursor identifies the final returned row. + */ +export function jobRunPageCursorIsConsistent(result: ListJobRunsResultValue): boolean { + if (result.nextCursor === undefined) return true; + const last = result.runs.at(-1); + return ( + last !== undefined && + last.id === result.nextCursor.id && + last.queuedAtMs === result.nextCursor.queuedAtMs + ); +} + +/** Stable global run page plus exact queue summary and continuation cursor. */ +export const listJobRunsResultSchema = v.pipe( + listJobRunsResultObjectSchema, + v.check(jobRunPageCursorIsConsistent, "Job run page cursor is inconsistent") +); + +/** + * @param events Durable run events to inspect. + * @returns Whether they use strict newest-first sequence order. + */ +export function newestJobRunEventOrderIsStable(events: JobRunEvent[]): boolean { + return events.every((event, index) => { + const previous = events[index - 1]; + return previous === undefined || event.sequence < previous.sequence; + }); +} + +const jobRunEventPageSchema = v.pipe( + v.array(jobRunEventSchema, "Job run event page is invalid"), + v.maxLength(jobRunEventPageMaximum, "Job run event page is outside its budget"), + v.check(newestJobRunEventOrderIsStable, "Job run event page order is invalid") +); + +const jobRunDetailObjectSchema = v.strictObject({ + events: jobRunEventPageSchema, + nextEventCursor: v.optional(jobRunEventCursorSchema), + result: v.optional(jobRunResultSchema), + run: jobRunSummarySchema, +}); + +export type JobRunDetail = v.InferOutput; + +/** + * @param detail Public run detail to inspect. + * @returns Whether it agrees with run state, event count, and cursor. + */ +export function jobRunDetailIsConsistent(detail: JobRunDetail): boolean { + if ((detail.run.state === "succeeded") !== (detail.result !== undefined)) { + return false; + } + if ( + detail.events.some( + (event) => + event.sequence > detail.run.eventCount || + event.attempt > detail.run.attemptCount + ) + ) { + return false; + } + if (detail.nextEventCursor === undefined) return true; + return detail.events.at(-1)?.sequence === detail.nextEventCursor.sequence; +} + +/** Complete public run detail without raw input or lease/fencing internals. */ +export const jobRunDetailSchema = v.pipe( + jobRunDetailObjectSchema, + v.check(jobRunDetailIsConsistent, "Job run detail is inconsistent") +); + +/** Exact run lookup with one bounded newest-first event page. */ +export const getJobRunInputSchema = v.strictObject({ + eventCursor: v.optional(jobRunEventCursorSchema), + eventLimit: v.optional(jobRunEventLimitSchema, jobRunEventPageDefault), + id: jobRunIdSchema, +}); + +/** Exact session-owned cancellation request. */ +export const cancelJobRunInputSchema = v.strictObject({ id: jobRunIdSchema }); + +/** Versioned cross-process claim-pause update. */ +export const setJobClaimingPausedInputSchema = v.strictObject({ + expectedVersion: jobVersionSchema, + paused: v.boolean("Worker claiming state is invalid"), +}); + +/** Durable job inventory, detail, cancellation, and worker-control contracts. */ +export const jobProcedureContracts = [ + { + access: jobReadAccess, + domain: "jobs", + errors: ["FORBIDDEN", "UNAUTHORIZED"], + input: listJobRunsInputSchema, + inputSchemaId: "jobs.listRuns.input", + kind: "query", + name: "jobs.listRuns", + output: listJobRunsResultSchema, + outputSchemaId: "jobs.listRuns.output", + summary: "Lists stable newest-first durable run history and queue state.", + transport: jobQueryTransport, + }, + { + access: jobReadAccess, + domain: "jobs", + errors: ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + input: getJobRunInputSchema, + inputSchemaId: "jobs.getRun.input", + kind: "query", + name: "jobs.getRun", + output: jobRunDetailSchema, + outputSchemaId: "jobs.getRun.output", + summary: "Loads one durable run with bounded newest-first events.", + transport: jobQueryTransport, + }, + { + access: jobSessionWriteAccess, + domain: "jobs", + errors: [ + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + input: cancelJobRunInputSchema, + inputSchemaId: "jobs.cancelRun.input", + kind: "mutation", + name: "jobs.cancelRun", + output: jobRunSummarySchema, + outputSchemaId: "jobs.cancelRun.output", + summary: "Cancels a queued run or requests cooperative running cancellation.", + transport: jobMutationTransport, + }, + { + access: jobSessionWriteAccess, + domain: "jobs", + errors: ["CONFLICT", "FORBIDDEN", "SERVICE_UNAVAILABLE", "UNAUTHORIZED"], + input: setJobClaimingPausedInputSchema, + inputSchemaId: "jobs.setClaimingPaused.input", + kind: "mutation", + name: "jobs.setClaimingPaused", + output: jobWorkerControlSchema, + outputSchemaId: "jobs.setClaimingPaused.output", + summary: "Pauses or resumes new cross-process claims under version control.", + transport: jobMutationTransport, + }, +] as const satisfies readonly ProcedureContract[]; + +export type CancelJobRunInput = v.InferOutput; +export type GetJobRunInput = v.InferOutput; +export type JobQueueSummary = v.InferOutput; +export type ListJobRunsInput = v.InferOutput; +export type ListJobRunsResult = v.InferOutput; +export type SetJobClaimingPausedInput = v.InferOutput< + typeof setJobClaimingPausedInputSchema +>; diff --git a/greenfield/src/contracts/scheduleTimeZones.ts b/greenfield/src/contracts/scheduleTimeZones.ts new file mode 100644 index 000000000..7d414975a --- /dev/null +++ b/greenfield/src/contracts/scheduleTimeZones.ts @@ -0,0 +1,453 @@ +/** + * Canonical schedule time zones accepted by both transport and durable SQLite state. + * + * This checked-in list deliberately avoids runtime ICU alias drift at trust boundaries. + * Additions require a reviewed contract and migration update. + */ +export const canonicalScheduleTimeZones: readonly string[] = Object.freeze([ + "Africa/Abidjan", + "Africa/Accra", + "Africa/Addis_Ababa", + "Africa/Algiers", + "Africa/Asmara", + "Africa/Bamako", + "Africa/Bangui", + "Africa/Banjul", + "Africa/Bissau", + "Africa/Blantyre", + "Africa/Brazzaville", + "Africa/Bujumbura", + "Africa/Cairo", + "Africa/Casablanca", + "Africa/Ceuta", + "Africa/Conakry", + "Africa/Dakar", + "Africa/Dar_es_Salaam", + "Africa/Djibouti", + "Africa/Douala", + "Africa/El_Aaiun", + "Africa/Freetown", + "Africa/Gaborone", + "Africa/Harare", + "Africa/Johannesburg", + "Africa/Juba", + "Africa/Kampala", + "Africa/Khartoum", + "Africa/Kigali", + "Africa/Kinshasa", + "Africa/Lagos", + "Africa/Libreville", + "Africa/Lome", + "Africa/Luanda", + "Africa/Lubumbashi", + "Africa/Lusaka", + "Africa/Malabo", + "Africa/Maputo", + "Africa/Maseru", + "Africa/Mbabane", + "Africa/Mogadishu", + "Africa/Monrovia", + "Africa/Nairobi", + "Africa/Ndjamena", + "Africa/Niamey", + "Africa/Nouakchott", + "Africa/Ouagadougou", + "Africa/Porto-Novo", + "Africa/Sao_Tome", + "Africa/Tripoli", + "Africa/Tunis", + "Africa/Windhoek", + "America/Adak", + "America/Anchorage", + "America/Anguilla", + "America/Antigua", + "America/Araguaina", + "America/Argentina/Buenos_Aires", + "America/Argentina/Catamarca", + "America/Argentina/Cordoba", + "America/Argentina/Jujuy", + "America/Argentina/La_Rioja", + "America/Argentina/Mendoza", + "America/Argentina/Rio_Gallegos", + "America/Argentina/Salta", + "America/Argentina/San_Juan", + "America/Argentina/San_Luis", + "America/Argentina/Tucuman", + "America/Argentina/Ushuaia", + "America/Aruba", + "America/Asuncion", + "America/Atikokan", + "America/Bahia", + "America/Bahia_Banderas", + "America/Barbados", + "America/Belem", + "America/Belize", + "America/Blanc-Sablon", + "America/Boa_Vista", + "America/Bogota", + "America/Boise", + "America/Cambridge_Bay", + "America/Campo_Grande", + "America/Cancun", + "America/Caracas", + "America/Cayenne", + "America/Cayman", + "America/Chicago", + "America/Chihuahua", + "America/Ciudad_Juarez", + "America/Costa_Rica", + "America/Creston", + "America/Cuiaba", + "America/Curacao", + "America/Danmarkshavn", + "America/Dawson", + "America/Dawson_Creek", + "America/Denver", + "America/Detroit", + "America/Dominica", + "America/Edmonton", + "America/Eirunepe", + "America/El_Salvador", + "America/Fort_Nelson", + "America/Fortaleza", + "America/Glace_Bay", + "America/Goose_Bay", + "America/Grand_Turk", + "America/Grenada", + "America/Guadeloupe", + "America/Guatemala", + "America/Guayaquil", + "America/Guyana", + "America/Halifax", + "America/Havana", + "America/Hermosillo", + "America/Indiana/Indianapolis", + "America/Indiana/Knox", + "America/Indiana/Marengo", + "America/Indiana/Petersburg", + "America/Indiana/Tell_City", + "America/Indiana/Vevay", + "America/Indiana/Vincennes", + "America/Indiana/Winamac", + "America/Inuvik", + "America/Iqaluit", + "America/Jamaica", + "America/Juneau", + "America/Kentucky/Louisville", + "America/Kentucky/Monticello", + "America/Kralendijk", + "America/La_Paz", + "America/Lima", + "America/Los_Angeles", + "America/Lower_Princes", + "America/Maceio", + "America/Managua", + "America/Manaus", + "America/Marigot", + "America/Martinique", + "America/Matamoros", + "America/Mazatlan", + "America/Menominee", + "America/Merida", + "America/Metlakatla", + "America/Mexico_City", + "America/Miquelon", + "America/Moncton", + "America/Monterrey", + "America/Montevideo", + "America/Montserrat", + "America/Nassau", + "America/New_York", + "America/Nome", + "America/Noronha", + "America/North_Dakota/Beulah", + "America/North_Dakota/Center", + "America/North_Dakota/New_Salem", + "America/Nuuk", + "America/Ojinaga", + "America/Panama", + "America/Paramaribo", + "America/Phoenix", + "America/Port-au-Prince", + "America/Port_of_Spain", + "America/Porto_Velho", + "America/Puerto_Rico", + "America/Punta_Arenas", + "America/Rankin_Inlet", + "America/Recife", + "America/Regina", + "America/Resolute", + "America/Rio_Branco", + "America/Santarem", + "America/Santiago", + "America/Santo_Domingo", + "America/Sao_Paulo", + "America/Scoresbysund", + "America/Sitka", + "America/St_Barthelemy", + "America/St_Johns", + "America/St_Kitts", + "America/St_Lucia", + "America/St_Thomas", + "America/St_Vincent", + "America/Swift_Current", + "America/Tegucigalpa", + "America/Thule", + "America/Tijuana", + "America/Toronto", + "America/Tortola", + "America/Vancouver", + "America/Whitehorse", + "America/Winnipeg", + "America/Yakutat", + "Antarctica/Casey", + "Antarctica/Davis", + "Antarctica/DumontDUrville", + "Antarctica/Macquarie", + "Antarctica/Mawson", + "Antarctica/McMurdo", + "Antarctica/Palmer", + "Antarctica/Rothera", + "Antarctica/Syowa", + "Antarctica/Troll", + "Antarctica/Vostok", + "Arctic/Longyearbyen", + "Asia/Aden", + "Asia/Almaty", + "Asia/Amman", + "Asia/Anadyr", + "Asia/Aqtau", + "Asia/Aqtobe", + "Asia/Ashgabat", + "Asia/Atyrau", + "Asia/Baghdad", + "Asia/Bahrain", + "Asia/Baku", + "Asia/Bangkok", + "Asia/Barnaul", + "Asia/Beirut", + "Asia/Bishkek", + "Asia/Brunei", + "Asia/Chita", + "Asia/Choibalsan", + "Asia/Colombo", + "Asia/Damascus", + "Asia/Dhaka", + "Asia/Dili", + "Asia/Dubai", + "Asia/Dushanbe", + "Asia/Famagusta", + "Asia/Gaza", + "Asia/Hebron", + "Asia/Ho_Chi_Minh", + "Asia/Hong_Kong", + "Asia/Hovd", + "Asia/Irkutsk", + "Asia/Jakarta", + "Asia/Jayapura", + "Asia/Jerusalem", + "Asia/Kabul", + "Asia/Kamchatka", + "Asia/Karachi", + "Asia/Kathmandu", + "Asia/Khandyga", + "Asia/Kolkata", + "Asia/Krasnoyarsk", + "Asia/Kuala_Lumpur", + "Asia/Kuching", + "Asia/Kuwait", + "Asia/Macau", + "Asia/Magadan", + "Asia/Makassar", + "Asia/Manila", + "Asia/Muscat", + "Asia/Nicosia", + "Asia/Novokuznetsk", + "Asia/Novosibirsk", + "Asia/Omsk", + "Asia/Oral", + "Asia/Phnom_Penh", + "Asia/Pontianak", + "Asia/Pyongyang", + "Asia/Qatar", + "Asia/Qostanay", + "Asia/Qyzylorda", + "Asia/Riyadh", + "Asia/Sakhalin", + "Asia/Samarkand", + "Asia/Seoul", + "Asia/Shanghai", + "Asia/Singapore", + "Asia/Srednekolymsk", + "Asia/Taipei", + "Asia/Tashkent", + "Asia/Tbilisi", + "Asia/Tehran", + "Asia/Thimphu", + "Asia/Tokyo", + "Asia/Tomsk", + "Asia/Ulaanbaatar", + "Asia/Urumqi", + "Asia/Ust-Nera", + "Asia/Vientiane", + "Asia/Vladivostok", + "Asia/Yakutsk", + "Asia/Yangon", + "Asia/Yekaterinburg", + "Asia/Yerevan", + "Atlantic/Azores", + "Atlantic/Bermuda", + "Atlantic/Canary", + "Atlantic/Cape_Verde", + "Atlantic/Faroe", + "Atlantic/Madeira", + "Atlantic/Reykjavik", + "Atlantic/South_Georgia", + "Atlantic/St_Helena", + "Atlantic/Stanley", + "Australia/Adelaide", + "Australia/Brisbane", + "Australia/Broken_Hill", + "Australia/Darwin", + "Australia/Eucla", + "Australia/Hobart", + "Australia/Lindeman", + "Australia/Lord_Howe", + "Australia/Melbourne", + "Australia/Perth", + "Australia/Sydney", + "Etc/GMT+1", + "Etc/GMT+10", + "Etc/GMT+11", + "Etc/GMT+12", + "Etc/GMT+2", + "Etc/GMT+3", + "Etc/GMT+4", + "Etc/GMT+5", + "Etc/GMT+6", + "Etc/GMT+7", + "Etc/GMT+8", + "Etc/GMT+9", + "Etc/GMT-1", + "Etc/GMT-10", + "Etc/GMT-11", + "Etc/GMT-12", + "Etc/GMT-13", + "Etc/GMT-14", + "Etc/GMT-2", + "Etc/GMT-3", + "Etc/GMT-4", + "Etc/GMT-5", + "Etc/GMT-6", + "Etc/GMT-7", + "Etc/GMT-8", + "Etc/GMT-9", + "Europe/Amsterdam", + "Europe/Andorra", + "Europe/Astrakhan", + "Europe/Athens", + "Europe/Belgrade", + "Europe/Berlin", + "Europe/Bratislava", + "Europe/Brussels", + "Europe/Bucharest", + "Europe/Budapest", + "Europe/Busingen", + "Europe/Chisinau", + "Europe/Copenhagen", + "Europe/Dublin", + "Europe/Gibraltar", + "Europe/Guernsey", + "Europe/Helsinki", + "Europe/Isle_of_Man", + "Europe/Istanbul", + "Europe/Jersey", + "Europe/Kaliningrad", + "Europe/Kirov", + "Europe/Kyiv", + "Europe/Lisbon", + "Europe/Ljubljana", + "Europe/London", + "Europe/Luxembourg", + "Europe/Madrid", + "Europe/Malta", + "Europe/Mariehamn", + "Europe/Minsk", + "Europe/Monaco", + "Europe/Moscow", + "Europe/Oslo", + "Europe/Paris", + "Europe/Podgorica", + "Europe/Prague", + "Europe/Riga", + "Europe/Rome", + "Europe/Samara", + "Europe/San_Marino", + "Europe/Sarajevo", + "Europe/Saratov", + "Europe/Simferopol", + "Europe/Skopje", + "Europe/Sofia", + "Europe/Stockholm", + "Europe/Tallinn", + "Europe/Tirane", + "Europe/Ulyanovsk", + "Europe/Vaduz", + "Europe/Vatican", + "Europe/Vienna", + "Europe/Vilnius", + "Europe/Volgograd", + "Europe/Warsaw", + "Europe/Zagreb", + "Europe/Zurich", + "Indian/Antananarivo", + "Indian/Chagos", + "Indian/Christmas", + "Indian/Cocos", + "Indian/Comoro", + "Indian/Kerguelen", + "Indian/Mahe", + "Indian/Maldives", + "Indian/Mauritius", + "Indian/Mayotte", + "Indian/Reunion", + "Pacific/Apia", + "Pacific/Auckland", + "Pacific/Bougainville", + "Pacific/Chatham", + "Pacific/Chuuk", + "Pacific/Easter", + "Pacific/Efate", + "Pacific/Fakaofo", + "Pacific/Fiji", + "Pacific/Funafuti", + "Pacific/Galapagos", + "Pacific/Gambier", + "Pacific/Guadalcanal", + "Pacific/Guam", + "Pacific/Honolulu", + "Pacific/Kanton", + "Pacific/Kiritimati", + "Pacific/Kosrae", + "Pacific/Kwajalein", + "Pacific/Majuro", + "Pacific/Marquesas", + "Pacific/Midway", + "Pacific/Nauru", + "Pacific/Niue", + "Pacific/Norfolk", + "Pacific/Noumea", + "Pacific/Pago_Pago", + "Pacific/Palau", + "Pacific/Pitcairn", + "Pacific/Pohnpei", + "Pacific/Port_Moresby", + "Pacific/Rarotonga", + "Pacific/Saipan", + "Pacific/Tahiti", + "Pacific/Tarawa", + "Pacific/Tongatapu", + "Pacific/Wake", + "Pacific/Wallis", + "UTC", +]); diff --git a/greenfield/src/contracts/schedules.test.ts b/greenfield/src/contracts/schedules.test.ts new file mode 100644 index 000000000..edcce7b76 --- /dev/null +++ b/greenfield/src/contracts/schedules.test.ts @@ -0,0 +1,288 @@ +import { describe, expect, test } from "bun:test"; + +import * as v from "valibot"; + +import { + listScheduleRunsResultSchema, + listSchedulesInputSchema, + listSchedulesResultSchema, + runScheduleInputSchema, + scheduleProcedureContracts, + updateScheduleInputSchema, +} from "./schedules.ts"; + +const firstRunId = "018f6f50-6a9e-7b88-8000-000000000002"; +const secondRunId = "018f6f50-6a9e-7b88-8000-000000000001"; + +function schedule(id: string, enabled: boolean) { + return { + actionKey: "system.worker-smoke", + attemptLimit: 3, + cancellationPolicy: "cooperative" as const, + createdAtMs: 500, + description: "Checks the worker without host mutation.", + enabled, + id, + name: "Worker smoke", + ...(enabled ? { nextRunAtMs: 60_000 } : {}), + priority: 0, + resourceClass: "light" as const, + resourceKeys: ["database"], + retrySafe: true, + schedule: { intervalMs: 60_000, kind: "interval" as const }, + timeoutMs: 30_000, + updatedAtMs: 1000, + version: 1, + }; +} + +function queuedRun(id: string, queuedAtMs: number) { + return { + actionKey: "system.worker-smoke", + attemptCount: 0, + attemptLimit: 3, + availableAtMs: queuedAtMs, + cancellationPolicy: "cooperative" as const, + displayName: "Worker smoke", + eventCount: 1, + id, + priority: 0, + queuedAtMs, + resourceClass: "light" as const, + resourceKeys: ["database"], + retrySafe: true, + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + state: "queued" as const, + stateVersion: 1, + timeoutMs: 30_000, + triggerType: "manual" as const, + updatedAtMs: queuedAtMs, + }; +} + +describe("schedule procedure contracts", () => { + test("locks reads, session writes, and dual-principal manual runs", () => { + expect( + scheduleProcedureContracts.map(({ access, kind, name, transport }) => ({ + access, + batching: transport.batching, + kind, + name, + })) + ).toEqual([ + { + access: { + capabilities: ["jobs:read"], + capabilityPolicy: "all", + kind: "authenticated", + }, + batching: "adapter-default", + kind: "query", + name: "schedules.list", + }, + { + access: { + capabilities: ["jobs:read"], + capabilityPolicy: "all", + kind: "authenticated", + }, + batching: "adapter-default", + kind: "query", + name: "schedules.get", + }, + { + access: { + capabilities: ["jobs:write"], + capabilityPolicy: "all", + kind: "authenticated", + principalKinds: ["session"], + }, + batching: "forbidden", + kind: "mutation", + name: "schedules.update", + }, + { + access: { + capabilities: ["jobs:write"], + capabilityPolicy: "all", + kind: "authenticated", + }, + batching: "forbidden", + kind: "mutation", + name: "schedules.run", + }, + { + access: { + capabilities: ["jobs:read"], + capabilityPolicy: "all", + kind: "authenticated", + }, + batching: "adapter-default", + kind: "query", + name: "schedules.listRuns", + }, + ]); + }); + + test("defaults filters and requires stable ascending schedule cursors", () => { + expect(v.parse(listSchedulesInputSchema, {})).toEqual({ + enabled: "all", + limit: 50, + }); + const schedules = [schedule("alpha", false), schedule("zeta", true)]; + expect( + v + .parse(listSchedulesResultSchema, { + nextCursor: { id: "zeta" }, + schedules, + }) + .schedules.map(({ id }) => id) + ).toEqual(["alpha", "zeta"]); + + expect( + v.safeParse(listSchedulesResultSchema, { + schedules: schedules.toReversed(), + }).success + ).toBeFalse(); + expect( + v.safeParse(listSchedulesResultSchema, { + nextCursor: { id: "alpha" }, + schedules, + }).success + ).toBeFalse(); + }); + + test("requires explicit disable intent transitions and canonical schedule variants", () => { + expect( + v.parse(updateScheduleInputSchema, { + expectedVersion: 3, + id: "system.worker-smoke", + patch: { + disableIntent: { + expiresAtMs: 10_000, + reason: "Maintenance", + }, + enabled: false, + schedule: { + expression: "0\t9 * JAN MON-FRI", + kind: "cron", + timeZone: "Europe/Oslo", + }, + }, + }).patch.schedule + ).toEqual({ + expression: "0 9 * 1 1-5", + kind: "cron", + timeZone: "Europe/Oslo", + }); + expect( + v.parse(updateScheduleInputSchema, { + expectedVersion: 3, + id: "system.worker-smoke", + patch: { disableIntent: null, enabled: true }, + }).patch.enabled + ).toBeTrue(); + + for (const patch of [ + {}, + { enabled: false }, + { disableIntent: null, enabled: false }, + { enabled: true }, + { disableIntent: { reason: "Maintenance" } }, + { + disableIntent: { reason: "Maintenance" }, + schedule: { kind: "interval", intervalMs: 60_000 }, + }, + ]) { + expect( + v.safeParse(updateScheduleInputSchema, { + expectedVersion: 3, + id: "system.worker-smoke", + patch, + }).success + ).toBeFalse(); + } + }); + + test("accepts a canonical caller idempotency key and rejects padded tokens", () => { + const idempotencyKey = "aB_9-".repeat(7).slice(0, 32); + expect( + v.parse(runScheduleInputSchema, { + id: "system.worker-smoke", + idempotencyKey, + }).idempotencyKey + ).toBe(idempotencyKey); + expect( + v.safeParse(runScheduleInputSchema, { + id: "system.worker-smoke", + idempotencyKey: `${idempotencyKey}=`, + }).success + ).toBeFalse(); + for (const nonCanonical of [ + "A".repeat(33), + `${"A".repeat(33)}B`, + `${"A".repeat(34)}B`, + ]) { + expect( + v.safeParse(runScheduleInputSchema, { + id: "system.worker-smoke", + idempotencyKey: nonCanonical, + }).success + ).toBeFalse(); + } + }); + + test("validates newest-first schedule run pages and exact cursors", () => { + const runs = [queuedRun(firstRunId, 2000), queuedRun(secondRunId, 1000)]; + expect( + v.parse(listScheduleRunsResultSchema, { + nextCursor: { id: secondRunId, queuedAtMs: 1000 }, + runs, + }).runs.length + ).toBe(2); + expect( + v.safeParse(listScheduleRunsResultSchema, { + nextCursor: { id: firstRunId, queuedAtMs: 2000 }, + runs, + }).success + ).toBeFalse(); + }); + + test("declares BAD_REQUEST only where time-dependent intent validation needs it", () => { + expect( + scheduleProcedureContracts.map(({ errors, name }) => ({ errors, name })) + ).toEqual([ + { errors: ["FORBIDDEN", "UNAUTHORIZED"], name: "schedules.list" }, + { + errors: ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + name: "schedules.get", + }, + { + errors: [ + "BAD_REQUEST", + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + name: "schedules.update", + }, + { + errors: [ + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + name: "schedules.run", + }, + { + errors: ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + name: "schedules.listRuns", + }, + ]); + }); +}); diff --git a/greenfield/src/contracts/schedules.ts b/greenfield/src/contracts/schedules.ts new file mode 100644 index 000000000..c75f78e52 --- /dev/null +++ b/greenfield/src/contracts/schedules.ts @@ -0,0 +1,280 @@ +import * as v from "valibot"; + +import { boundedControlSafeTextSchema } from "../shared/validation.ts"; +import { + type ScheduleSummary, + jobDescriptionMaximumLength, + jobIdempotencyKeySchema, + jobRunSummarySchema, + jobTimestampSchema, + jobVersionSchema, + scheduleConfigurationSchema, + scheduleIdSchema, + scheduleSummarySchema, +} from "./jobModel.ts"; +import { + jobMutationTransport, + jobQueryTransport, + jobReadAccess, + jobSessionWriteAccess, +} from "./jobProcedurePolicies.ts"; +import { + jobRunCursorSchema, + jobRunPageDefault, + jobRunPageMaximum, + jobRunPageSchema, +} from "./jobs.ts"; +import type { ProcedureContract } from "./registry.ts"; + +/** Default schedules returned by one inventory request. */ +export const schedulePageDefault = 50; +/** Hard schedule-row budget for one inventory response. */ +export const schedulePageMaximum = 100; + +const scheduleLimitSchema = v.pipe( + v.number("Schedule page limit is invalid"), + v.safeInteger("Schedule page limit is invalid"), + v.minValue(1, "Schedule page limit is invalid"), + v.maxValue(schedulePageMaximum, "Schedule page limit is outside its budget") +); +const scheduleRunLimitSchema = v.pipe( + v.number("Schedule run page limit is invalid"), + v.safeInteger("Schedule run page limit is invalid"), + v.minValue(1, "Schedule run page limit is invalid"), + v.maxValue(jobRunPageMaximum, "Schedule run page limit is outside its budget") +); + +/** Stable ascending cursor for the code-owned schedule directory. */ +export const scheduleCursorSchema = v.strictObject({ id: scheduleIdSchema }); + +/** One stable keyset-paginated schedule request. */ +export const listSchedulesInputSchema = v.strictObject({ + cursor: v.optional(scheduleCursorSchema), + enabled: v.optional( + v.picklist(["all", "disabled", "enabled"], "Schedule enabled filter is invalid"), + "all" + ), + limit: v.optional(scheduleLimitSchema, schedulePageDefault), +}); + +/** + * @param schedules Schedule summaries to inspect. + * @returns Whether they use strict ascending identifier order. + */ +export function scheduleOrderIsStable(schedules: ScheduleSummary[]): boolean { + return schedules.every((schedule, index) => { + const previous = schedules[index - 1]; + return previous === undefined || schedule.id > previous.id; + }); +} + +const schedulePageSchema = v.pipe( + v.array(scheduleSummarySchema, "Schedule page is invalid"), + v.maxLength(schedulePageMaximum, "Schedule page is outside its budget"), + v.check(scheduleOrderIsStable, "Schedule page order is invalid") +); + +const listSchedulesResultObjectSchema = v.strictObject({ + nextCursor: v.optional(scheduleCursorSchema), + schedules: schedulePageSchema, +}); + +type ListSchedulesResultValue = v.InferOutput; + +/** + * @param result Schedule page and cursor to inspect. + * @returns Whether an optional cursor identifies the final returned row. + */ +export function schedulePageCursorIsConsistent( + result: ListSchedulesResultValue +): boolean { + if (result.nextCursor === undefined) return true; + return result.schedules.at(-1)?.id === result.nextCursor.id; +} + +/** One bounded schedule page plus its exact continuation cursor. */ +export const listSchedulesResultSchema = v.pipe( + listSchedulesResultObjectSchema, + v.check(schedulePageCursorIsConsistent, "Schedule page cursor is inconsistent") +); + +/** Exact schedule lookup request. */ +export const getScheduleInputSchema = v.strictObject({ id: scheduleIdSchema }); + +const scheduleDisableIntentInputSchema = v.strictObject({ + expiresAtMs: v.optional(jobTimestampSchema), + reason: boundedControlSafeTextSchema( + jobDescriptionMaximumLength, + "Schedule disable reason is invalid" + ), +}); + +const updateSchedulePatchObjectSchema = v.strictObject({ + disableIntent: v.optional(v.nullable(scheduleDisableIntentInputSchema)), + enabled: v.optional(v.boolean("Schedule enabled state is invalid")), + schedule: v.optional(scheduleConfigurationSchema), +}); + +export type UpdateSchedulePatch = v.InferOutput; + +/** + * @param patch Schedule patch to inspect. + * @returns Whether it is non-empty and has one explicit disable transition. + */ +export function scheduleUpdatePatchIsConsistent(patch: UpdateSchedulePatch): boolean { + if (Object.values(patch).every((value) => value === undefined)) return false; + if (patch.enabled === false) { + return patch.disableIntent !== undefined && patch.disableIntent !== null; + } + if (patch.enabled === true) return patch.disableIntent === null; + return patch.disableIntent === undefined; +} + +/** Versioned operator update with a complete mutually exclusive schedule variant. */ +export const updateScheduleInputSchema = v.strictObject({ + expectedVersion: jobVersionSchema, + id: scheduleIdSchema, + patch: v.pipe( + updateSchedulePatchObjectSchema, + v.check(scheduleUpdatePatchIsConsistent, "Schedule update patch is inconsistent") + ), +}); + +/** Lost-response-safe manual schedule run request. */ +export const runScheduleInputSchema = v.strictObject({ + id: scheduleIdSchema, + idempotencyKey: jobIdempotencyKeySchema, +}); + +/** One stable newest-first schedule run-history request. */ +export const listScheduleRunsInputSchema = v.strictObject({ + cursor: v.optional(jobRunCursorSchema), + id: scheduleIdSchema, + limit: v.optional(scheduleRunLimitSchema, jobRunPageDefault), +}); + +const listScheduleRunsResultObjectSchema = v.strictObject({ + nextCursor: v.optional(jobRunCursorSchema), + runs: jobRunPageSchema, +}); + +type ListScheduleRunsResultValue = v.InferOutput< + typeof listScheduleRunsResultObjectSchema +>; + +/** + * @param result Schedule run page and cursor to inspect. + * @returns Whether an optional cursor identifies the final row. + */ +export function scheduleRunPageCursorIsConsistent( + result: ListScheduleRunsResultValue +): boolean { + if (result.nextCursor === undefined) return true; + const last = result.runs.at(-1); + return ( + last !== undefined && + last.id === result.nextCursor.id && + last.queuedAtMs === result.nextCursor.queuedAtMs + ); +} + +/** One bounded schedule-scoped run page plus its exact continuation cursor. */ +export const listScheduleRunsResultSchema = v.pipe( + listScheduleRunsResultObjectSchema, + v.check(scheduleRunPageCursorIsConsistent, "Schedule run page cursor is inconsistent") +); + +const scheduleRunAccess = { + capabilities: ["jobs:write"], + capabilityPolicy: "all", + kind: "authenticated", +} as const; +/** Dashboard-local schedule inventory, update, and manual-run contracts. */ +export const scheduleProcedureContracts = [ + { + access: jobReadAccess, + domain: "schedules", + errors: ["FORBIDDEN", "UNAUTHORIZED"], + input: listSchedulesInputSchema, + inputSchemaId: "schedules.list.input", + kind: "query", + name: "schedules.list", + output: listSchedulesResultSchema, + outputSchemaId: "schedules.list.output", + summary: "Lists the stable code-owned Dashboard schedule directory.", + transport: jobQueryTransport, + }, + { + access: jobReadAccess, + domain: "schedules", + errors: ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + input: getScheduleInputSchema, + inputSchemaId: "schedules.get.input", + kind: "query", + name: "schedules.get", + output: scheduleSummarySchema, + outputSchemaId: "schedules.get.output", + summary: "Loads one code-owned schedule and its latest durable run state.", + transport: jobQueryTransport, + }, + { + access: jobSessionWriteAccess, + domain: "schedules", + errors: [ + "BAD_REQUEST", + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + input: updateScheduleInputSchema, + inputSchemaId: "schedules.update.input", + kind: "mutation", + name: "schedules.update", + output: scheduleSummarySchema, + outputSchemaId: "schedules.update.output", + summary: "Updates one schedule or its explicit disable intent by version.", + transport: jobMutationTransport, + }, + { + access: scheduleRunAccess, + domain: "schedules", + errors: [ + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + input: runScheduleInputSchema, + inputSchemaId: "schedules.run.input", + kind: "mutation", + name: "schedules.run", + output: jobRunSummarySchema, + outputSchemaId: "schedules.run.output", + summary: "Enqueues one caller-scoped idempotent manual schedule run.", + transport: jobMutationTransport, + }, + { + access: jobReadAccess, + domain: "schedules", + errors: ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + input: listScheduleRunsInputSchema, + inputSchemaId: "schedules.listRuns.input", + kind: "query", + name: "schedules.listRuns", + output: listScheduleRunsResultSchema, + outputSchemaId: "schedules.listRuns.output", + summary: "Lists stable newest-first durable history for one schedule.", + transport: jobQueryTransport, + }, +] as const satisfies readonly ProcedureContract[]; + +export type GetScheduleInput = v.InferOutput; +export type ListScheduleRunsInput = v.InferOutput; +export type ListScheduleRunsResult = v.InferOutput; +export type ListSchedulesInput = v.InferOutput; +export type ListSchedulesResult = v.InferOutput; +export type RunScheduleInput = v.InferOutput; +export type UpdateScheduleInput = v.InferOutput; diff --git a/greenfield/src/contracts/security.test.ts b/greenfield/src/contracts/security.test.ts index 6bda2545d..52538b99d 100644 --- a/greenfield/src/contracts/security.test.ts +++ b/greenfield/src/contracts/security.test.ts @@ -3,6 +3,8 @@ import { describe, expect, test } from "bun:test"; import * as v from "valibot"; import { + applicationCapabilities, + applicationCapabilityListSchema, authenticationMethods, multiFactorAuthenticationMethods, requestAuthenticationSchema, @@ -12,6 +14,25 @@ const userId = "019fc968-1a9b-7770-8f1b-d5b863b0e7b4"; const sessionSelector = "a".repeat(32); describe("request authentication contract", () => { + test("includes canonical least-privilege job capabilities", () => { + expect(applicationCapabilities).toEqual([ + "agents:read", + "agents:write", + "jobs:read", + "jobs:write", + "monitoring:write", + "notifications:read", + "notifications:write", + "reports:read", + "reports:write", + "tasks:read", + "tasks:write", + ]); + expect( + v.parse(applicationCapabilityListSchema, ["jobs:write", "jobs:read"]) + ).toEqual(["jobs:read", "jobs:write"]); + }); + test("advertises only authentication methods implemented by this slice", () => { expect(authenticationMethods).toEqual([ "password", diff --git a/greenfield/src/contracts/security.ts b/greenfield/src/contracts/security.ts index 6ff84578a..8575dc33a 100644 --- a/greenfield/src/contracts/security.ts +++ b/greenfield/src/contracts/security.ts @@ -94,6 +94,8 @@ export const securityRecordIdSchema = lowercaseUuidV7Schema( export const applicationCapabilities = [ "agents:read", "agents:write", + "jobs:read", + "jobs:write", "monitoring:write", "notifications:read", "notifications:write", diff --git a/greenfield/src/server/database/migrations/jobsSchema.test.ts b/greenfield/src/server/database/migrations/jobsSchema.test.ts new file mode 100644 index 000000000..61988136e --- /dev/null +++ b/greenfield/src/server/database/migrations/jobsSchema.test.ts @@ -0,0 +1,2132 @@ +import { describe, expect, test } from "bun:test"; + +import * as v from "valibot"; + +import { scheduleCronExpressionSchema } from "../../../contracts/jobModel.ts"; +import { canonicalScheduleTimeZones } from "../../../contracts/scheduleTimeZones.ts"; +import { openFreshMigratedDatabase } from "../../test/support/freshDatabase.ts"; + +type TestDatabase = Awaited>; + +type CancellationPolicy = "cooperative" | "never" | "queued-only"; +type ScheduleKind = "cron" | "daily" | "interval"; +type TriggerType = "manual" | "schedule" | "startup" | "system"; + +interface ScheduleFixture { + actionKey: string; + actionPayloadJson: string; + attemptLimit: number; + cancellationPolicy: CancellationPolicy; + createdAt: number; + cronExpression: string | null; + description: string; + enabled: number; + id: string; + intervalMs: number | null; + name: string; + nextRunAt: number | null; + priority: number; + resourceClass: string; + resourceKeysJson: string; + retrySafe: number; + scheduleKind: ScheduleKind; + timeOfDay: string | null; + timeZone: string | null; + timeoutMs: number; + updatedAt: number; + version: number; +} + +interface QueuedRunFixture { + attemptLimit: number; + cancellationPolicy: CancellationPolicy; + id: string; + idempotencyKey: string; + requestedById: string; + requestedByKind: "automation" | "system" | "user"; + resourceKeysJson: string; + retrySafe: number; + scheduledForAt: number | null; + scheduledJobId: string | null; + scheduledJobVersion: number | null; + triggerType: TriggerType; +} + +interface EventFixture { + attempt: number; + jobRunId: string; + kind: string; + message: string | null; + occurredAt: number; + progressJson: string | null; + sequence: number; + workerInstanceId: string | null; +} + +interface QueryPlanRow { + detail: string; +} + +const userId = "019fdf00-0000-7000-8000-000000000001"; +const releaseId = "a".repeat(40); +const enqueueSha256 = "b".repeat(64); + +function uuid(index: number): string { + return `019fdf00-0000-7000-8000-${String(index).padStart(12, "0")}`; +} + +function idempotencyKey(index: number): string { + return index.toString(16).padStart(32, "0"); +} + +function insertSchedule( + database: TestDatabase, + overrides: Partial = {} +): void { + const fixture: ScheduleFixture = { + actionKey: "system.worker-smoke", + actionPayloadJson: "{}", + attemptLimit: 3, + cancellationPolicy: "cooperative", + createdAt: 1000, + cronExpression: null, + description: "Safe worker smoke check", + enabled: 1, + id: "system.worker-smoke", + intervalMs: 60_000, + name: "Worker smoke", + nextRunAt: 61_000, + priority: 0, + resourceClass: "light", + resourceKeysJson: '["host.smoke"]', + retrySafe: 1, + scheduleKind: "interval", + timeOfDay: null, + timeZone: null, + timeoutMs: 10_000, + updatedAt: 1000, + version: 1, + ...overrides, + }; + + database.sqlite.run( + `INSERT INTO scheduled_jobs ( + action_key, action_payload_json, attempt_limit, cancellation_policy, + created_at, cron_expression, description, enabled, id, interval_ms, + name, next_run_at, priority, resource_class, resource_keys_json, + retry_safe, schedule_kind, time_of_day, time_zone, timeout_ms, + updated_at, version + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + [ + fixture.actionKey, + fixture.actionPayloadJson, + fixture.attemptLimit, + fixture.cancellationPolicy, + fixture.createdAt, + fixture.cronExpression, + fixture.description, + fixture.enabled, + fixture.id, + fixture.intervalMs, + fixture.name, + fixture.nextRunAt, + fixture.priority, + fixture.resourceClass, + fixture.resourceKeysJson, + fixture.retrySafe, + fixture.scheduleKind, + fixture.timeOfDay, + fixture.timeZone, + fixture.timeoutMs, + fixture.updatedAt, + fixture.version, + ] + ); +} + +function insertScheduleDisableIntent( + database: TestDatabase, + id: string, + createdAt: number +): void { + database.sqlite.run( + `INSERT INTO job_disable_intents ( + created_at, created_by_id, created_by_kind, ended_at, ended_by_id, + ended_by_kind, ended_reason, expires_at, external_job_id, + external_provider, id, reason, scheduled_job_id, target_kind + ) VALUES (?, ?, 'user', NULL, NULL, NULL, NULL, NULL, NULL, NULL, ?, ?, ?, + 'dashboard-schedule')`, + [createdAt, userId, id, "Operator maintenance", "system.worker-smoke"] + ); +} + +function insertQueuedRun( + database: TestDatabase, + overrides: Partial & Pick +): void { + const fixture: QueuedRunFixture = { + attemptLimit: 3, + cancellationPolicy: "cooperative", + requestedById: "job-scheduler", + requestedByKind: "system", + resourceKeysJson: "[]", + retrySafe: 1, + scheduledForAt: null, + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + ...overrides, + }; + + database.sqlite.run( + `INSERT INTO job_runs ( + action_key, attempt_limit, available_at, cancellation_policy, + display_name, enqueue_sha256, id, idempotency_key, payload_json, + priority, queued_at, requested_by_id, requested_by_kind, + resource_class, resource_keys_json, retry_safe, scheduled_for_at, + scheduled_job_id, scheduled_job_version, state, timeout_ms, + trigger_type, updated_at + ) VALUES ( + 'system.worker-smoke', ?, 1000, ?, 'Worker smoke', ?, ?, ?, '{}', + 0, 1000, ?, ?, 'light', ?, ?, ?, ?, ?, 'queued', 10000, ?, 1000 + )`, + [ + fixture.attemptLimit, + fixture.cancellationPolicy, + enqueueSha256, + fixture.id, + fixture.idempotencyKey, + fixture.requestedById, + fixture.requestedByKind, + fixture.resourceKeysJson, + fixture.retrySafe, + fixture.scheduledForAt, + fixture.scheduledJobId, + fixture.scheduledJobVersion, + fixture.triggerType, + ] + ); +} + +function insertWorker(database: TestDatabase, id: string, pid = 1000): void { + database.sqlite.run( + `INSERT INTO worker_instances ( + capacity, heartbeat_at, id, pid, release_id, started_at, state + ) VALUES (4, 1000, ?, ?, ?, 1000, 'online')`, + [id, pid, releaseId] + ); +} + +function claimRun( + database: TestDatabase, + runId: string, + workerId: string, + leaseToken: string, + startedAt = 2000, + leaseExpiresAt = 5000 +): void { + database.sqlite.run( + `UPDATE job_runs + SET attempt_count = attempt_count + 1, + first_started_at = COALESCE(first_started_at, ?), + heartbeat_at = ?, + last_attempt_started_at = ?, + lease_expires_at = ?, + lease_owner_id = ?, + lease_token = ?, + state = 'running', + state_version = state_version + 1, + updated_at = ? + WHERE id = ?`, + [ + startedAt, + startedAt, + startedAt, + leaseExpiresAt, + workerId, + leaseToken, + startedAt, + runId, + ] + ); +} + +function insertEvent( + database: TestDatabase, + overrides: Partial & + Pick +): void { + const fixture: EventFixture = { + attempt: 0, + message: null, + occurredAt: 1000, + progressJson: null, + workerInstanceId: null, + ...overrides, + }; + database.sqlite.run( + `INSERT INTO job_run_events ( + attempt, job_run_id, kind, message, occurred_at, progress_json, + sequence, worker_instance_id + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + [ + fixture.attempt, + fixture.jobRunId, + fixture.kind, + fixture.message, + fixture.occurredAt, + fixture.progressJson, + fixture.sequence, + fixture.workerInstanceId, + ] + ); +} + +function expectUsesIndexWithoutTemporarySort( + database: TestDatabase, + query: string, + indexName: string, + parameter?: string, + requiredDetail?: string +): void { + const statement = `EXPLAIN QUERY PLAN ${query}`; + const plan = + parameter === undefined + ? database.sqlite.query(statement).all() + : database.sqlite.query(statement).all(parameter); + expect(plan.some(({ detail }) => detail.includes(indexName))).toBeTrue(); + expect(plan.some(({ detail }) => detail.includes("USE TEMP B-TREE"))).toBeFalse(); + if (requiredDetail !== undefined) { + expect(plan.some(({ detail }) => detail.includes(requiredDetail))).toBeTrue(); + } +} + +describe("jobs baseline schema", () => { + test("creates strict hardened tables and seeds required worker control", async () => { + const database = await openFreshMigratedDatabase(); + + try { + expect( + database.sqlite + .query<{ name: string; strict: number; wr: number }, []>( + ` + SELECT name, strict, wr + FROM pragma_table_list + WHERE name IN ( + 'job_disable_intents', 'job_run_events', 'job_runs', + 'job_worker_control', 'resource_leases', + 'scheduled_jobs', 'worker_instances' + ) + ORDER BY name + ` + ) + .all() + ).toEqual([ + { name: "job_disable_intents", strict: 1, wr: 1 }, + { name: "job_run_events", strict: 1, wr: 1 }, + { name: "job_runs", strict: 1, wr: 1 }, + { name: "job_worker_control", strict: 1, wr: 0 }, + { name: "resource_leases", strict: 1, wr: 1 }, + { name: "scheduled_jobs", strict: 1, wr: 1 }, + { name: "worker_instances", strict: 1, wr: 1 }, + ]); + expect( + database.sqlite + .query< + { + claiming_paused: number; + id: number; + updated_at: number; + updated_by_id: string | null; + updated_by_kind: string | null; + version: number; + }, + [] + >("SELECT * FROM job_worker_control") + .get() + ).toEqual({ + claiming_paused: 0, + id: 1, + updated_at: 0, + updated_by_id: null, + updated_by_kind: null, + version: 1, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("rejects malformed schedule, resource, provenance, and idempotency shapes", async () => { + const database = await openFreshMigratedDatabase(); + + try { + expect(() => + insertSchedule(database, { + id: "invalid.daily", + intervalMs: null, + nextRunAt: 70_000, + scheduleKind: "daily", + timeOfDay: "aa:bb", + timeZone: "UTC", + }) + ).toThrow("scheduled_jobs_schedule_shape_check"); + expect(() => + insertSchedule(database, { + id: "invalid.daily-nul-suffix", + intervalMs: null, + scheduleKind: "daily", + timeOfDay: "09:00\0hidden", + timeZone: "UTC", + }) + ).toThrow("scheduled_jobs_schedule_shape_check"); + for (const [index, cronExpression] of [ + "foo foo foo foo foo", + "0 9 * JAN MON-FRI", + "0\t9 * * *", + "0 9 * * *\0hidden", + ].entries()) { + expect(() => + insertSchedule(database, { + cronExpression, + id: `invalid.cron.${index}`, + intervalMs: null, + scheduleKind: "cron", + timeZone: "UTC", + }) + ).toThrow("scheduled_jobs_schedule_shape_check"); + } + for (const [index, cronExpression] of [ + "99 99 99 99 99", + "*/0 * * * *", + "60 * * * *", + "* 24 * * *", + "* * 0 * *", + "* * 32 * *", + "* * * 0 *", + "* * * 13 *", + "* * * * 8", + "*/60 * * * *", + "* */24 * * *", + "* * */32 * *", + "* * * */13 *", + "* * * * */8", + "1-0 * * * *", + "1//2 * * * *", + "1-2-3 * * * *", + "1,,2 * * * *", + "0 0 30 2 *", + "0 0 31 4 *", + "0 0 31/1 2 */2", + "0 0 30 2 */01", + "0 0 31 2 *,1", + "0 0 31 2-2/2 *", + "0 0 31 2/12 *", + ].entries()) { + expect( + v.safeParse(scheduleCronExpressionSchema, cronExpression).success + ).toBeFalse(); + expect(() => + insertSchedule(database, { + cronExpression, + id: `invalid.cron-semantic.${index}`, + intervalMs: null, + scheduleKind: "cron", + timeZone: "UTC", + }) + ).toThrow("scheduled_jobs cron expression must be semantically valid"); + } + const canonicalCronExpressions = [ + "* * * * *", + "0 9 * 1 1-5", + "*/15 0-23/2 1,15 1-12/3 0-7", + "00 09 01 01 07", + "1/59 1/23 1/31 1/12 0/7", + "*,5 *,6 *,7 *,8 *,7", + "0-59/59 0-23/23 1-31/31 1-12/12 0-7/7", + "0 0 29 2 *", + "0 0 31 2 1", + "0 0 */31 2 1", + "0 0 28,31 2 *", + "0 0 31 2/2 *", + ] as const; + const fieldSamples = [ + { + baseline: ["0", "0", "1", "1", "*"], + fieldIndex: 0, + tokens: [ + "*", + "*/1", + "*/59", + "0", + "59", + "01", + "1/59", + "0-59", + "1-59/2", + "0,15,30,45", + "*,5", + ], + }, + { + baseline: ["0", "0", "1", "1", "*"], + fieldIndex: 1, + tokens: [ + "*", + "*/1", + "*/23", + "0", + "23", + "01", + "1/23", + "0-23", + "1-23/2", + "0,6,12,18", + "*,5", + ], + }, + { + baseline: ["0", "0", "1", "*", "*"], + fieldIndex: 2, + tokens: [ + "*", + "*/1", + "*/31", + "1", + "31", + "01", + "1/31", + "1-31", + "2-30/2", + "1,15,31", + "*,7", + ], + }, + { + baseline: ["0", "0", "1", "1", "*"], + fieldIndex: 3, + tokens: [ + "*", + "*/1", + "*/12", + "1", + "12", + "01", + "1/12", + "1-12", + "2-12/2", + "1,6,12", + "*,7", + ], + }, + { + baseline: ["0", "0", "1", "1", "0"], + fieldIndex: 4, + tokens: [ + "*", + "*/1", + "*/7", + "0", + "7", + "01", + "0/7", + "0-7", + "1-7/2", + "0,1,6,7", + "*,5", + ], + }, + ] as const; + const generatedContractValidExpressions = fieldSamples.flatMap( + ({ baseline, fieldIndex, tokens }) => + tokens.map((token) => { + const fields: string[] = [...baseline]; + fields[fieldIndex] = token; + return fields.join(" "); + }) + ); + const contractValidExpressions = [ + ...new Set([ + ...canonicalCronExpressions, + ...generatedContractValidExpressions, + ]), + ]; + for (const [index, cronExpression] of contractValidExpressions.entries()) { + expect(v.parse(scheduleCronExpressionSchema, cronExpression)).toBe( + cronExpression + ); + insertSchedule(database, { + cronExpression, + id: `valid.canonical-cron.${index}`, + intervalMs: null, + scheduleKind: "cron", + timeZone: "UTC", + }); + } + for (const cronExpression of ["*/0 * * * *", "0 0 30 2 *"]) { + expect(() => + database.sqlite.run( + `UPDATE scheduled_jobs + SET cron_expression = ?, updated_at = 2000, version = 2 + WHERE id = 'valid.canonical-cron.0'`, + [cronExpression] + ) + ).toThrow("scheduled_jobs cron expression must be semantically valid"); + } + expect( + database.sqlite + .query<{ cron_expression: string; version: number }, []>( + `SELECT cron_expression, version + FROM scheduled_jobs + WHERE id = 'valid.canonical-cron.0'` + ) + .get() + ).toEqual({ cron_expression: "* * * * *", version: 1 }); + for (const [index, nextRunAt] of [-1, 8_640_000_000_000_001].entries()) { + expect(() => + insertSchedule(database, { + enabled: 0, + id: `invalid.dormant-cursor.${index}`, + nextRunAt, + }) + ).toThrow("scheduled_jobs_next_run_check"); + } + insertSchedule(database, { + enabled: 0, + id: "valid.null-dormant-cursor", + nextRunAt: null, + }); + expect(() => + insertSchedule(database, { + id: "invalid.enabled-null-cursor", + nextRunAt: null, + }) + ).toThrow("scheduled_jobs_next_run_check"); + for (const [index, timeZone] of canonicalScheduleTimeZones.entries()) { + insertSchedule(database, { + id: `valid.time-zone.${index}`, + intervalMs: null, + scheduleKind: "daily", + timeOfDay: "09:00", + timeZone, + }); + } + for (const [index, timeZone] of [ + "US/Eastern", + "GMT", + "+01:00", + "local", + ].entries()) { + expect(() => + insertSchedule(database, { + id: `invalid.time-zone.${index}`, + intervalMs: null, + scheduleKind: "daily", + timeOfDay: "09:00", + timeZone, + }) + ).toThrow("scheduled_jobs_time_zone_check"); + } + expect(() => + insertSchedule(database, { + id: "invalid.resources", + resourceKeysJson: '["host.smoke","host.smoke"]', + }) + ).toThrow("scheduled_jobs resource keys must be canonical"); + + insertSchedule(database); + expect(() => + insertQueuedRun(database, { + id: uuid(10), + idempotencyKey: idempotencyKey(10), + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + triggerType: "system", + }) + ).toThrow("job_runs_schedule_check"); + expect(() => + insertQueuedRun(database, { + id: uuid(11), + idempotencyKey: "A".repeat(33), + }) + ).toThrow("job_runs_idempotency_key_check"); + expect(() => + insertQueuedRun(database, { + id: uuid(12), + idempotencyKey: idempotencyKey(12), + resourceKeysJson: '["UPPER",7,"duplicate","duplicate"]', + }) + ).toThrow("job_runs resource keys must be canonical"); + } finally { + database.sqlite.close(true); + } + }); + + test("separates scheduler cursor movement from versioned schedule configuration", async () => { + const database = await openFreshMigratedDatabase(); + + try { + insertSchedule(database); + database.sqlite.run( + "UPDATE scheduled_jobs SET next_run_at = 121000 WHERE id = ?", + ["system.worker-smoke"] + ); + expect( + database.sqlite + .query< + { next_run_at: number; updated_at: number; version: number }, + [string] + >( + "SELECT next_run_at, updated_at, version FROM scheduled_jobs WHERE id = ?" + ) + .get("system.worker-smoke") + ).toEqual({ next_run_at: 121_000, updated_at: 1000, version: 1 }); + + expect(() => + database.sqlite.run( + "UPDATE scheduled_jobs SET next_run_at = 181000, version = 2 WHERE id = ?", + ["system.worker-smoke"] + ) + ).toThrow("scheduled_jobs version transition is invalid"); + expect(() => + database.sqlite.run( + "UPDATE scheduled_jobs SET name = 'Changed' WHERE id = ?", + ["system.worker-smoke"] + ) + ).toThrow("scheduled_jobs version transition is invalid"); + + database.sqlite.run( + `UPDATE scheduled_jobs + SET name = 'Changed', updated_at = 2000, version = 2 + WHERE id = ?`, + ["system.worker-smoke"] + ); + expect( + database.sqlite + .query< + { name: string; updated_at: number; version: number }, + [string] + >("SELECT name, updated_at, version FROM scheduled_jobs WHERE id = ?") + .get("system.worker-smoke") + ).toEqual({ name: "Changed", updated_at: 2000, version: 2 }); + + database.sqlite.run( + `UPDATE scheduled_jobs + SET enabled = 0, updated_at = 3000, version = 3 + WHERE id = ?`, + ["system.worker-smoke"] + ); + expect( + database.sqlite + .query< + { + enabled: number; + next_run_at: number; + updated_at: number; + version: number; + }, + [string] + >( + "SELECT enabled, next_run_at, updated_at, version FROM scheduled_jobs WHERE id = ?" + ) + .get("system.worker-smoke") + ).toEqual({ + enabled: 0, + next_run_at: 121_000, + updated_at: 3000, + version: 3, + }); + + expect(() => + database.sqlite.run( + `UPDATE scheduled_jobs + SET updated_at = 4000, version = 4 + WHERE id = ?`, + ["system.worker-smoke"] + ) + ).toThrow("scheduled_jobs version transition is invalid"); + + const originalIntentId = uuid(13); + insertScheduleDisableIntent(database, originalIntentId, 3000); + expect(() => + database.sqlite.run( + `UPDATE scheduled_jobs + SET updated_at = 4000, version = 4 + WHERE id = ?`, + ["system.worker-smoke"] + ) + ).toThrow("scheduled_jobs version transition is invalid"); + + database.sqlite.run( + `UPDATE job_disable_intents + SET ended_at = 4000, ended_by_id = ?, ended_by_kind = 'user', + ended_reason = 'replaced' + WHERE id = ?`, + [userId, originalIntentId] + ); + insertScheduleDisableIntent(database, uuid(14), 4000); + database.sqlite.run( + `UPDATE scheduled_jobs + SET updated_at = 4000, version = 4 + WHERE id = ?`, + ["system.worker-smoke"] + ); + expect( + database.sqlite + .query< + { next_run_at: number; updated_at: number; version: number }, + [string] + >( + "SELECT next_run_at, updated_at, version FROM scheduled_jobs WHERE id = ?" + ) + .get("system.worker-smoke") + ).toEqual({ next_run_at: 121_000, updated_at: 4000, version: 4 }); + + expect(() => + database.sqlite.run( + `UPDATE scheduled_jobs + SET updated_at = 5000, version = 5 + WHERE id = ?`, + ["system.worker-smoke"] + ) + ).toThrow("scheduled_jobs version transition is invalid"); + expect(() => + database.sqlite.run( + `UPDATE scheduled_jobs + SET next_run_at = 181000, version = 5 + WHERE id = ?`, + ["system.worker-smoke"] + ) + ).toThrow("scheduled_jobs version transition is invalid"); + } finally { + database.sqlite.close(true); + } + }); + + test("enforces immutable run snapshots and legal retry and cancellation lifecycles", async () => { + const database = await openFreshMigratedDatabase(); + const workerId = uuid(20); + + try { + insertWorker(database, workerId); + + const cooperativeRunId = uuid(21); + insertQueuedRun(database, { + id: cooperativeRunId, + idempotencyKey: idempotencyKey(21), + }); + insertEvent(database, { + jobRunId: cooperativeRunId, + kind: "queued", + sequence: 1, + }); + expect(() => + database.sqlite.run( + "UPDATE job_runs SET action_key = 'system.changed' WHERE id = ?", + [cooperativeRunId] + ) + ).toThrow("job_runs execution snapshot is immutable"); + + claimRun(database, cooperativeRunId, workerId, uuid(22)); + insertEvent(database, { + attempt: 1, + jobRunId: cooperativeRunId, + kind: "claimed", + occurredAt: 2000, + sequence: 2, + workerInstanceId: workerId, + }); + database.sqlite.run( + `UPDATE job_runs + SET cancel_requested_at = 2500, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + state_version = state_version + 1, + updated_at = 2500 + WHERE id = ?`, + [userId, cooperativeRunId] + ); + insertEvent(database, { + attempt: 1, + jobRunId: cooperativeRunId, + kind: "cancel-requested", + occurredAt: 2500, + sequence: 3, + }); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET available_at = 3000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'queued', + state_version = state_version + 1, + updated_at = 2600 + WHERE id = ?`, + [cooperativeRunId] + ) + ).toThrow("job_runs lifecycle transition is invalid"); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 3000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'cancelled', + state_version = state_version + 1, + terminal_code = 'job/cancelled', + terminal_message = 'Cancelled cooperatively', + updated_at = 3000 + WHERE id = ?`, + [cooperativeRunId] + ); + insertEvent(database, { + attempt: 1, + jobRunId: cooperativeRunId, + kind: "cancelled", + occurredAt: 3000, + sequence: 4, + }); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET available_at = 4000, finished_at = NULL, state = 'queued', + state_version = state_version + 1, terminal_code = NULL, + terminal_message = NULL, updated_at = 4000 + WHERE id = ?`, + [cooperativeRunId] + ) + ).toThrow("job_runs lifecycle transition is invalid"); + + const retryRunId = uuid(23); + insertQueuedRun(database, { + id: retryRunId, + idempotencyKey: idempotencyKey(23), + }); + insertEvent(database, { jobRunId: retryRunId, kind: "queued", sequence: 1 }); + claimRun(database, retryRunId, workerId, uuid(24)); + insertEvent(database, { + attempt: 1, + jobRunId: retryRunId, + kind: "claimed", + occurredAt: 2000, + sequence: 2, + workerInstanceId: workerId, + }); + database.sqlite.run( + `UPDATE job_runs + SET available_at = 3000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'queued', + state_version = state_version + 1, + updated_at = 2500 + WHERE id = ?`, + [retryRunId] + ); + insertEvent(database, { + attempt: 1, + jobRunId: retryRunId, + kind: "retry-scheduled", + occurredAt: 2500, + sequence: 3, + }); + claimRun(database, retryRunId, workerId, uuid(25), 3500, 6000); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 4000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'failed', + state_version = state_version + 1, + terminal_code = 'failed/worker-lease-expired', + terminal_message = 'Worker lease expired', + updated_at = 4000 + WHERE id = ?`, + [retryRunId] + ); + expect( + database.sqlite + .query< + { attempt_count: number; state: string; terminal_code: string }, + [string] + >( + "SELECT attempt_count, state, terminal_code FROM job_runs WHERE id = ?" + ) + .get(retryRunId) + ).toEqual({ + attempt_count: 2, + state: "failed", + terminal_code: "failed/worker-lease-expired", + }); + + const neverRunId = uuid(26); + insertQueuedRun(database, { + cancellationPolicy: "never", + id: neverRunId, + idempotencyKey: idempotencyKey(26), + }); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 1500, state = 'cancelled', state_version = 2, + terminal_code = 'job/cancelled', terminal_message = 'Cancelled', + updated_at = 1500 + WHERE id = ?`, + [neverRunId] + ) + ).toThrow("job_runs lifecycle transition is invalid"); + + const queuedOnlyRunId = uuid(27); + insertQueuedRun(database, { + cancellationPolicy: "queued-only", + id: queuedOnlyRunId, + idempotencyKey: idempotencyKey(27), + }); + claimRun(database, queuedOnlyRunId, workerId, uuid(28)); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET cancel_requested_at = 2500, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + state_version = state_version + 1, + updated_at = 2500 + WHERE id = ?`, + [userId, queuedOnlyRunId] + ) + ).toThrow("job_runs lifecycle transition is invalid"); + } finally { + database.sqlite.close(true); + } + }); + + test("enforces durable event attempts for every lifecycle kind", async () => { + const database = await openFreshMigratedDatabase(); + const workerId = uuid(70); + + const prepareStartedRun = (runIndex: number, leaseIndex: number): string => { + const runId = uuid(runIndex); + insertQueuedRun(database, { + id: runId, + idempotencyKey: idempotencyKey(runIndex), + }); + insertEvent(database, { jobRunId: runId, kind: "queued", sequence: 1 }); + claimRun(database, runId, workerId, uuid(leaseIndex)); + return runId; + }; + const expectStartedEventAttempts = ( + runId: string, + sequence: number, + event: Partial & Pick, + occurredAt = 2000 + ): void => { + expect(() => + insertEvent(database, { + ...event, + attempt: 0, + jobRunId: runId, + occurredAt, + sequence, + workerInstanceId: workerId, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + insertEvent(database, { + ...event, + attempt: 1, + jobRunId: runId, + occurredAt, + sequence, + workerInstanceId: workerId, + }); + }; + + try { + insertWorker(database, workerId); + + const runningRunId = prepareStartedRun(71, 81); + const runningEvents = [ + { kind: "claimed" }, + { kind: "failed", message: "Attempt failed" }, + { kind: "lease-expired" }, + { kind: "output-truncated" }, + { kind: "progress", progressJson: "{}" }, + { kind: "stderr", message: "stderr" }, + { kind: "stdout", message: "stdout" }, + ]; + let sequence = 2; + for (const event of runningEvents) { + expectStartedEventAttempts(runningRunId, sequence, event); + sequence += 1; + } + expect(() => + insertEvent(database, { + attempt: 2, + jobRunId: runningRunId, + kind: "lease-expired", + occurredAt: 2000, + sequence, + workerInstanceId: workerId, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + expect(() => + database.sqlite.run( + `UPDATE job_run_events SET attempt = 0 + WHERE job_run_id = ? AND sequence = 2`, + [runningRunId] + ) + ).toThrow("job_run_events are append-only"); + expect(() => + database.sqlite.run( + `INSERT OR REPLACE INTO job_run_events ( + attempt, job_run_id, kind, message, occurred_at, + progress_json, sequence, worker_instance_id + ) VALUES (1, ?, 'claimed', NULL, 2000, NULL, 2, ?)`, + [runningRunId, workerId] + ) + ).toThrow(); + expect(() => + database.sqlite.run( + `INSERT OR REPLACE INTO job_run_events ( + attempt, job_run_id, kind, message, occurred_at, + progress_json, sequence, worker_instance_id + ) VALUES (0, ?, 'stdout', 'stdout', 2000, NULL, ?, ?)`, + [runningRunId, sequence, workerId] + ) + ).toThrow("job_run_events must follow the parent run lifecycle"); + + const retryRunId = prepareStartedRun(72, 82); + database.sqlite.run( + `UPDATE job_runs + SET available_at = 3000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'queued', + state_version = state_version + 1, + updated_at = 2500 + WHERE id = ?`, + [retryRunId] + ); + expectStartedEventAttempts(retryRunId, 2, { kind: "retry-scheduled" }, 2500); + + const succeededRunId = prepareStartedRun(73, 83); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 2500, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + result_json = '{}', + state = 'succeeded', + state_version = state_version + 1, + updated_at = 2500 + WHERE id = ?`, + [succeededRunId] + ); + expectStartedEventAttempts(succeededRunId, 2, { kind: "succeeded" }, 2500); + + const timedOutRunId = prepareStartedRun(74, 84); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 2500, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'timed-out', + state_version = state_version + 1, + terminal_code = 'action-timeout', + terminal_message = 'Action timed out', + updated_at = 2500 + WHERE id = ?`, + [timedOutRunId] + ); + expectStartedEventAttempts(timedOutRunId, 2, { kind: "timed-out" }, 2500); + + const cancelledRunId = uuid(75); + insertQueuedRun(database, { + id: cancelledRunId, + idempotencyKey: idempotencyKey(75), + }); + insertEvent(database, { + jobRunId: cancelledRunId, + kind: "queued", + sequence: 1, + }); + database.sqlite.run( + `UPDATE job_runs + SET cancel_requested_at = 1500, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + finished_at = 1500, + state = 'cancelled', + state_version = state_version + 1, + terminal_code = 'job/cancelled', + terminal_message = 'Cancelled before start', + updated_at = 1500 + WHERE id = ?`, + [userId, cancelledRunId] + ); + insertEvent(database, { + attempt: 0, + jobRunId: cancelledRunId, + kind: "cancel-requested", + occurredAt: 1500, + sequence: 2, + }); + insertEvent(database, { + attempt: 0, + jobRunId: cancelledRunId, + kind: "cancelled", + occurredAt: 1500, + sequence: 3, + }); + for (const kind of ["cancel-requested", "cancelled"]) { + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: cancelledRunId, + kind, + occurredAt: 1500, + sequence: 4, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + } + + const queuedRunId = uuid(76); + insertQueuedRun(database, { + id: queuedRunId, + idempotencyKey: idempotencyKey(76), + }); + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: queuedRunId, + kind: "queued", + sequence: 1, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + expect(() => + insertEvent(database, { + attempt: 0, + jobRunId: queuedRunId, + kind: "queued", + sequence: 1, + workerInstanceId: workerId, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + expect(() => + database.sqlite.run( + `INSERT OR REPLACE INTO job_run_events ( + attempt, job_run_id, kind, message, occurred_at, + progress_json, sequence, worker_instance_id + ) VALUES (0, ?, 'queued', NULL, 1000, NULL, 1, ?)`, + [queuedRunId, workerId] + ) + ).toThrow("job_run_events must follow the parent run lifecycle"); + insertEvent(database, { + attempt: 0, + jobRunId: queuedRunId, + kind: "queued", + sequence: 1, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("serializes append-only events and enforces count, payload, and byte budgets", async () => { + const database = await openFreshMigratedDatabase(); + const workerId = uuid(30); + + try { + insertWorker(database, workerId); + + const payloadRunId = uuid(31); + insertQueuedRun(database, { + id: payloadRunId, + idempotencyKey: idempotencyKey(31), + }); + insertEvent(database, { + jobRunId: payloadRunId, + kind: "queued", + sequence: 1, + }); + claimRun(database, payloadRunId, workerId, uuid(32)); + insertEvent(database, { + attempt: 1, + jobRunId: payloadRunId, + kind: "claimed", + occurredAt: 2000, + sequence: 2, + workerInstanceId: workerId, + }); + for (let index = 0; index < 967; index += 1) { + insertEvent(database, { + attempt: 1, + jobRunId: payloadRunId, + kind: "stdout", + message: "x", + occurredAt: 2000, + sequence: index + 3, + workerInstanceId: workerId, + }); + } + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: payloadRunId, + kind: "stdout", + message: "x", + occurredAt: 2000, + sequence: 970, + workerInstanceId: workerId, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + expect( + database.sqlite + .query< + { + event_bytes: number; + event_count: number; + payload_event_count: number; + }, + [string] + >( + `SELECT event_bytes, event_count, payload_event_count + FROM job_runs WHERE id = ?` + ) + .get(payloadRunId) + ).toEqual({ event_bytes: 967, event_count: 969, payload_event_count: 967 }); + + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: payloadRunId, + kind: "lease-expired", + occurredAt: 2000, + sequence: 971, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + expect(() => + database.sqlite.run( + `UPDATE job_run_events SET occurred_at = 2001 + WHERE job_run_id = ? AND sequence = 1`, + [payloadRunId] + ) + ).toThrow("job_run_events are append-only"); + expect(() => + database.sqlite.run( + "DELETE FROM job_run_events WHERE job_run_id = ? AND sequence = 1", + [payloadRunId] + ) + ).toThrow("job_run_events are append-only"); + + const totalRunId = uuid(33); + insertQueuedRun(database, { + id: totalRunId, + idempotencyKey: idempotencyKey(33), + }); + insertEvent(database, { jobRunId: totalRunId, kind: "queued", sequence: 1 }); + claimRun(database, totalRunId, workerId, uuid(34)); + insertEvent(database, { + attempt: 1, + jobRunId: totalRunId, + kind: "claimed", + occurredAt: 2000, + sequence: 2, + }); + for (let sequence = 3; sequence <= 1000; sequence += 1) { + insertEvent(database, { + attempt: 1, + jobRunId: totalRunId, + kind: "lease-expired", + occurredAt: 2000, + sequence, + }); + } + expect( + database.sqlite + .query<{ event_count: number }, [string]>( + "SELECT event_count FROM job_runs WHERE id = ?" + ) + .get(totalRunId) + ).toEqual({ event_count: 1000 }); + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: totalRunId, + kind: "lease-expired", + occurredAt: 2000, + sequence: 1001, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + + const bytesRunId = uuid(35); + insertQueuedRun(database, { + id: bytesRunId, + idempotencyKey: idempotencyKey(35), + }); + insertEvent(database, { jobRunId: bytesRunId, kind: "queued", sequence: 1 }); + claimRun(database, bytesRunId, workerId, uuid(36)); + insertEvent(database, { + attempt: 1, + jobRunId: bytesRunId, + kind: "claimed", + occurredAt: 2000, + sequence: 2, + }); + const chunk = "x".repeat(4096); + for (let index = 0; index < 246; index += 1) { + insertEvent(database, { + attempt: 1, + jobRunId: bytesRunId, + kind: "stdout", + message: chunk, + occurredAt: 2000, + sequence: index + 3, + }); + } + expect( + database.sqlite + .query<{ event_bytes: number }, [string]>( + "SELECT event_bytes FROM job_runs WHERE id = ?" + ) + .get(bytesRunId) + ).toEqual({ event_bytes: 1_007_616 }); + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: bytesRunId, + kind: "stdout", + message: "x", + occurredAt: 2000, + sequence: 249, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + for (let index = 0; index < 9; index += 1) { + insertEvent(database, { + attempt: 1, + jobRunId: bytesRunId, + kind: "failed", + message: chunk, + occurredAt: 2000, + sequence: index + 249, + workerInstanceId: workerId, + }); + } + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 2000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'failed', + state_version = state_version + 1, + terminal_code = 'job/failed', + terminal_message = 'Terminal failure', + updated_at = 2000 + WHERE id = ?`, + [bytesRunId] + ); + insertEvent(database, { + attempt: 1, + jobRunId: bytesRunId, + kind: "failed", + message: chunk, + occurredAt: 2000, + sequence: 258, + workerInstanceId: workerId, + }); + expect( + database.sqlite + .query<{ event_bytes: number }, [string]>( + "SELECT event_bytes FROM job_runs WHERE id = ?" + ) + .get(bytesRunId) + ).toEqual({ event_bytes: 1024 * 1024 }); + + const runningFailureRunId = uuid(60); + insertQueuedRun(database, { + id: runningFailureRunId, + idempotencyKey: idempotencyKey(60), + }); + insertEvent(database, { + jobRunId: runningFailureRunId, + kind: "queued", + sequence: 1, + }); + claimRun(database, runningFailureRunId, workerId, uuid(61)); + insertEvent(database, { + attempt: 1, + jobRunId: runningFailureRunId, + kind: "failed", + message: "Retryable attempt failed", + occurredAt: 2000, + sequence: 2, + workerInstanceId: workerId, + }); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 2000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'failed', + state_version = state_version + 1, + terminal_code = 'job/failed', + terminal_message = 'Terminal failure', + updated_at = 2000 + WHERE id = ?`, + [runningFailureRunId] + ); + insertEvent(database, { + attempt: 1, + jobRunId: runningFailureRunId, + kind: "failed", + message: "Terminal failure", + occurredAt: 2000, + sequence: 3, + workerInstanceId: workerId, + }); + + const queuedFailureRunId = uuid(62); + insertQueuedRun(database, { + id: queuedFailureRunId, + idempotencyKey: idempotencyKey(62), + }); + insertEvent(database, { + jobRunId: queuedFailureRunId, + kind: "queued", + sequence: 1, + }); + expect(() => + insertEvent(database, { + jobRunId: queuedFailureRunId, + kind: "failed", + message: "Invalid queued failure", + occurredAt: 1000, + sequence: 2, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + + const succeededFailureRunId = uuid(63); + insertQueuedRun(database, { + id: succeededFailureRunId, + idempotencyKey: idempotencyKey(63), + }); + insertEvent(database, { + jobRunId: succeededFailureRunId, + kind: "queued", + sequence: 1, + }); + claimRun(database, succeededFailureRunId, workerId, uuid(64)); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 2000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + result_json = '{}', + state = 'succeeded', + state_version = state_version + 1, + updated_at = 2000 + WHERE id = ?`, + [succeededFailureRunId] + ); + expect(() => + insertEvent(database, { + attempt: 1, + jobRunId: succeededFailureRunId, + kind: "failed", + message: "Invalid post-success failure", + occurredAt: 2000, + sequence: 2, + workerInstanceId: workerId, + }) + ).toThrow("job_run_events must follow the parent run lifecycle"); + } finally { + database.sqlite.close(true); + } + }, 30_000); + + test("enforces singleton CAS and replacement and deletion resistance", async () => { + const database = await openFreshMigratedDatabase(); + + try { + database.sqlite.run( + `UPDATE job_worker_control + SET claiming_paused = 1, updated_at = 1000, + updated_by_id = ?, updated_by_kind = 'user', version = 2 + WHERE id = 1`, + [userId] + ); + expect(() => + database.sqlite.run( + `UPDATE job_worker_control + SET claiming_paused = 0, updated_at = 2000, + updated_by_id = ?, updated_by_kind = 'user', version = 2 + WHERE id = 1`, + [userId] + ) + ).toThrow("job_worker_control transition is invalid"); + expect(() => + database.sqlite.run( + `INSERT OR REPLACE INTO job_worker_control ( + claiming_paused, id, updated_at, updated_by_id, + updated_by_kind, version + ) VALUES (0, 1, 2000, ?, 'user', 3)`, + [userId] + ) + ).toThrow("job_worker_control singleton already exists"); + expect(() => database.sqlite.run("DELETE FROM job_worker_control")).toThrow( + "job_worker_control singleton cannot be deleted" + ); + } finally { + database.sqlite.close(true); + } + }); + + test("enforces worker lifecycle and resource-lease fencing", async () => { + const database = await openFreshMigratedDatabase(); + const workerId = uuid(40); + const otherWorkerId = uuid(41); + const lifecycleWorkerId = uuid(42); + const runId = uuid(43); + const leaseToken = uuid(44); + + try { + insertWorker(database, workerId, 1001); + insertWorker(database, otherWorkerId, 1002); + insertWorker(database, lifecycleWorkerId, 1003); + insertQueuedRun(database, { + id: runId, + idempotencyKey: idempotencyKey(43), + resourceKeysJson: '["host.smoke"]', + }); + claimRun(database, runId, workerId, leaseToken); + + expect(() => + database.sqlite.run( + `INSERT INTO resource_leases ( + acquired_at, expires_at, job_run_id, lease_token, + renewed_at, resource_key, worker_instance_id + ) VALUES (2000, 5000, ?, ?, 2000, 'host.other', ?)`, + [runId, leaseToken, workerId] + ) + ).toThrow("resource_leases must match one active fenced claim"); + expect(() => + database.sqlite.run( + `INSERT INTO resource_leases ( + acquired_at, expires_at, job_run_id, lease_token, + renewed_at, resource_key, worker_instance_id + ) VALUES (2000, 5000, ?, ?, 2000, 'host.smoke', ?)`, + [runId, uuid(45), otherWorkerId] + ) + ).toThrow("resource_leases must match one active fenced claim"); + database.sqlite.run( + `INSERT INTO resource_leases ( + acquired_at, expires_at, job_run_id, lease_token, + renewed_at, resource_key, worker_instance_id + ) VALUES (2000, 5000, ?, ?, 2000, 'host.smoke', ?)`, + [runId, leaseToken, workerId] + ); + + database.sqlite.run( + `UPDATE job_runs + SET heartbeat_at = 2500, lease_expires_at = 6000, updated_at = 2500 + WHERE id = ?`, + [runId] + ); + database.sqlite.run( + `UPDATE resource_leases + SET expires_at = 6000, renewed_at = 2500 + WHERE resource_key = 'host.smoke'` + ); + expect(() => + database.sqlite.run( + `UPDATE resource_leases + SET expires_at = 7000, renewed_at = 3000 + WHERE resource_key = 'host.smoke'` + ) + ).toThrow("resource_leases renewal is not fenced"); + expect(() => + database.sqlite.run( + `UPDATE resource_leases SET resource_key = 'host.changed' + WHERE resource_key = 'host.smoke'` + ) + ).toThrow("resource_leases renewal is not fenced"); + + database.sqlite.run( + "UPDATE worker_instances SET heartbeat_at = 1500 WHERE id = ?", + [lifecycleWorkerId] + ); + expect(() => + database.sqlite.run( + "UPDATE worker_instances SET heartbeat_at = 1400 WHERE id = ?", + [lifecycleWorkerId] + ) + ).toThrow("worker_instances lifecycle transition is invalid"); + expect(() => + database.sqlite.run( + `UPDATE worker_instances + SET draining_at = 1600, heartbeat_at = 1700, state = 'stopped', + stopped_at = 1700 + WHERE id = ?`, + [lifecycleWorkerId] + ) + ).toThrow("worker_instances lifecycle transition is invalid"); + database.sqlite.run( + `UPDATE worker_instances + SET draining_at = 1600, heartbeat_at = 1600, state = 'draining' + WHERE id = ?`, + [lifecycleWorkerId] + ); + database.sqlite.run( + `UPDATE worker_instances + SET heartbeat_at = 1700, state = 'stopped', stopped_at = 1700 + WHERE id = ?`, + [lifecycleWorkerId] + ); + expect(() => + database.sqlite.run( + "UPDATE worker_instances SET heartbeat_at = 1800 WHERE id = ?", + [lifecycleWorkerId] + ) + ).toThrow("worker_instances lifecycle transition is invalid"); + } finally { + database.sqlite.close(true); + } + }); + + test("enforces stored run time ordering across raw insert and update boundaries", async () => { + const database = await openFreshMigratedDatabase(); + const workerId = uuid(90); + const runningRunId = uuid(93); + + const transitionToRunning = ( + firstStartedAt: number, + lastAttemptStartedAt: number, + heartbeatAt: number, + leaseExpiresAt: number, + updatedAt: number + ): void => { + database.sqlite.run( + `UPDATE job_runs + SET attempt_count = 1, + first_started_at = ?, + heartbeat_at = ?, + last_attempt_started_at = ?, + lease_expires_at = ?, + lease_owner_id = ?, + lease_token = ?, + state = 'running', + state_version = 2, + updated_at = ? + WHERE id = ?`, + [ + firstStartedAt, + heartbeatAt, + lastAttemptStartedAt, + leaseExpiresAt, + workerId, + uuid(94), + updatedAt, + runningRunId, + ] + ); + }; + + try { + insertSchedule(database); + insertWorker(database, workerId); + + expect(() => + insertQueuedRun(database, { + id: uuid(91), + idempotencyKey: idempotencyKey(91), + scheduledForAt: 1001, + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + triggerType: "schedule", + }) + ).toThrow("job_runs_schedule_check"); + const scheduledRunId = uuid(92); + insertQueuedRun(database, { + id: scheduledRunId, + idempotencyKey: idempotencyKey(92), + scheduledForAt: 1000, + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + triggerType: "schedule", + }); + expect(() => + database.sqlite.run( + `INSERT OR REPLACE INTO job_runs + SELECT * FROM job_runs WHERE id = ?`, + [scheduledRunId] + ) + ).toThrow("job_runs identity is immutable"); + expect(() => + database.sqlite.run( + "UPDATE OR REPLACE job_runs SET scheduled_for_at = 1001 WHERE id = ?", + [scheduledRunId] + ) + ).toThrow("job_runs execution snapshot is immutable"); + + insertQueuedRun(database, { + id: runningRunId, + idempotencyKey: idempotencyKey(93), + }); + expect(() => + database.sqlite.run( + "UPDATE job_runs SET available_at = 999 WHERE id = ?", + [runningRunId] + ) + ).toThrow("job_runs_available_at_check"); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET cancel_requested_at = 999, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + state_version = 2 + WHERE id = ?`, + [userId, runningRunId] + ) + ).toThrow("job_runs_cancel_request_check"); + expect(() => + database.sqlite.run( + `UPDATE OR REPLACE job_runs + SET cancel_requested_at = 1500, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + state_version = 2, + updated_at = 1200 + WHERE id = ?`, + [userId, runningRunId] + ) + ).toThrow("job_runs_cancel_request_check"); + expect(() => + database.sqlite.run( + `UPDATE OR REPLACE job_runs + SET finished_at = 1000, + state = 'cancelled', + state_version = 2, + terminal_code = 'job/cancelled', + terminal_message = 'Missing durable request' + WHERE id = ?`, + [runningRunId] + ) + ).toThrow("job_runs_cancel_request_check"); + + expect(() => transitionToRunning(999, 999, 999, 5000, 1000)).toThrow( + "job_runs_time_check" + ); + expect(() => transitionToRunning(1200, 1200, 1200, 5000, 1100)).toThrow( + "job_runs_time_check" + ); + expect(() => transitionToRunning(1100, 1050, 1100, 5000, 1100)).toThrow( + "job_runs_time_check" + ); + expect(() => transitionToRunning(1100, 1100, 1050, 5000, 1100)).toThrow( + "job_runs_lease_check" + ); + expect(() => transitionToRunning(1100, 1100, 1200, 5000, 1150)).toThrow( + "job_runs_time_check" + ); + expect(() => transitionToRunning(1100, 1100, 1100, 1100, 1100)).toThrow( + "job_runs_lease_check" + ); + + transitionToRunning(1100, 1100, 1100, 5000, 1100); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET cancel_requested_at = 1300, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + state_version = 3, + updated_at = 1200 + WHERE id = ?`, + [userId, runningRunId] + ) + ).toThrow("job_runs_cancel_request_check"); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 1050, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'failed', + state_version = 3, + terminal_code = 'job/failed', + terminal_message = 'Backdated terminal time', + updated_at = 1200 + WHERE id = ?`, + [runningRunId] + ) + ).toThrow("job_runs_time_check"); + expect(() => + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 1300, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'failed', + state_version = 3, + terminal_code = 'job/failed', + terminal_message = 'Future terminal time', + updated_at = 1200 + WHERE id = ?`, + [runningRunId] + ) + ).toThrow("job_runs_time_check"); + + database.sqlite.run( + `UPDATE job_runs + SET available_at = 5000, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'queued', + state_version = 3 + WHERE id = ?`, + [runningRunId] + ); + claimRun(database, runningRunId, workerId, uuid(95), 1100, 5000); + database.sqlite.run( + `UPDATE job_runs + SET finished_at = 1100, + heartbeat_at = NULL, + lease_expires_at = NULL, + lease_owner_id = NULL, + lease_token = NULL, + state = 'failed', + state_version = 5, + terminal_code = 'job/failed', + terminal_message = 'Clock-clamped terminal failure', + updated_at = 1100 + WHERE id = ?`, + [runningRunId] + ); + expect( + database.sqlite + .query< + { + attempt_count: number; + available_at: number; + finished_at: number; + first_started_at: number; + last_attempt_started_at: number; + state: string; + updated_at: number; + }, + [string] + >( + `SELECT attempt_count, available_at, finished_at, + first_started_at, last_attempt_started_at, state, + updated_at + FROM job_runs WHERE id = ?` + ) + .get(runningRunId) + ).toEqual({ + attempt_count: 2, + available_at: 5000, + finished_at: 1100, + first_started_at: 1100, + last_attempt_started_at: 1100, + state: "failed", + updated_at: 1100, + }); + + const cancelledRunId = uuid(96); + insertQueuedRun(database, { + id: cancelledRunId, + idempotencyKey: idempotencyKey(96), + }); + database.sqlite.run( + `UPDATE OR REPLACE job_runs + SET cancel_requested_at = 1000, + cancel_requested_by_id = ?, + cancel_requested_by_kind = 'user', + finished_at = 1000, + state = 'cancelled', + state_version = 2, + terminal_code = 'job/cancelled', + terminal_message = 'Clock-clamped queued cancellation' + WHERE id = ?`, + [userId, cancelledRunId] + ); + expect( + database.sqlite + .query< + { + cancel_requested_at: number; + finished_at: number; + state: string; + updated_at: number; + }, + [string] + >( + `SELECT cancel_requested_at, finished_at, state, updated_at + FROM job_runs WHERE id = ?` + ) + .get(cancelledRunId) + ).toEqual({ + cancel_requested_at: 1000, + finished_at: 1000, + state: "cancelled", + updated_at: 1000, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("uses bounded indexes for claim, history, expiry, and scheduler reads", async () => { + const database = await openFreshMigratedDatabase(); + + try { + expectUsesIndexWithoutTemporarySort( + database, + `SELECT id FROM job_runs + WHERE state = 'queued' AND available_at <= 1000 + ORDER BY available_at ASC, priority DESC, queued_at ASC, id ASC + LIMIT 32`, + "job_runs_claim_idx" + ); + expectUsesIndexWithoutTemporarySort( + database, + `SELECT id FROM job_runs + WHERE state = 'queued' AND available_at <= 1000 + AND available_at = 900 AND priority = 2 + AND queued_at = 800 AND id > '019f0000-0000-7000-8000-000000000001' + ORDER BY available_at ASC, priority DESC, queued_at ASC, id ASC + LIMIT 32`, + "job_runs_claim_idx", + undefined, + "id>?" + ); + expectUsesIndexWithoutTemporarySort( + database, + `SELECT id FROM job_runs + WHERE state = 'queued' AND available_at <= 1000 + AND available_at = 900 AND priority = 2 AND queued_at > 800 + ORDER BY available_at ASC, priority DESC, queued_at ASC, id ASC + LIMIT 32`, + "job_runs_claim_idx", + undefined, + "queued_at>?" + ); + expectUsesIndexWithoutTemporarySort( + database, + `SELECT id FROM job_runs + WHERE state = 'queued' AND available_at <= 1000 + AND available_at = 900 AND priority < 2 + ORDER BY available_at ASC, priority DESC, queued_at ASC, id ASC + LIMIT 32`, + "job_runs_claim_idx", + undefined, + "priority 900 + ORDER BY available_at ASC, priority DESC, queued_at ASC, id ASC + LIMIT 32`, + "job_runs_claim_idx", + undefined, + "available_at>? AND available_at + (900, 'system.worker-smoke-001') + ORDER BY next_run_at ASC, id ASC LIMIT 32`, + "scheduled_jobs_due_idx", + undefined, + "(next_run_at,id)>(?,?)" + ); + expectUsesIndexWithoutTemporarySort( + database, + `SELECT id FROM worker_instances + WHERE heartbeat_at < 1000 ORDER BY heartbeat_at ASC, id ASC`, + "worker_instances_heartbeat_id_idx" + ); + expectUsesIndexWithoutTemporarySort( + database, + `SELECT resource_key FROM resource_leases + WHERE expires_at <= 1000 ORDER BY expires_at ASC, resource_key ASC`, + "resource_leases_expiry_key_idx" + ); + const eventPlan = database.sqlite + .query( + ` + EXPLAIN QUERY PLAN + SELECT sequence FROM job_run_events + WHERE job_run_id = ? + ORDER BY sequence DESC LIMIT 50 + ` + ) + .all(uuid(51)); + expect( + eventPlan.some(({ detail }) => detail.includes("PRIMARY KEY")) + ).toBeTrue(); + expect( + eventPlan.some(({ detail }) => detail.includes("USE TEMP B-TREE")) + ).toBeFalse(); + } finally { + database.sqlite.close(true); + } + }); +}); diff --git a/greenfield/src/server/database/migrations/migrationGraph.test.ts b/greenfield/src/server/database/migrations/migrationGraph.test.ts index e2137d7e6..fef89d4fc 100644 --- a/greenfield/src/server/database/migrations/migrationGraph.test.ts +++ b/greenfield/src/server/database/migrations/migrationGraph.test.ts @@ -41,10 +41,16 @@ const expectedTables: string[] = [ "automation_principals", "incident_observations", "incidents", + "job_disable_intents", + "job_run_events", + "job_runs", + "job_worker_control", "monitor_runs", "notifications", "realtime_events", "reports", + "resource_leases", + "scheduled_jobs", "schema_migrations", "task_automation_profiles", "task_events", @@ -56,6 +62,7 @@ const expectedTables: string[] = [ "user_totp_factors", "user_webauthn_credentials", "users", + "worker_instances", ]; describe("database migration graph", () => { test("bounds schema inventory by the largest valid prefix before later object drops", () => { @@ -110,13 +117,43 @@ describe("database migration graph", () => { "incidents_validate_details_update", "incident_observations_validate_details_insert", "incident_observations_validate_details_update", + "job_disable_intents_reject_closed_update", + "job_disable_intents_reject_content_update", + "job_disable_intents_reject_delete", + "job_disable_intents_reject_replace", + "job_run_events_reject_delete", + "job_run_events_reject_replace", + "job_run_events_reject_update", + "job_run_events_update_parent_counters", + "job_run_events_validate_insert", + "job_runs_reject_delete", + "job_runs_reject_replace", + "job_runs_reject_snapshot_update", + "job_runs_validate_lifecycle_update", + "job_runs_validate_resource_keys_insert", + "job_worker_control_reject_delete", + "job_worker_control_reject_replace", + "job_worker_control_validate_update", + "resource_leases_reject_identity_update", + "resource_leases_validate_insert", + "resource_leases_validate_renewal_update", "schema_migrations_reject_replace", "schema_migrations_reject_update", "schema_migrations_reject_delete", + "scheduled_jobs_reject_delete", + "scheduled_jobs_reject_identity_update", + "scheduled_jobs_reject_replace", + "scheduled_jobs_validate_resource_keys_insert", + "scheduled_jobs_validate_resource_keys_update", + "scheduled_jobs_validate_version_update", "task_events_validate_payload", "task_events_reject_replace", "task_events_reject_update", "task_events_reject_delete", + "worker_instances_reject_active_delete", + "worker_instances_reject_identity_update", + "worker_instances_reject_replace", + "worker_instances_validate_lifecycle_update", ]) { expect(foundationSql).toContain(`CREATE TRIGGER ${trigger}`); } @@ -157,6 +194,18 @@ describe("database migration graph", () => { tableDefinitions.find((row) => row.name === "task_notification_outbox") ?.wr ).toBe(1); + for (const tableName of [ + "job_disable_intents", + "job_run_events", + "job_runs", + "resource_leases", + "scheduled_jobs", + "worker_instances", + ]) { + expect(tableDefinitions.find((row) => row.name === tableName)?.wr).toBe( + 1 + ); + } const textPrimaryKeys = database.sqlite .query(` SELECT diff --git a/greenfield/src/server/database/schema/automationPrincipalCapabilities.ts b/greenfield/src/server/database/schema/automationPrincipalCapabilities.ts index 7f109f897..d84c95cbd 100644 --- a/greenfield/src/server/database/schema/automationPrincipalCapabilities.ts +++ b/greenfield/src/server/database/schema/automationPrincipalCapabilities.ts @@ -18,7 +18,7 @@ export const automationPrincipalCapabilities = sqliteTable( (table) => [ check( "automation_principal_capabilities_capability_check", - sql`${table.capability} IN ('agents:read', 'agents:write', 'monitoring:write', 'notifications:read', 'notifications:write', 'reports:read', 'reports:write', 'tasks:read', 'tasks:write')` + sql`${table.capability} IN ('agents:read', 'agents:write', 'jobs:read', 'jobs:write', 'monitoring:write', 'notifications:read', 'notifications:write', 'reports:read', 'reports:write', 'tasks:read', 'tasks:write')` ), check( "automation_principal_capabilities_granted_at_check", diff --git a/greenfield/src/server/database/schema/checks.ts b/greenfield/src/server/database/schema/checks.ts index 8490132ce..d2eec38d3 100644 --- a/greenfield/src/server/database/schema/checks.ts +++ b/greenfield/src/server/database/schema/checks.ts @@ -20,7 +20,7 @@ export function nulFreeTextCheck(column: SQLWrapper) { * @param exactLength Required hexadecimal character count. * @returns Drizzle SQL expression for the storage constraint. */ -export function lowercaseHexTextCheck(column: SQLWrapper, exactLength: 32 | 64) { +export function lowercaseHexTextCheck(column: SQLWrapper, exactLength: 32 | 40 | 64) { const exactLengthSql = sql.raw(String(exactLength)); return sql`length(${column}) = ${exactLengthSql} AND ${nulFreeTextCheck(column)} AND ${column} NOT GLOB '*[^0-9a-f]*'`; } diff --git a/greenfield/src/server/database/schema/drizzleSchema.ts b/greenfield/src/server/database/schema/drizzleSchema.ts index afc32329f..e4867ff79 100644 --- a/greenfield/src/server/database/schema/drizzleSchema.ts +++ b/greenfield/src/server/database/schema/drizzleSchema.ts @@ -13,10 +13,16 @@ export { automationPrincipalCapabilities } from "./automationPrincipalCapabiliti export { automationPrincipals } from "./automationPrincipals.ts"; export { incidentObservations } from "./incidentObservations.ts"; export { incidents } from "./incidents.ts"; +export { jobDisableIntents } from "./jobDisableIntents.ts"; +export { jobRunEvents } from "./jobRunEvents.ts"; +export { jobRuns } from "./jobRuns.ts"; +export { jobWorkerControl } from "./jobWorkerControl.ts"; export { monitorRuns } from "./monitorRuns.ts"; export { notifications } from "./notifications.ts"; export { realtimeEvents } from "./realtime.ts"; +export { resourceLeases } from "./resourceLeases.ts"; export { reports } from "./reports.ts"; +export { scheduledJobs } from "./scheduledJobs.ts"; export { schemaMigrations } from "./schemaMigrations.ts"; export { taskAutomationProfiles } from "./taskAutomationProfiles.ts"; export { taskEvents } from "./taskEvents.ts"; @@ -28,3 +34,4 @@ export { userRecoveryCodes } from "./userRecoveryCodes.ts"; export { userTotpFactors } from "./userTotpFactors.ts"; export { userWebAuthnCredentials } from "./userWebAuthnCredentials.ts"; export { users } from "./users.ts"; +export { workerInstances } from "./workerInstances.ts"; diff --git a/greenfield/src/server/database/schema/jobChecks.ts b/greenfield/src/server/database/schema/jobChecks.ts new file mode 100644 index 000000000..5458e28de --- /dev/null +++ b/greenfield/src/server/database/schema/jobChecks.ts @@ -0,0 +1,81 @@ +import { sql, type SQLWrapper } from "drizzle-orm"; + +import { + boundedControlSafeTextCheck, + boundedNonBlankTextCheck, + nulFreeTextCheck, + uuidV7TextCheck, +} from "./checks.ts"; + +/** + * Canonical lowercase identifier used by schedules, actions, and resource leases. + * @returns SQL predicate for the bounded canonical key. + */ +export function boundedJobKeyCheck(column: SQLWrapper, maximumLength: number) { + const maximumLengthSql = sql.raw(String(maximumLength)); + return sql`length(${column}) BETWEEN 1 AND ${maximumLengthSql} AND ${nulFreeTextCheck(column)} AND ${column} = lower(${column}) AND substr(${column}, 1, 1) GLOB '[a-z0-9]' AND ${column} NOT GLOB '*[^a-z0-9._-]*'`; +} + +/** + * UTF-8 byte-bounded JSON object stored in canonical text form. + * @returns SQL predicate for an object-root JSON value within its byte budget. + */ +export function boundedJsonObjectCheck(column: SQLWrapper, maximumBytes: number) { + const maximumBytesSql = sql.raw(String(maximumBytes)); + return sql`length(CAST(${column} AS BLOB)) <= ${maximumBytesSql} AND CASE WHEN json_valid(${column}) THEN json_type(${column}) = 'object' ELSE 0 END`; +} + +/** + * UTF-8 byte-bounded JSON array stored in canonical text form. + * @returns SQL predicate for an array-root JSON value within its byte budget. + */ +export function boundedJsonArrayCheck(column: SQLWrapper, maximumBytes: number) { + const maximumBytesSql = sql.raw(String(maximumBytes)); + return sql`length(CAST(${column} AS BLOB)) <= ${maximumBytesSql} AND CASE WHEN json_valid(${column}) THEN json_type(${column}) = 'array' ELSE 0 END`; +} + +/** + * Actor identity accepted by durable job mutations and lifecycle transitions. + * @returns SQL predicate for an allowed actor kind and canonical identity. + */ +export function jobActorCheck( + kind: SQLWrapper, + id: SQLWrapper, + options: { readonly allowSystem?: boolean } = {} +) { + const system = + options.allowSystem === true + ? sql` OR (${kind} = 'system' AND ${boundedJobKeyCheck(id, 128)})` + : sql``; + return sql`((${kind} = 'user' AND ${uuidV7TextCheck(id)}) OR (${kind} = 'automation' AND ${boundedJobKeyCheck(id, 64)})${system})`; +} + +/** + * Optional bounded human-readable terminal or progress message. + * @returns SQL predicate for a null or bounded safe message. + */ +export function optionalJobMessageCheck( + column: SQLWrapper, + maximumCodePoints: number, + maximumBytes: number +) { + const maximumBytesSql = sql.raw(String(maximumBytes)); + return sql`(${column} IS NULL OR (${boundedControlSafeTextCheck(column, maximumCodePoints)} AND length(CAST(${column} AS BLOB)) <= ${maximumBytesSql}))`; +} + +/** + * Optional lowercase terminal code with one slash-delimited namespace. + * @returns SQL predicate for a null or canonical terminal code. + */ +export function optionalJobTerminalCodeCheck(column: SQLWrapper, maximumLength: number) { + const maximumLengthSql = sql.raw(String(maximumLength)); + return sql`(${column} IS NULL OR (length(${column}) BETWEEN 1 AND ${maximumLengthSql} AND ${nulFreeTextCheck(column)} AND ${column} = lower(${column}) AND substr(${column}, 1, 1) GLOB '[a-z0-9]' AND ${column} NOT GLOB '*[^a-z0-9._/-]*'))`; +} + +/** + * Optional bounded, nonblank identifier that preserves case. + * @returns SQL predicate for a null or bounded identifier. + */ +export function optionalBoundedJobTextCheck(column: SQLWrapper, maximumLength: number) { + return sql`(${column} IS NULL OR (${boundedNonBlankTextCheck(column, maximumLength)}))`; +} diff --git a/greenfield/src/server/database/schema/jobDisableIntents.ts b/greenfield/src/server/database/schema/jobDisableIntents.ts new file mode 100644 index 000000000..79bdd2dd0 --- /dev/null +++ b/greenfield/src/server/database/schema/jobDisableIntents.ts @@ -0,0 +1,101 @@ +import { sql } from "drizzle-orm"; +import { + check, + index, + integer, + sqliteTable, + text, + uniqueIndex, +} from "drizzle-orm/sqlite-core"; + +import { + boundedControlSafeTextCheck, + boundedNonBlankTextCheck, + timestampMillisecondsCheck, + uuidV7TextCheck, +} from "./checks.ts"; +import { jobActorCheck } from "./jobChecks.ts"; +import { scheduledJobs } from "./scheduledJobs.ts"; + +/** Append-only operator intent explaining why one schedule or external cron is disabled. */ +export const jobDisableIntents = sqliteTable( + "job_disable_intents", + { + createdAt: integer("created_at", { mode: "timestamp_ms" }).notNull(), + createdById: text("created_by_id").notNull(), + createdByKind: text("created_by_kind", { + enum: ["automation", "user"], + }).notNull(), + endedAt: integer("ended_at", { mode: "timestamp_ms" }), + endedById: text("ended_by_id"), + endedByKind: text("ended_by_kind", { + enum: ["automation", "system", "user"], + }), + endedReason: text("ended_reason", { + enum: ["expired", "re-enabled", "replaced"], + }), + expiresAt: integer("expires_at", { mode: "timestamp_ms" }), + externalJobId: text("external_job_id"), + externalProvider: text("external_provider", { enum: ["openclaw"] }), + id: text("id").notNull().primaryKey(), + reason: text("reason").notNull(), + scheduledJobId: text("scheduled_job_id").references(() => scheduledJobs.id, { + onDelete: "restrict", + onUpdate: "restrict", + }), + targetKind: text("target_kind", { + enum: ["dashboard-schedule", "openclaw-cron"], + }).notNull(), + }, + (table) => [ + check( + "job_disable_intents_created_at_check", + timestampMillisecondsCheck(table.createdAt) + ), + check( + "job_disable_intents_created_actor_check", + jobActorCheck(table.createdByKind, table.createdById) + ), + check( + "job_disable_intents_end_check", + sql`(${table.endedAt} IS NULL AND ${table.endedByKind} IS NULL AND ${table.endedById} IS NULL AND ${table.endedReason} IS NULL) OR (${table.endedAt} IS NOT NULL AND ${timestampMillisecondsCheck(table.endedAt)} AND ${table.endedAt} >= ${table.createdAt} AND ${table.endedByKind} IS NOT NULL AND ${table.endedById} IS NOT NULL AND ${table.endedReason} IN ('expired', 're-enabled', 'replaced') AND ${jobActorCheck(table.endedByKind, table.endedById, { allowSystem: true })} AND (${table.endedReason} <> 'expired' OR (${table.endedByKind} = 'system' AND ${table.expiresAt} IS NOT NULL AND ${table.endedAt} >= ${table.expiresAt})))` + ), + check( + "job_disable_intents_expiry_check", + sql`${table.expiresAt} IS NULL OR (${timestampMillisecondsCheck(table.expiresAt)} AND ${table.expiresAt} > ${table.createdAt})` + ), + check( + "job_disable_intents_external_job_id_check", + sql`${table.externalJobId} IS NULL OR (${boundedNonBlankTextCheck(table.externalJobId, 256)})` + ), + check("job_disable_intents_id_check", uuidV7TextCheck(table.id)), + check( + "job_disable_intents_reason_check", + sql`${boundedControlSafeTextCheck(table.reason, 1000)} AND length(CAST(${table.reason} AS BLOB)) <= 4000` + ), + check( + "job_disable_intents_target_check", + sql`(${table.targetKind} = 'dashboard-schedule' AND ${table.scheduledJobId} IS NOT NULL AND ${table.externalProvider} IS NULL AND ${table.externalJobId} IS NULL) OR (${table.targetKind} = 'openclaw-cron' AND ${table.scheduledJobId} IS NULL AND ${table.externalProvider} = 'openclaw' AND ${table.externalJobId} IS NOT NULL)` + ), + uniqueIndex("job_disable_intents_active_schedule_unique") + .on(table.scheduledJobId) + .where(sql`${table.scheduledJobId} IS NOT NULL AND ${table.endedAt} IS NULL`), + uniqueIndex("job_disable_intents_active_external_unique") + .on(table.externalProvider, table.externalJobId) + .where(sql`${table.externalJobId} IS NOT NULL AND ${table.endedAt} IS NULL`), + index("job_disable_intents_active_expiry_idx") + .on(table.expiresAt, table.id) + .where(sql`${table.expiresAt} IS NOT NULL AND ${table.endedAt} IS NULL`), + index("job_disable_intents_schedule_created_id_idx").on( + table.scheduledJobId, + table.createdAt, + table.id + ), + index("job_disable_intents_external_created_id_idx").on( + table.externalProvider, + table.externalJobId, + table.createdAt, + table.id + ), + ] +); diff --git a/greenfield/src/server/database/schema/jobRunEvents.ts b/greenfield/src/server/database/schema/jobRunEvents.ts new file mode 100644 index 000000000..4b7ee2ed9 --- /dev/null +++ b/greenfield/src/server/database/schema/jobRunEvents.ts @@ -0,0 +1,85 @@ +import { sql } from "drizzle-orm"; +import { + check, + index, + integer, + primaryKey, + sqliteTable, + text, +} from "drizzle-orm/sqlite-core"; + +import { timestampMillisecondsCheck } from "./checks.ts"; +import { boundedJsonObjectCheck, optionalJobMessageCheck } from "./jobChecks.ts"; +import { jobRuns } from "./jobRuns.ts"; +import { workerInstances } from "./workerInstances.ts"; + +/** Ordered bounded progress and lifecycle timeline for one durable job run. */ +export const jobRunEvents = sqliteTable( + "job_run_events", + { + attempt: integer("attempt").notNull(), + jobRunId: text("job_run_id") + .notNull() + .references(() => jobRuns.id, { + onDelete: "cascade", + onUpdate: "restrict", + }), + kind: text("kind", { + enum: [ + "cancel-requested", + "cancelled", + "claimed", + "failed", + "lease-expired", + "output-truncated", + "progress", + "queued", + "retry-scheduled", + "stderr", + "stdout", + "succeeded", + "timed-out", + ], + }).notNull(), + message: text("message"), + occurredAt: integer("occurred_at", { mode: "timestamp_ms" }).notNull(), + progressJson: text("progress_json"), + sequence: integer("sequence").notNull(), + workerInstanceId: text("worker_instance_id").references( + () => workerInstances.id, + { + onDelete: "set null", + onUpdate: "restrict", + } + ), + }, + (table) => [ + check("job_run_events_attempt_check", sql`${table.attempt} BETWEEN 0 AND 10`), + check( + "job_run_events_kind_check", + sql`${table.kind} IN ('cancel-requested', 'cancelled', 'claimed', 'failed', 'lease-expired', 'output-truncated', 'progress', 'queued', 'retry-scheduled', 'stderr', 'stdout', 'succeeded', 'timed-out')` + ), + check( + "job_run_events_message_check", + optionalJobMessageCheck(table.message, 4096, 4096) + ), + check( + "job_run_events_occurred_at_check", + timestampMillisecondsCheck(table.occurredAt) + ), + check( + "job_run_events_payload_shape_check", + sql`(${table.kind} = 'progress' AND ${table.progressJson} IS NOT NULL AND ${boundedJsonObjectCheck(table.progressJson, 16_384)}) OR (${table.kind} IN ('stderr', 'stdout') AND ${table.message} IS NOT NULL AND ${table.progressJson} IS NULL) OR (${table.kind} NOT IN ('progress', 'stderr', 'stdout') AND ${table.progressJson} IS NULL)` + ), + check("job_run_events_sequence_check", sql`${table.sequence} BETWEEN 1 AND 1000`), + primaryKey({ + columns: [table.jobRunId, table.sequence], + name: "job_run_events_pk", + }), + index("job_run_events_occurred_run_sequence_idx").on( + table.occurredAt, + table.jobRunId, + table.sequence + ), + ] +); diff --git a/greenfield/src/server/database/schema/jobRuns.ts b/greenfield/src/server/database/schema/jobRuns.ts new file mode 100644 index 000000000..ee0b529cf --- /dev/null +++ b/greenfield/src/server/database/schema/jobRuns.ts @@ -0,0 +1,222 @@ +import { asc, desc, sql } from "drizzle-orm"; +import { + check, + index, + integer, + sqliteTable, + text, + uniqueIndex, +} from "drizzle-orm/sqlite-core"; + +import { + jobRunEventMaximum, + jobRunOutputMaximumBytes, + jobRunPayloadEventMaximum, +} from "../../../contracts/jobModel.ts"; +import { + boundedCanonicalBase64UrlTextCheck, + boundedControlSafeTextCheck, + lowercaseHexTextCheck, + timestampMillisecondsCheck, + uuidV7TextCheck, +} from "./checks.ts"; +import { + boundedJobKeyCheck, + boundedJsonArrayCheck, + boundedJsonObjectCheck, + jobActorCheck, + optionalJobMessageCheck, + optionalJobTerminalCodeCheck, +} from "./jobChecks.ts"; +import { scheduledJobs } from "./scheduledJobs.ts"; +import { workerInstances } from "./workerInstances.ts"; + +/** Durable queue, execution state, and bounded terminal snapshot for one job run. */ +export const jobRuns = sqliteTable( + "job_runs", + { + actionKey: text("action_key").notNull(), + attemptCount: integer("attempt_count").notNull().default(0), + attemptLimit: integer("attempt_limit").notNull(), + availableAt: integer("available_at", { mode: "timestamp_ms" }).notNull(), + cancellationPolicy: text("cancellation_policy", { + enum: ["cooperative", "never", "queued-only"], + }).notNull(), + cancelRequestedAt: integer("cancel_requested_at", { mode: "timestamp_ms" }), + cancelRequestedById: text("cancel_requested_by_id"), + cancelRequestedByKind: text("cancel_requested_by_kind", { + enum: ["automation", "system", "user"], + }), + displayName: text("display_name").notNull(), + enqueueSha256: text("enqueue_sha256").notNull(), + eventBytes: integer("event_bytes").notNull().default(0), + eventCount: integer("event_count").notNull().default(0), + finishedAt: integer("finished_at", { mode: "timestamp_ms" }), + firstStartedAt: integer("first_started_at", { mode: "timestamp_ms" }), + heartbeatAt: integer("heartbeat_at", { mode: "timestamp_ms" }), + id: text("id").notNull().primaryKey(), + idempotencyKey: text("idempotency_key").notNull(), + lastAttemptStartedAt: integer("last_attempt_started_at", { + mode: "timestamp_ms", + }), + leaseExpiresAt: integer("lease_expires_at", { mode: "timestamp_ms" }), + leaseOwnerId: text("lease_owner_id").references(() => workerInstances.id, { + onDelete: "restrict", + onUpdate: "restrict", + }), + leaseToken: text("lease_token"), + payloadEventCount: integer("payload_event_count").notNull().default(0), + payloadJson: text("payload_json").notNull(), + priority: integer("priority").notNull(), + queuedAt: integer("queued_at", { mode: "timestamp_ms" }).notNull(), + requestedById: text("requested_by_id").notNull(), + requestedByKind: text("requested_by_kind", { + enum: ["automation", "system", "user"], + }).notNull(), + resourceClass: text("resource_class", { + enum: ["exclusive", "host-heavy", "interactive", "light", "network"], + }).notNull(), + resourceKeysJson: text("resource_keys_json").notNull(), + resultJson: text("result_json"), + retrySafe: integer("retry_safe", { mode: "boolean" }).notNull(), + scheduledForAt: integer("scheduled_for_at", { mode: "timestamp_ms" }), + scheduledJobId: text("scheduled_job_id").references(() => scheduledJobs.id, { + onDelete: "restrict", + onUpdate: "restrict", + }), + scheduledJobVersion: integer("scheduled_job_version"), + state: text("state", { + enum: ["cancelled", "failed", "queued", "running", "succeeded", "timed-out"], + }).notNull(), + stateVersion: integer("state_version").notNull().default(1), + terminalCode: text("terminal_code"), + terminalMessage: text("terminal_message"), + timeoutMs: integer("timeout_ms").notNull(), + triggerType: text("trigger_type", { + enum: ["manual", "schedule", "startup", "system"], + }).notNull(), + updatedAt: integer("updated_at", { mode: "timestamp_ms" }).notNull(), + }, + (table) => [ + check("job_runs_action_key_check", boundedJobKeyCheck(table.actionKey, 128)), + check( + "job_runs_attempt_check", + sql`${table.attemptLimit} BETWEEN 1 AND 10 AND ${table.attemptCount} BETWEEN 0 AND ${table.attemptLimit} AND ((${table.attemptCount} = 0 AND ${table.firstStartedAt} IS NULL AND ${table.lastAttemptStartedAt} IS NULL) OR (${table.attemptCount} > 0 AND ${table.firstStartedAt} IS NOT NULL AND ${table.lastAttemptStartedAt} IS NOT NULL))` + ), + check( + "job_runs_available_at_check", + sql`${timestampMillisecondsCheck(table.availableAt)} AND ${table.availableAt} >= ${table.queuedAt}` + ), + check( + "job_runs_cancellation_policy_check", + sql`${table.cancellationPolicy} IN ('cooperative', 'never', 'queued-only')` + ), + check( + "job_runs_cancel_request_check", + sql`(${table.state} <> 'cancelled' AND ${table.cancelRequestedAt} IS NULL AND ${table.cancelRequestedByKind} IS NULL AND ${table.cancelRequestedById} IS NULL) OR (${table.cancellationPolicy} <> 'never' AND ${table.cancelRequestedAt} IS NOT NULL AND ${timestampMillisecondsCheck(table.cancelRequestedAt)} AND ${table.cancelRequestedAt} >= ${table.queuedAt} AND ${table.cancelRequestedAt} <= ${table.updatedAt} AND ${table.cancelRequestedByKind} IS NOT NULL AND ${table.cancelRequestedById} IS NOT NULL AND ${jobActorCheck(table.cancelRequestedByKind, table.cancelRequestedById, { allowSystem: true })})` + ), + check( + "job_runs_display_name_check", + sql`${boundedControlSafeTextCheck(table.displayName, 160)} AND length(CAST(${table.displayName} AS BLOB)) <= 640` + ), + check( + "job_runs_enqueue_sha256_check", + lowercaseHexTextCheck(table.enqueueSha256, 64) + ), + check( + "job_runs_event_budget_check", + sql`${table.eventCount} BETWEEN 0 AND ${sql.raw(String(jobRunEventMaximum))} AND ${table.payloadEventCount} BETWEEN 0 AND ${sql.raw(String(jobRunPayloadEventMaximum))} AND ${table.payloadEventCount} <= ${table.eventCount} AND ${table.eventBytes} BETWEEN 0 AND ${sql.raw(String(jobRunOutputMaximumBytes))}` + ), + check("job_runs_id_check", uuidV7TextCheck(table.id)), + check( + "job_runs_idempotency_key_check", + boundedCanonicalBase64UrlTextCheck(table.idempotencyKey, 32, 128) + ), + check( + "job_runs_lease_check", + sql`(${table.state} <> 'running' AND ${table.leaseOwnerId} IS NULL AND ${table.leaseToken} IS NULL AND ${table.leaseExpiresAt} IS NULL AND ${table.heartbeatAt} IS NULL) OR (${table.state} = 'running' AND ${table.leaseOwnerId} IS NOT NULL AND ${table.leaseToken} IS NOT NULL AND ${uuidV7TextCheck(table.leaseToken)} AND ${table.leaseExpiresAt} IS NOT NULL AND ${table.heartbeatAt} IS NOT NULL AND ${timestampMillisecondsCheck(table.heartbeatAt)} AND ${timestampMillisecondsCheck(table.leaseExpiresAt)} AND ${table.heartbeatAt} >= ${table.lastAttemptStartedAt} AND ${table.leaseExpiresAt} > ${table.heartbeatAt})` + ), + check( + "job_runs_payload_json_check", + boundedJsonObjectCheck(table.payloadJson, 65_536) + ), + check("job_runs_priority_check", sql`${table.priority} BETWEEN -100 AND 100`), + check( + "job_runs_requested_actor_check", + jobActorCheck(table.requestedByKind, table.requestedById, { + allowSystem: true, + }) + ), + check( + "job_runs_resource_class_check", + sql`${table.resourceClass} IN ('exclusive', 'host-heavy', 'interactive', 'light', 'network')` + ), + check( + "job_runs_resource_keys_json_check", + boundedJsonArrayCheck(table.resourceKeysJson, 4096) + ), + check( + "job_runs_result_json_check", + sql`${table.resultJson} IS NULL OR (${boundedJsonObjectCheck(table.resultJson, 65_536)})` + ), + check("job_runs_retry_safe_check", sql`${table.retrySafe} IN (0, 1)`), + check( + "job_runs_schedule_check", + sql`(${table.triggerType} = 'schedule' AND ${table.scheduledJobId} IS NOT NULL AND ${table.scheduledJobVersion} BETWEEN 1 AND 9007199254740991 AND ${table.scheduledForAt} IS NOT NULL AND ${timestampMillisecondsCheck(table.scheduledForAt)} AND ${table.scheduledForAt} <= ${table.queuedAt}) OR (${table.triggerType} = 'manual' AND ${table.scheduledJobId} IS NOT NULL AND ${table.scheduledJobVersion} BETWEEN 1 AND 9007199254740991 AND ${table.scheduledForAt} IS NULL) OR (${table.triggerType} IN ('startup', 'system') AND ${table.scheduledJobId} IS NULL AND ${table.scheduledJobVersion} IS NULL AND ${table.scheduledForAt} IS NULL)` + ), + check( + "job_runs_state_check", + sql`${table.state} IN ('cancelled', 'failed', 'queued', 'running', 'succeeded', 'timed-out') AND ((${table.state} = 'queued' AND ${table.finishedAt} IS NULL AND ${table.resultJson} IS NULL AND ${table.terminalCode} IS NULL AND ${table.terminalMessage} IS NULL) OR (${table.state} = 'running' AND ${table.attemptCount} > 0 AND ${table.finishedAt} IS NULL AND ${table.resultJson} IS NULL AND ${table.terminalCode} IS NULL AND ${table.terminalMessage} IS NULL) OR (${table.state} = 'succeeded' AND ${table.attemptCount} > 0 AND ${table.finishedAt} IS NOT NULL AND ${table.resultJson} IS NOT NULL AND ${table.terminalCode} IS NULL AND ${table.terminalMessage} IS NULL) OR (${table.state} IN ('failed', 'timed-out') AND ${table.attemptCount} > 0 AND ${table.finishedAt} IS NOT NULL AND ${table.resultJson} IS NULL AND ${table.terminalCode} IS NOT NULL AND ${table.terminalMessage} IS NOT NULL) OR (${table.state} = 'cancelled' AND ${table.finishedAt} IS NOT NULL AND ${table.resultJson} IS NULL AND ${table.terminalCode} IS NOT NULL AND ${table.terminalMessage} IS NOT NULL))` + ), + check( + "job_runs_state_version_check", + sql`${table.stateVersion} BETWEEN 1 AND 9007199254740991` + ), + check( + "job_runs_terminal_code_check", + optionalJobTerminalCodeCheck(table.terminalCode, 128) + ), + check( + "job_runs_terminal_message_check", + optionalJobMessageCheck(table.terminalMessage, 2000, 8000) + ), + check( + "job_runs_timeout_check", + sql`${table.timeoutMs} BETWEEN 1000 AND 86400000` + ), + check( + "job_runs_time_check", + sql`${timestampMillisecondsCheck(table.queuedAt)} AND ${timestampMillisecondsCheck(table.updatedAt)} AND ${table.updatedAt} >= ${table.queuedAt} AND (${table.firstStartedAt} IS NULL OR (${timestampMillisecondsCheck(table.firstStartedAt)} AND ${table.firstStartedAt} BETWEEN ${table.queuedAt} AND ${table.updatedAt})) AND (${table.lastAttemptStartedAt} IS NULL OR (${table.firstStartedAt} IS NOT NULL AND ${timestampMillisecondsCheck(table.lastAttemptStartedAt)} AND ${table.lastAttemptStartedAt} BETWEEN ${table.firstStartedAt} AND ${table.updatedAt})) AND (${table.heartbeatAt} IS NULL OR (${table.lastAttemptStartedAt} IS NOT NULL AND ${timestampMillisecondsCheck(table.heartbeatAt)} AND ${table.heartbeatAt} BETWEEN ${table.lastAttemptStartedAt} AND ${table.updatedAt})) AND (${table.cancelRequestedAt} IS NULL OR (${timestampMillisecondsCheck(table.cancelRequestedAt)} AND ${table.cancelRequestedAt} BETWEEN ${table.queuedAt} AND ${table.updatedAt})) AND (${table.finishedAt} IS NULL OR (${timestampMillisecondsCheck(table.finishedAt)} AND ${table.finishedAt} BETWEEN COALESCE(${table.lastAttemptStartedAt}, ${table.queuedAt}) AND ${table.updatedAt}))` + ), + uniqueIndex("job_runs_idempotency_unique").on( + table.requestedByKind, + table.requestedById, + table.idempotencyKey + ), + index("job_runs_claim_idx") + .on( + asc(table.availableAt), + desc(table.priority), + asc(table.queuedAt), + asc(table.id) + ) + .where(sql`${table.state} = 'queued'`), + uniqueIndex("job_runs_one_active_schedule_idx") + .on(table.scheduledJobId) + .where( + sql`${table.scheduledJobId} IS NOT NULL AND ${table.state} IN ('queued', 'running')` + ), + index("job_runs_queued_id_idx").on(table.queuedAt, table.id), + index("job_runs_schedule_queued_id_idx").on( + table.scheduledJobId, + table.queuedAt, + table.id + ), + index("job_runs_running_lease_idx") + .on(table.leaseExpiresAt, table.id) + .where(sql`${table.state} = 'running'`), + index("job_runs_running_owner_id_idx") + .on(table.leaseOwnerId, table.id) + .where(sql`${table.state} = 'running'`), + ] +); diff --git a/greenfield/src/server/database/schema/jobWorkerControl.ts b/greenfield/src/server/database/schema/jobWorkerControl.ts new file mode 100644 index 000000000..6708066a3 --- /dev/null +++ b/greenfield/src/server/database/schema/jobWorkerControl.ts @@ -0,0 +1,39 @@ +import { sql } from "drizzle-orm"; +import { check, integer, sqliteTable, text } from "drizzle-orm/sqlite-core"; + +import { timestampMillisecondsCheck } from "./checks.ts"; +import { jobActorCheck } from "./jobChecks.ts"; + +/** Required singleton controlling cross-process admission of new worker claims. */ +export const jobWorkerControl = sqliteTable( + "job_worker_control", + { + claimingPaused: integer("claiming_paused", { mode: "boolean" }).notNull(), + id: integer("id").notNull().primaryKey(), + updatedAt: integer("updated_at", { mode: "timestamp_ms" }).notNull(), + updatedById: text("updated_by_id"), + updatedByKind: text("updated_by_kind", { + enum: ["automation", "user"], + }), + version: integer("version").notNull(), + }, + (table) => [ + check( + "job_worker_control_actor_check", + sql`(${table.updatedByKind} IS NULL AND ${table.updatedById} IS NULL) OR (${table.updatedByKind} IS NOT NULL AND ${table.updatedById} IS NOT NULL AND ${jobActorCheck(table.updatedByKind, table.updatedById)})` + ), + check( + "job_worker_control_claiming_paused_check", + sql`${table.claimingPaused} IN (0, 1)` + ), + check("job_worker_control_id_check", sql`${table.id} = 1`), + check( + "job_worker_control_updated_at_check", + timestampMillisecondsCheck(table.updatedAt) + ), + check( + "job_worker_control_version_check", + sql`${table.version} BETWEEN 1 AND 9007199254740991` + ), + ] +); diff --git a/greenfield/src/server/database/schema/resourceLeases.ts b/greenfield/src/server/database/schema/resourceLeases.ts new file mode 100644 index 000000000..ff485d7e3 --- /dev/null +++ b/greenfield/src/server/database/schema/resourceLeases.ts @@ -0,0 +1,44 @@ +import { sql } from "drizzle-orm"; +import { check, index, integer, sqliteTable, text } from "drizzle-orm/sqlite-core"; + +import { timestampMillisecondsCheck, uuidV7TextCheck } from "./checks.ts"; +import { boundedJobKeyCheck } from "./jobChecks.ts"; +import { jobRuns } from "./jobRuns.ts"; +import { workerInstances } from "./workerInstances.ts"; + +/** Cross-run exclusivity lease acquired atomically with one fenced run claim. */ +export const resourceLeases = sqliteTable( + "resource_leases", + { + acquiredAt: integer("acquired_at", { mode: "timestamp_ms" }).notNull(), + expiresAt: integer("expires_at", { mode: "timestamp_ms" }).notNull(), + jobRunId: text("job_run_id") + .notNull() + .references(() => jobRuns.id, { + onDelete: "restrict", + onUpdate: "restrict", + }), + leaseToken: text("lease_token").notNull(), + renewedAt: integer("renewed_at", { mode: "timestamp_ms" }).notNull(), + resourceKey: text("resource_key").notNull().primaryKey(), + workerInstanceId: text("worker_instance_id") + .notNull() + .references(() => workerInstances.id, { + onDelete: "restrict", + onUpdate: "restrict", + }), + }, + (table) => [ + check("resource_leases_lease_token_check", uuidV7TextCheck(table.leaseToken)), + check( + "resource_leases_resource_key_check", + boundedJobKeyCheck(table.resourceKey, 128) + ), + check( + "resource_leases_time_check", + sql`${timestampMillisecondsCheck(table.acquiredAt)} AND ${timestampMillisecondsCheck(table.renewedAt)} AND ${timestampMillisecondsCheck(table.expiresAt)} AND ${table.renewedAt} >= ${table.acquiredAt} AND ${table.expiresAt} > ${table.renewedAt}` + ), + index("resource_leases_expiry_key_idx").on(table.expiresAt, table.resourceKey), + index("resource_leases_run_key_idx").on(table.jobRunId, table.resourceKey), + ] +); diff --git a/greenfield/src/server/database/schema/scheduledJobs.ts b/greenfield/src/server/database/schema/scheduledJobs.ts new file mode 100644 index 000000000..244f51d3d --- /dev/null +++ b/greenfield/src/server/database/schema/scheduledJobs.ts @@ -0,0 +1,128 @@ +import { sql } from "drizzle-orm"; +import { check, index, integer, sqliteTable, text } from "drizzle-orm/sqlite-core"; + +import { canonicalScheduleTimeZones } from "../../../contracts/scheduleTimeZones.ts"; +import { + boundedControlSafeTextCheck, + nulFreeTextCheck, + timestampMillisecondsCheck, +} from "./checks.ts"; +import { + boundedJobKeyCheck, + boundedJsonArrayCheck, + boundedJsonObjectCheck, +} from "./jobChecks.ts"; + +const canonicalScheduleTimeZoneSql = sql.raw( + canonicalScheduleTimeZones + .map((timeZone) => `'${timeZone.replaceAll("'", "''")}'`) + .join(", ") +); + +/** Dashboard-owned recurring job definitions reconciled against the action registry. */ +export const scheduledJobs = sqliteTable( + "scheduled_jobs", + { + actionKey: text("action_key").notNull(), + actionPayloadJson: text("action_payload_json").notNull(), + attemptLimit: integer("attempt_limit").notNull(), + cancellationPolicy: text("cancellation_policy", { + enum: ["cooperative", "never", "queued-only"], + }).notNull(), + createdAt: integer("created_at", { mode: "timestamp_ms" }).notNull(), + cronExpression: text("cron_expression"), + description: text("description").notNull(), + enabled: integer("enabled", { mode: "boolean" }).notNull(), + id: text("id").notNull().primaryKey(), + intervalMs: integer("interval_ms"), + name: text("name").notNull(), + nextRunAt: integer("next_run_at", { mode: "timestamp_ms" }), + priority: integer("priority").notNull(), + resourceClass: text("resource_class", { + enum: ["exclusive", "host-heavy", "interactive", "light", "network"], + }).notNull(), + resourceKeysJson: text("resource_keys_json").notNull(), + retrySafe: integer("retry_safe", { mode: "boolean" }).notNull(), + scheduleKind: text("schedule_kind", { + enum: ["cron", "daily", "interval"], + }).notNull(), + timeOfDay: text("time_of_day"), + timeZone: text("time_zone"), + timeoutMs: integer("timeout_ms").notNull(), + updatedAt: integer("updated_at", { mode: "timestamp_ms" }).notNull(), + version: integer("version").notNull(), + }, + (table) => [ + check( + "scheduled_jobs_action_key_check", + boundedJobKeyCheck(table.actionKey, 128) + ), + check( + "scheduled_jobs_action_payload_json_check", + boundedJsonObjectCheck(table.actionPayloadJson, 65_536) + ), + check( + "scheduled_jobs_attempt_limit_check", + sql`${table.attemptLimit} BETWEEN 1 AND 10` + ), + check( + "scheduled_jobs_cancellation_policy_check", + sql`${table.cancellationPolicy} IN ('cooperative', 'never', 'queued-only')` + ), + check( + "scheduled_jobs_created_at_check", + timestampMillisecondsCheck(table.createdAt) + ), + check( + "scheduled_jobs_description_check", + sql`${boundedControlSafeTextCheck(table.description, 1000)} AND length(CAST(${table.description} AS BLOB)) <= 4000` + ), + check("scheduled_jobs_enabled_check", sql`${table.enabled} IN (0, 1)`), + check("scheduled_jobs_id_check", boundedJobKeyCheck(table.id, 80)), + check( + "scheduled_jobs_name_check", + sql`${boundedControlSafeTextCheck(table.name, 160)} AND length(CAST(${table.name} AS BLOB)) <= 640` + ), + check( + "scheduled_jobs_next_run_check", + sql`(${table.nextRunAt} IS NULL OR ${timestampMillisecondsCheck(table.nextRunAt)}) AND (${table.enabled} = 0 OR ${table.nextRunAt} IS NOT NULL)` + ), + check( + "scheduled_jobs_priority_check", + sql`${table.priority} BETWEEN -100 AND 100` + ), + check( + "scheduled_jobs_resource_class_check", + sql`${table.resourceClass} IN ('exclusive', 'host-heavy', 'interactive', 'light', 'network')` + ), + check( + "scheduled_jobs_resource_keys_json_check", + boundedJsonArrayCheck(table.resourceKeysJson, 4096) + ), + check("scheduled_jobs_retry_safe_check", sql`${table.retrySafe} IN (0, 1)`), + check( + "scheduled_jobs_schedule_shape_check", + sql`(${table.scheduleKind} = 'interval' AND ${table.intervalMs} BETWEEN 60000 AND 31536000000 AND ${table.timeOfDay} IS NULL AND ${table.cronExpression} IS NULL AND ${table.timeZone} IS NULL) OR (${table.scheduleKind} = 'daily' AND ${table.intervalMs} IS NULL AND ${table.timeOfDay} IS NOT NULL AND ${nulFreeTextCheck(table.timeOfDay)} AND ${table.timeOfDay} GLOB '[0-2][0-9]:[0-5][0-9]' AND CAST(substr(${table.timeOfDay}, 1, 2) AS INTEGER) BETWEEN 0 AND 23 AND ${table.cronExpression} IS NULL AND ${table.timeZone} IS NOT NULL) OR (${table.scheduleKind} = 'cron' AND ${table.intervalMs} IS NULL AND ${table.timeOfDay} IS NULL AND ${table.cronExpression} IS NOT NULL AND length(${table.cronExpression}) BETWEEN 9 AND 200 AND ${nulFreeTextCheck(table.cronExpression)} AND ${table.cronExpression} = trim(${table.cronExpression}) AND ${table.cronExpression} NOT LIKE '% %' AND ${table.cronExpression} NOT GLOB '*[^-0-9*,/ ]*' AND length(${table.cronExpression}) - length(replace(${table.cronExpression}, ' ', '')) = 4 AND ${table.timeZone} IS NOT NULL)` + ), + check( + "scheduled_jobs_time_zone_check", + sql`${table.timeZone} IS NULL OR ${table.timeZone} IN (${canonicalScheduleTimeZoneSql})` + ), + check( + "scheduled_jobs_timeout_check", + sql`${table.timeoutMs} BETWEEN 1000 AND 86400000` + ), + check( + "scheduled_jobs_updated_at_check", + sql`${timestampMillisecondsCheck(table.updatedAt)} AND ${table.updatedAt} >= ${table.createdAt}` + ), + check( + "scheduled_jobs_version_check", + sql`${table.version} BETWEEN 1 AND 9007199254740991` + ), + index("scheduled_jobs_due_idx") + .on(table.nextRunAt, table.id) + .where(sql`${table.enabled} = 1`), + index("scheduled_jobs_updated_id_idx").on(table.updatedAt, table.id), + ] +); diff --git a/greenfield/src/server/database/schema/workerInstances.ts b/greenfield/src/server/database/schema/workerInstances.ts new file mode 100644 index 000000000..bfc069049 --- /dev/null +++ b/greenfield/src/server/database/schema/workerInstances.ts @@ -0,0 +1,42 @@ +import { sql } from "drizzle-orm"; +import { check, index, integer, sqliteTable, text } from "drizzle-orm/sqlite-core"; + +import { + lowercaseHexTextCheck, + timestampMillisecondsCheck, + uuidV7TextCheck, +} from "./checks.ts"; + +/** Durable worker registration and heartbeat state shared across rolling releases. */ +export const workerInstances = sqliteTable( + "worker_instances", + { + capacity: integer("capacity").notNull(), + drainingAt: integer("draining_at", { mode: "timestamp_ms" }), + heartbeatAt: integer("heartbeat_at", { mode: "timestamp_ms" }).notNull(), + id: text("id").notNull().primaryKey(), + pid: integer("pid").notNull(), + releaseId: text("release_id").notNull(), + startedAt: integer("started_at", { mode: "timestamp_ms" }).notNull(), + state: text("state", { enum: ["draining", "online", "stopped"] }).notNull(), + stoppedAt: integer("stopped_at", { mode: "timestamp_ms" }), + }, + (table) => [ + check("worker_instances_capacity_check", sql`${table.capacity} BETWEEN 1 AND 16`), + check("worker_instances_id_check", uuidV7TextCheck(table.id)), + check("worker_instances_pid_check", sql`${table.pid} BETWEEN 1 AND 2147483647`), + check( + "worker_instances_release_id_check", + lowercaseHexTextCheck(table.releaseId, 40) + ), + check( + "worker_instances_state_check", + sql`(${table.state} = 'online' AND ${table.drainingAt} IS NULL AND ${table.stoppedAt} IS NULL) OR (${table.state} = 'draining' AND ${table.drainingAt} IS NOT NULL AND ${table.stoppedAt} IS NULL) OR (${table.state} = 'stopped' AND ${table.drainingAt} IS NOT NULL AND ${table.stoppedAt} IS NOT NULL)` + ), + check( + "worker_instances_time_check", + sql`${timestampMillisecondsCheck(table.startedAt)} AND ${timestampMillisecondsCheck(table.heartbeatAt)} AND ${table.heartbeatAt} >= ${table.startedAt} AND (${table.drainingAt} IS NULL OR (${timestampMillisecondsCheck(table.drainingAt)} AND ${table.drainingAt} >= ${table.startedAt})) AND (${table.stoppedAt} IS NULL OR (${timestampMillisecondsCheck(table.stoppedAt)} AND ${table.stoppedAt} >= ${table.drainingAt}))` + ), + index("worker_instances_heartbeat_id_idx").on(table.heartbeatAt, table.id), + ] +); diff --git a/greenfield/src/server/database/validation/jobActors.ts b/greenfield/src/server/database/validation/jobActors.ts new file mode 100644 index 000000000..fb2ee6f76 --- /dev/null +++ b/greenfield/src/server/database/validation/jobActors.ts @@ -0,0 +1,26 @@ +import * as v from "valibot"; + +import { jobActionKeySchema } from "../../../contracts/jobModel.ts"; +import { + automationPrincipalIdSchema, + securityRecordIdSchema, +} from "../../../contracts/security.ts"; + +/** + * Validates one persisted durable-job actor identity against its principal kind. + * @param kind Durable actor principal kind. + * @param id Persisted actor identifier. + * @returns Whether the identifier is canonical for that principal kind. + */ +export function jobActorIdentityIsValid( + kind: "automation" | "system" | "user", + id: string +): boolean { + if (kind === "automation") { + return v.safeParse(automationPrincipalIdSchema, id).success; + } + if (kind === "user") return v.safeParse(securityRecordIdSchema, id).success; + // System actor ids intentionally share the bounded lowercase action-key shape. + if (kind === "system") return v.safeParse(jobActionKeySchema, id).success; + return false; +} diff --git a/greenfield/src/server/database/validation/jobDisableIntents.ts b/greenfield/src/server/database/validation/jobDisableIntents.ts new file mode 100644 index 000000000..601aaa148 --- /dev/null +++ b/greenfield/src/server/database/validation/jobDisableIntents.ts @@ -0,0 +1,147 @@ +import { createInsertSchema, createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { jobDescriptionSchema, scheduleIdSchema } from "../../../contracts/jobModel.ts"; +import { boundedNonBlankTextSchema } from "../../../shared/validation.ts"; +import { jobDisableIntents } from "../schema/jobDisableIntents.ts"; +import { jobActorIdentityIsValid } from "./jobActors.ts"; +import { nonnegativeDateSchema, uuidV7TextSchema } from "./scalars.ts"; + +const createdActorKindSchema = v.picklist(["automation", "user"]); +const endedActorKindSchema = v.picklist(["automation", "system", "user"]); +const endedReasonSchema = v.picklist(["expired", "re-enabled", "replaced"]); +const externalJobIdSchema = boundedNonBlankTextSchema(256, "External job id is invalid"); + +interface StoredDisableIntent { + readonly createdAt: Date; + readonly createdById: string; + readonly createdByKind: "automation" | "user"; + readonly endedAt?: Date | null; + readonly endedById?: string | null; + readonly endedByKind?: "automation" | "system" | "user" | null; + readonly endedReason?: "expired" | "re-enabled" | "replaced" | null; + readonly expiresAt?: Date | null; + readonly externalJobId?: string | null; + readonly externalProvider?: "openclaw" | null; + readonly scheduledJobId?: string | null; + readonly targetKind: "dashboard-schedule" | "openclaw-cron"; +} + +function disableIntentIsConsistent(intent: StoredDisableIntent): boolean { + if (!jobActorIdentityIsValid(intent.createdByKind, intent.createdById)) return false; + + const expiresAt = intent.expiresAt ?? null; + if (expiresAt !== null && expiresAt.getTime() <= intent.createdAt.getTime()) { + return false; + } + + const targetsDashboard = intent.targetKind === "dashboard-schedule"; + if ( + targetsDashboard !== (intent.scheduledJobId != null) || + targetsDashboard === (intent.externalProvider != null) || + targetsDashboard === (intent.externalJobId != null) + ) { + return false; + } + + const endedAt = intent.endedAt ?? null; + const endedById = intent.endedById ?? null; + const endedByKind = intent.endedByKind ?? null; + const endedReason = intent.endedReason ?? null; + const isOpen = endedAt === null; + if ( + isOpen !== (endedById === null) || + isOpen !== (endedByKind === null) || + isOpen !== (endedReason === null) + ) { + return false; + } + if (isOpen) return true; + if ( + endedById === null || + endedByKind === null || + endedReason === null || + !jobActorIdentityIsValid(endedByKind, endedById) || + endedAt.getTime() < intent.createdAt.getTime() + ) { + return false; + } + return ( + endedReason !== "expired" || + (endedByKind === "system" && + expiresAt !== null && + endedAt.getTime() >= expiresAt.getTime()) + ); +} + +const disableIntentRefinements = { + createdAt: nonnegativeDateSchema, + createdById: () => v.string(), + createdByKind: () => createdActorKindSchema, + endedAt: nonnegativeDateSchema, + endedById: () => v.nullable(v.string()), + endedByKind: () => v.nullable(endedActorKindSchema), + endedReason: () => v.nullable(endedReasonSchema), + expiresAt: nonnegativeDateSchema, + externalJobId: () => v.nullable(externalJobIdSchema), + id: uuidV7TextSchema, + reason: () => jobDescriptionSchema, + scheduledJobId: () => v.nullable(scheduleIdSchema), +}; + +const generatedDisableIntentSelectSchema = createSelectSchema( + jobDisableIntents, + disableIntentRefinements +); +const disableIntentSelectObjectSchema = v.strictObject( + generatedDisableIntentSelectSchema.entries +); + +/** Validates one append-only disable-intent row read from SQLite. */ +export const jobDisableIntentSelectSchema = v.pipe( + disableIntentSelectObjectSchema, + v.check( + (intent) => disableIntentIsConsistent(intent), + "Stored job disable intent is inconsistent" + ) +); + +const generatedDisableIntentInsertSchema = createInsertSchema( + jobDisableIntents, + disableIntentRefinements +); +const disableIntentInsertObjectSchema = v.strictObject( + generatedDisableIntentInsertSchema.entries +); + +/** Validates one open disable intent before insertion. */ +export const jobDisableIntentInsertSchema = v.pipe( + disableIntentInsertObjectSchema, + v.check( + (intent) => + intent.endedAt == null && + intent.endedById == null && + intent.endedByKind == null && + intent.endedReason == null && + disableIntentIsConsistent(intent), + "New job disable intent must be open and consistent" + ) +); + +const jobDisableIntentCloseObjectSchema = v.strictObject({ + endedAt: nonnegativeDateSchema(v.date()), + endedById: v.string("Job disable-intent closure actor is invalid"), + endedByKind: endedActorKindSchema, + endedReason: endedReasonSchema, +}); + +/** Validates the complete closure projection for an existing disable intent. */ +export const jobDisableIntentCloseSchema = v.pipe( + jobDisableIntentCloseObjectSchema, + v.check( + (closure) => + jobActorIdentityIsValid(closure.endedByKind, closure.endedById) && + (closure.endedReason !== "expired" || closure.endedByKind === "system"), + "Job disable-intent closure actor is invalid" + ) +); diff --git a/greenfield/src/server/database/validation/jobRunEvents.ts b/greenfield/src/server/database/validation/jobRunEvents.ts new file mode 100644 index 000000000..b660028d1 --- /dev/null +++ b/greenfield/src/server/database/validation/jobRunEvents.ts @@ -0,0 +1,142 @@ +import { createInsertSchema, createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { + jobAttemptCountSchema, + jobRunEventKindSchema, + jobRunEventMessageFitsBudget, + jobRunEventMessageMaximumLength, + jobRunEventProgressMaximumBytes, + jobRunEventProgressSchema, + jobRunEventSequenceSchema, +} from "../../../contracts/jobModel.ts"; +import { utf8ByteLength } from "../../../shared/encoding.ts"; +import { parseJsonText } from "../../../shared/json.ts"; +import { boundedControlSafeTextSchema } from "../../../shared/validation.ts"; +import { jobRunEvents } from "../schema/jobRunEvents.ts"; +import { nonnegativeDateSchema, uuidV7TextSchema } from "./scalars.ts"; + +const eventMessageSchema = v.pipe( + boundedControlSafeTextSchema( + jobRunEventMessageMaximumLength, + "Stored job event message is invalid" + ), + v.check(jobRunEventMessageFitsBudget, "Stored job event message is invalid") +); +const eventProgressJsonSchema = v.pipe( + v.string("Stored job event progress is invalid"), + v.check( + (value) => utf8ByteLength(value) <= jobRunEventProgressMaximumBytes, + "Stored job event progress is outside its byte budget" + ), + v.check((value) => { + return v.safeParse(jobRunEventProgressSchema, parseJsonText(value)).success; + }, "Stored job event progress must contain a bounded JSON object") +); + +interface StoredJobRunEvent { + readonly attempt: number; + readonly kind: + | "cancel-requested" + | "cancelled" + | "claimed" + | "failed" + | "lease-expired" + | "output-truncated" + | "progress" + | "queued" + | "retry-scheduled" + | "stderr" + | "stdout" + | "succeeded" + | "timed-out"; + readonly message?: string | null; + readonly progressJson?: string | null; + readonly workerInstanceId?: string | null; +} + +function eventPayloadIsConsistent(event: StoredJobRunEvent): boolean { + const message = event.message ?? null; + const progressJson = event.progressJson ?? null; + if (event.kind === "progress") return progressJson !== null; + if (event.kind === "stderr" || event.kind === "stdout") { + return message !== null && progressJson === null; + } + return progressJson === null; +} + +function eventAttemptIsConsistent(event: StoredJobRunEvent): boolean { + if (event.kind === "queued") { + return event.attempt === 0 && event.workerInstanceId == null; + } + if ( + [ + "claimed", + "failed", + "lease-expired", + "output-truncated", + "progress", + "retry-scheduled", + "stderr", + "stdout", + "succeeded", + "timed-out", + ].includes(event.kind) + ) { + return event.attempt > 0; + } + return true; +} + +const eventRefinements = { + attempt: () => jobAttemptCountSchema, + jobRunId: uuidV7TextSchema, + kind: () => jobRunEventKindSchema, + message: () => v.nullable(eventMessageSchema), + occurredAt: nonnegativeDateSchema, + progressJson: () => v.nullable(eventProgressJsonSchema), + sequence: () => jobRunEventSequenceSchema, + workerInstanceId: uuidV7TextSchema, +}; + +const generatedJobRunEventSelectSchema = createSelectSchema( + jobRunEvents, + eventRefinements +); +const jobRunEventSelectObjectSchema = v.strictObject( + generatedJobRunEventSelectSchema.entries +); + +/** Validates one immutable durable job event read from SQLite. */ +export const jobRunEventSelectSchema = v.pipe( + jobRunEventSelectObjectSchema, + v.check( + (event) => eventPayloadIsConsistent(event), + "Stored job event payload is inconsistent" + ), + v.check( + (event) => eventAttemptIsConsistent(event), + "Stored job event attempt is inconsistent" + ) +); + +const generatedJobRunEventInsertSchema = createInsertSchema( + jobRunEvents, + eventRefinements +); +const jobRunEventInsertObjectSchema = v.strictObject( + generatedJobRunEventInsertSchema.entries +); + +/** Validates one immutable durable job event before insertion. */ +export const jobRunEventInsertSchema = v.pipe( + jobRunEventInsertObjectSchema, + v.check( + (event) => eventPayloadIsConsistent(event), + "Stored job event payload is inconsistent" + ), + v.check( + (event) => eventAttemptIsConsistent(event), + "Stored job event attempt is inconsistent" + ) +); diff --git a/greenfield/src/server/database/validation/jobRuns.ts b/greenfield/src/server/database/validation/jobRuns.ts new file mode 100644 index 000000000..50cbe43b7 --- /dev/null +++ b/greenfield/src/server/database/validation/jobRuns.ts @@ -0,0 +1,342 @@ +import { createInsertSchema, createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { + jobActionKeySchema, + jobAttemptCountSchema, + jobAttemptLimitSchema, + jobCancellationPolicySchema, + jobDisplayNameSchema, + jobIdempotencyKeySchema, + jobPayloadMaximumBytes, + jobPayloadSchema, + jobPrioritySchema, + jobResourceClassSchema, + jobResourceKeysMaximumBytes, + jobResourceKeysSchema, + jobRunEventMaximum, + jobRunOutputMaximumBytes, + jobRunPayloadEventMaximum, + jobRunResultMaximumBytes, + jobRunResultSchema, + jobRunStateSchema, + jobRunTerminalCodeSchema, + jobRunTerminalMessageSchema, + jobTimeoutSchema, + jobTriggerTypeSchema, + jobVersionSchema, + scheduleIdSchema, +} from "../../../contracts/jobModel.ts"; +import { utf8ByteLength } from "../../../shared/encoding.ts"; +import { parseJsonText } from "../../../shared/json.ts"; +import { nonnegativeSafeIntegerSchema } from "../../../shared/validation.ts"; +import { jobRuns } from "../schema/jobRuns.ts"; +import { jobActorIdentityIsValid } from "./jobActors.ts"; +import { nonnegativeDateSchema, uuidV7TextSchema } from "./scalars.ts"; +import { sha256TextSchema } from "./securityScalars.ts"; + +const actorKindSchema = v.picklist(["automation", "system", "user"]); + +function jsonObjectTextSchema( + maximumBytes: number, + objectSchema: typeof jobPayloadSchema | typeof jobRunResultSchema, + message: string +) { + return v.pipe( + v.string(message), + v.check((value) => utf8ByteLength(value) <= maximumBytes, message), + v.check( + (value) => v.safeParse(objectSchema, parseJsonText(value)).success, + message + ) + ); +} + +const payloadJsonSchema = jsonObjectTextSchema( + jobPayloadMaximumBytes, + jobPayloadSchema, + "Stored job payload is invalid" +); +const resultJsonSchema = jsonObjectTextSchema( + jobRunResultMaximumBytes, + jobRunResultSchema, + "Stored job result is invalid" +); +const resourceKeysJsonSchema = v.pipe( + v.string("Stored job resource keys are invalid"), + v.check( + (value) => utf8ByteLength(value) <= jobResourceKeysMaximumBytes, + "Stored job resource keys are outside their byte budget" + ), + v.check( + (value) => v.safeParse(jobResourceKeysSchema, parseJsonText(value)).success, + "Stored job resource keys are not canonical" + ) +); +const eventCountSchema = v.pipe( + nonnegativeSafeIntegerSchema("Stored job event count is invalid"), + v.maxValue(jobRunEventMaximum, "Stored job event count is invalid") +); +const payloadEventCountSchema = v.pipe( + nonnegativeSafeIntegerSchema("Stored job payload-event count is invalid"), + v.maxValue(jobRunPayloadEventMaximum, "Stored job payload-event count is invalid") +); +const eventBytesSchema = v.pipe( + nonnegativeSafeIntegerSchema("Stored job event-byte count is invalid"), + v.maxValue(jobRunOutputMaximumBytes, "Stored job event-byte count is invalid") +); + +interface StoredJobRun { + readonly attemptCount: number; + readonly attemptLimit: number; + readonly availableAt: Date; + readonly cancellationPolicy: "cooperative" | "never" | "queued-only"; + readonly cancelRequestedAt: Date | null; + readonly cancelRequestedById: string | null; + readonly cancelRequestedByKind: "automation" | "system" | "user" | null; + readonly eventBytes: number; + readonly eventCount: number; + readonly finishedAt: Date | null; + readonly firstStartedAt: Date | null; + readonly heartbeatAt: Date | null; + readonly lastAttemptStartedAt: Date | null; + readonly leaseExpiresAt: Date | null; + readonly leaseOwnerId: string | null; + readonly leaseToken: string | null; + readonly payloadEventCount: number; + readonly queuedAt: Date; + readonly requestedById: string; + readonly requestedByKind: "automation" | "system" | "user"; + readonly resultJson: string | null; + readonly scheduledForAt: Date | null; + readonly scheduledJobId: string | null; + readonly scheduledJobVersion: number | null; + readonly state: + | "cancelled" + | "failed" + | "queued" + | "running" + | "succeeded" + | "timed-out"; + readonly terminalCode: string | null; + readonly terminalMessage: string | null; + readonly triggerType: "manual" | "schedule" | "startup" | "system"; + readonly updatedAt: Date; +} + +function scheduleProvenanceIsConsistent(run: StoredJobRun): boolean { + const hasSchedule = run.scheduledJobId !== null; + if (hasSchedule !== (run.scheduledJobVersion !== null)) return false; + if ( + (run.triggerType === "manual" || run.triggerType === "schedule") !== hasSchedule + ) { + return false; + } + return ( + (run.triggerType === "schedule") === (run.scheduledForAt !== null) && + (run.scheduledForAt === null || + run.scheduledForAt.getTime() <= run.queuedAt.getTime()) + ); +} + +function attemptsAreConsistent(run: StoredJobRun): boolean { + if (run.attemptCount > run.attemptLimit) return false; + const hasStarted = run.firstStartedAt !== null; + if ( + hasStarted !== (run.lastAttemptStartedAt !== null) || + hasStarted !== run.attemptCount > 0 + ) { + return false; + } + return ( + !["failed", "running", "succeeded", "timed-out"].includes(run.state) || + run.attemptCount > 0 + ); +} + +function leaseIsConsistent(run: StoredJobRun): boolean { + const hasLease = run.leaseOwnerId !== null; + if ( + hasLease !== (run.leaseToken !== null) || + hasLease !== (run.leaseExpiresAt !== null) || + hasLease !== (run.heartbeatAt !== null) || + hasLease !== (run.state === "running") + ) { + return false; + } + if (!hasLease) return true; + return ( + run.lastAttemptStartedAt !== null && + run.heartbeatAt !== null && + run.leaseExpiresAt !== null && + run.heartbeatAt.getTime() >= run.lastAttemptStartedAt.getTime() && + run.leaseExpiresAt.getTime() > run.heartbeatAt.getTime() + ); +} + +function cancellationIsConsistent(run: StoredJobRun): boolean { + const requested = run.cancelRequestedAt !== null; + if ( + requested !== (run.cancelRequestedById !== null) || + requested !== (run.cancelRequestedByKind !== null) || + (requested && run.cancellationPolicy === "never") || + (run.state === "cancelled" && !requested) + ) { + return false; + } + return ( + !requested || + (run.cancelRequestedById !== null && + run.cancelRequestedByKind !== null && + jobActorIdentityIsValid(run.cancelRequestedByKind, run.cancelRequestedById)) + ); +} + +function terminalStateIsConsistent(run: StoredJobRun): boolean { + const hasFinished = run.finishedAt !== null; + const hasResult = run.resultJson !== null; + const hasTerminalCode = run.terminalCode !== null; + const hasTerminalMessage = run.terminalMessage !== null; + if (hasTerminalCode !== hasTerminalMessage) return false; + if (run.state === "succeeded") { + return hasFinished && hasResult && !hasTerminalCode; + } + if (["cancelled", "failed", "timed-out"].includes(run.state)) { + return hasFinished && !hasResult && hasTerminalCode; + } + return !hasFinished && !hasResult && !hasTerminalCode; +} + +function timestampsAreConsistent(run: StoredJobRun): boolean { + const queuedAt = run.queuedAt.getTime(); + const updatedAt = run.updatedAt.getTime(); + if (run.availableAt.getTime() < queuedAt || updatedAt < queuedAt) return false; + + const durableTransitionTimes = [ + run.firstStartedAt, + run.lastAttemptStartedAt, + run.heartbeatAt, + run.cancelRequestedAt, + run.finishedAt, + ].filter((date): date is Date => date !== null); + if ( + durableTransitionTimes.some( + (timestamp) => + timestamp.getTime() < queuedAt || timestamp.getTime() > updatedAt + ) + ) { + return false; + } + return ( + (run.firstStartedAt === null || + run.lastAttemptStartedAt === null || + run.lastAttemptStartedAt.getTime() >= run.firstStartedAt.getTime()) && + (run.finishedAt === null || + run.lastAttemptStartedAt === null || + run.finishedAt.getTime() >= run.lastAttemptStartedAt.getTime()) + ); +} + +function jobRunIsConsistent(run: StoredJobRun): boolean { + return ( + jobActorIdentityIsValid(run.requestedByKind, run.requestedById) && + run.payloadEventCount <= run.eventCount && + scheduleProvenanceIsConsistent(run) && + attemptsAreConsistent(run) && + leaseIsConsistent(run) && + cancellationIsConsistent(run) && + terminalStateIsConsistent(run) && + timestampsAreConsistent(run) + ); +} + +const jobRunRefinements = { + actionKey: () => jobActionKeySchema, + attemptCount: () => jobAttemptCountSchema, + attemptLimit: () => jobAttemptLimitSchema, + availableAt: nonnegativeDateSchema, + cancellationPolicy: () => jobCancellationPolicySchema, + cancelRequestedAt: nonnegativeDateSchema, + cancelRequestedById: () => v.nullable(v.string()), + cancelRequestedByKind: () => v.nullable(actorKindSchema), + displayName: () => jobDisplayNameSchema, + enqueueSha256: sha256TextSchema, + eventBytes: () => eventBytesSchema, + eventCount: () => eventCountSchema, + finishedAt: nonnegativeDateSchema, + firstStartedAt: nonnegativeDateSchema, + heartbeatAt: nonnegativeDateSchema, + id: uuidV7TextSchema, + idempotencyKey: () => jobIdempotencyKeySchema, + lastAttemptStartedAt: nonnegativeDateSchema, + leaseExpiresAt: nonnegativeDateSchema, + leaseOwnerId: uuidV7TextSchema, + leaseToken: uuidV7TextSchema, + payloadEventCount: () => payloadEventCountSchema, + payloadJson: () => payloadJsonSchema, + priority: () => jobPrioritySchema, + queuedAt: nonnegativeDateSchema, + requestedById: () => v.string(), + requestedByKind: () => actorKindSchema, + resourceClass: () => jobResourceClassSchema, + resourceKeysJson: () => resourceKeysJsonSchema, + resultJson: () => v.nullable(resultJsonSchema), + scheduledForAt: nonnegativeDateSchema, + scheduledJobId: () => v.nullable(scheduleIdSchema), + scheduledJobVersion: () => v.nullable(jobVersionSchema), + state: () => jobRunStateSchema, + stateVersion: () => jobVersionSchema, + terminalCode: () => v.nullable(jobRunTerminalCodeSchema), + terminalMessage: () => v.nullable(jobRunTerminalMessageSchema), + timeoutMs: () => jobTimeoutSchema, + triggerType: () => jobTriggerTypeSchema, + updatedAt: nonnegativeDateSchema, +}; + +const generatedJobRunSelectSchema = createSelectSchema(jobRuns, jobRunRefinements); +const jobRunSelectObjectSchema = v.strictObject(generatedJobRunSelectSchema.entries); + +/** Validates one complete durable job-run row read from SQLite. */ +export const jobRunSelectSchema = v.pipe( + jobRunSelectObjectSchema, + v.check((run) => jobRunIsConsistent(run), "Stored job run is inconsistent") +); + +const generatedJobRunInsertSchema = v.omit( + createInsertSchema(jobRuns, jobRunRefinements), + ["attemptCount", "eventBytes", "eventCount", "payloadEventCount", "stateVersion"] +); +const jobRunInsertObjectSchema = v.strictObject(generatedJobRunInsertSchema.entries); + +/** Validates the initial queued projection before inserting a durable run. */ +export const jobRunInsertSchema = v.pipe( + jobRunInsertObjectSchema, + v.check( + (run) => + run.state === "queued" && + run.cancelRequestedAt == null && + run.cancelRequestedById == null && + run.cancelRequestedByKind == null && + run.finishedAt == null && + run.firstStartedAt == null && + run.heartbeatAt == null && + run.lastAttemptStartedAt == null && + run.leaseExpiresAt == null && + run.leaseOwnerId == null && + run.leaseToken == null && + run.resultJson == null && + run.terminalCode == null && + run.terminalMessage == null && + jobActorIdentityIsValid(run.requestedByKind, run.requestedById) && + run.availableAt.getTime() >= run.queuedAt.getTime() && + run.updatedAt.getTime() >= run.queuedAt.getTime() && + scheduleProvenanceIsConsistent({ + ...run, + attemptCount: 0, + eventBytes: 0, + eventCount: 0, + payloadEventCount: 0, + }), + "New job run must be an internally consistent queued row" + ) +); diff --git a/greenfield/src/server/database/validation/jobWorkerControl.ts b/greenfield/src/server/database/validation/jobWorkerControl.ts new file mode 100644 index 000000000..46cad72e5 --- /dev/null +++ b/greenfield/src/server/database/validation/jobWorkerControl.ts @@ -0,0 +1,76 @@ +import { createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { jobVersionSchema } from "../../../contracts/jobModel.ts"; +import { jobWorkerControl } from "../schema/jobWorkerControl.ts"; +import { jobActorIdentityIsValid } from "./jobActors.ts"; +import { nonnegativeDateSchema } from "./scalars.ts"; + +const controlActorKindSchema = v.picklist(["automation", "user"]); + +interface StoredWorkerControl { + readonly claimingPaused: boolean; + readonly id: number; + readonly updatedAt: Date; + readonly updatedById: string | null; + readonly updatedByKind: "automation" | "user" | null; + readonly version: number; +} + +function workerControlIsConsistent(control: StoredWorkerControl): boolean { + if (control.id !== 1) return false; + const hasActor = control.updatedByKind !== null; + if (hasActor !== (control.updatedById !== null)) return false; + if (!hasActor) { + return ( + control.version === 1 && + !control.claimingPaused && + control.updatedAt.getTime() === 0 + ); + } + return ( + control.version > 1 && + control.updatedByKind !== null && + control.updatedById !== null && + jobActorIdentityIsValid(control.updatedByKind, control.updatedById) + ); +} + +const controlRefinements = { + updatedAt: nonnegativeDateSchema, + updatedById: () => v.nullable(v.string()), + updatedByKind: () => v.nullable(controlActorKindSchema), + version: () => jobVersionSchema, +}; +const generatedWorkerControlSelectSchema = createSelectSchema( + jobWorkerControl, + controlRefinements +); +const workerControlSelectObjectSchema = v.strictObject( + generatedWorkerControlSelectSchema.entries +); + +/** Validates the required singleton worker-control row read from SQLite. */ +export const jobWorkerControlSelectSchema = v.pipe( + workerControlSelectObjectSchema, + v.check(workerControlIsConsistent, "Stored worker control is inconsistent") +); + +const workerControlUpdateObjectSchema = v.strictObject({ + claimingPaused: v.boolean("Worker claiming state is invalid"), + updatedAt: nonnegativeDateSchema(v.date("Worker control timestamp is invalid")), + updatedById: v.string("Worker control actor id is invalid"), + updatedByKind: controlActorKindSchema, + version: jobVersionSchema, +}); + +/** Validates one complete versioned worker-control mutation. */ +export const jobWorkerControlUpdateSchema = v.pipe( + workerControlUpdateObjectSchema, + v.check( + (control) => + control.version > 1 && + jobActorIdentityIsValid(control.updatedByKind, control.updatedById), + "Worker control update actor is invalid" + ) +); diff --git a/greenfield/src/server/database/validation/resourceLeases.ts b/greenfield/src/server/database/validation/resourceLeases.ts new file mode 100644 index 000000000..41eaf227e --- /dev/null +++ b/greenfield/src/server/database/validation/resourceLeases.ts @@ -0,0 +1,63 @@ +import { createInsertSchema, createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { jobResourceKeySchema } from "../../../contracts/jobModel.ts"; +import { resourceLeases } from "../schema/resourceLeases.ts"; +import { nonnegativeDateSchema, uuidV7TextSchema } from "./scalars.ts"; + +interface StoredResourceLease { + readonly acquiredAt: Date; + readonly expiresAt: Date; + readonly renewedAt: Date; +} + +function resourceLeaseTimesAreConsistent(lease: StoredResourceLease): boolean { + return ( + lease.renewedAt.getTime() >= lease.acquiredAt.getTime() && + lease.expiresAt.getTime() > lease.renewedAt.getTime() + ); +} + +const resourceLeaseRefinements = { + acquiredAt: nonnegativeDateSchema, + expiresAt: nonnegativeDateSchema, + jobRunId: uuidV7TextSchema, + leaseToken: uuidV7TextSchema, + renewedAt: nonnegativeDateSchema, + resourceKey: () => jobResourceKeySchema, + workerInstanceId: uuidV7TextSchema, +}; + +const generatedResourceLeaseSelectSchema = createSelectSchema( + resourceLeases, + resourceLeaseRefinements +); +const resourceLeaseSelectObjectSchema = v.strictObject( + generatedResourceLeaseSelectSchema.entries +); + +/** Validates one fenced resource lease read from SQLite. */ +export const resourceLeaseSelectSchema = v.pipe( + resourceLeaseSelectObjectSchema, + v.check( + (lease) => resourceLeaseTimesAreConsistent(lease), + "Stored resource lease timestamps are inconsistent" + ) +); + +const generatedResourceLeaseInsertSchema = createInsertSchema( + resourceLeases, + resourceLeaseRefinements +); +const resourceLeaseInsertObjectSchema = v.strictObject( + generatedResourceLeaseInsertSchema.entries +); + +/** Validates one fenced resource lease before atomic acquisition. */ +export const resourceLeaseInsertSchema = v.pipe( + resourceLeaseInsertObjectSchema, + v.check( + (lease) => resourceLeaseTimesAreConsistent(lease), + "Stored resource lease timestamps are inconsistent" + ) +); diff --git a/greenfield/src/server/database/validation/rowSchemas.test.ts b/greenfield/src/server/database/validation/rowSchemas.test.ts index 1f87ba8d9..111dcee65 100644 --- a/greenfield/src/server/database/validation/rowSchemas.test.ts +++ b/greenfield/src/server/database/validation/rowSchemas.test.ts @@ -8,6 +8,17 @@ import { incidentSelectSchema, incidentUpdateSchema, } from "./incidents.ts"; +import { + jobDisableIntentCloseSchema, + jobDisableIntentInsertSchema, + jobDisableIntentSelectSchema, +} from "./jobDisableIntents.ts"; +import { jobRunEventInsertSchema, jobRunEventSelectSchema } from "./jobRunEvents.ts"; +import { jobRunInsertSchema, jobRunSelectSchema } from "./jobRuns.ts"; +import { + jobWorkerControlSelectSchema, + jobWorkerControlUpdateSchema, +} from "./jobWorkerControl.ts"; import { monitorRunInsertSchema, monitorRunUpdateSchema } from "./monitorRuns.ts"; import { notificationInsertSchema, notificationUpdateSchema } from "./notifications.ts"; import { @@ -17,6 +28,11 @@ import { realtimeEventSelectSchema, } from "./realtimeEvents.ts"; import { reportInsertSchema } from "./reports.ts"; +import { + resourceLeaseInsertSchema, + resourceLeaseSelectSchema, +} from "./resourceLeases.ts"; +import { scheduledJobInsertSchema, scheduledJobSelectSchema } from "./scheduledJobs.ts"; import { schemaMigrationInsertSchema } from "./schemaMigrations.ts"; import { incidentFingerprint, @@ -31,6 +47,103 @@ import { validObservationValues, validRealtimeEventValues, } from "./testSupport/rows.ts"; +import { + workerInstanceInsertSchema, + workerInstanceSelectSchema, +} from "./workerInstances.ts"; + +const jobUserId = "019fc968-1a9b-7764-bf1b-d5b863b0e7b4"; +const jobRunId = "019fc968-1a9b-7765-8f1b-d5b863b0e7b4"; +const jobEventRunId = "019fc968-1a9b-7766-9f1b-d5b863b0e7b4"; +const jobWorkerId = "019fc968-1a9b-7767-af1b-d5b863b0e7b4"; +const jobLeaseToken = "019fc968-1a9b-7768-bf1b-d5b863b0e7b4"; +const jobDisableIntentId = "019fc968-1a9b-7769-8f1b-d5b863b0e7b4"; +const jobScheduleId = "system.worker-smoke"; +const jobCreatedAt = new Date(1000); +const jobUpdatedAt = new Date(2000); +const jobNextRunAt = new Date(61_000); + +const validScheduledJobRow = Object.freeze({ + actionKey: "system.worker-smoke", + actionPayloadJson: "{}", + attemptLimit: 2, + cancellationPolicy: "cooperative" as const, + createdAt: jobCreatedAt, + cronExpression: null, + description: "Verifies the worker runtime without external side effects.", + enabled: true, + id: jobScheduleId, + intervalMs: 60_000, + name: "Worker smoke check", + nextRunAt: jobNextRunAt, + priority: 0, + resourceClass: "light" as const, + resourceKeysJson: '["database"]', + retrySafe: true, + scheduleKind: "interval" as const, + timeOfDay: null, + timeZone: null, + timeoutMs: 30_000, + updatedAt: jobUpdatedAt, + version: 1, +}); + +const validJobRunRow = Object.freeze({ + actionKey: "system.worker-smoke", + attemptCount: 0, + attemptLimit: 2, + availableAt: jobUpdatedAt, + cancellationPolicy: "cooperative" as const, + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: "Worker smoke check", + enqueueSha256: "a".repeat(64), + eventBytes: 0, + eventCount: 0, + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id: jobRunId, + idempotencyKey: "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcY", + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadEventCount: 0, + payloadJson: "{}", + priority: 0, + queuedAt: jobUpdatedAt, + requestedById: jobUserId, + requestedByKind: "user" as const, + resourceClass: "light" as const, + resourceKeysJson: '["database"]', + resultJson: null, + retrySafe: true, + scheduledForAt: null, + scheduledJobId: jobScheduleId, + scheduledJobVersion: 1, + state: "queued" as const, + stateVersion: 1, + terminalCode: null, + terminalMessage: null, + timeoutMs: 30_000, + triggerType: "manual" as const, + updatedAt: jobUpdatedAt, +}); + +const generatedJobRunInsertFields = new Set([ + "attemptCount", + "eventBytes", + "eventCount", + "payloadEventCount", + "stateVersion", +]); +const validJobRunInsert = Object.fromEntries( + Object.entries(validJobRunRow).filter( + ([key]) => !generatedJobRunInsertFields.has(key) + ) +); describe("Drizzle-generated Valibot row schemas", () => { test("validate every foundation table at its database boundary", () => { @@ -329,4 +442,258 @@ describe("Drizzle-generated Valibot row schemas", () => { }) ).toThrow(); }); + + test("validates all seven durable job table boundaries", () => { + expect(v.parse(scheduledJobInsertSchema, validScheduledJobRow)).toBeDefined(); + expect(v.parse(scheduledJobSelectSchema, validScheduledJobRow)).toBeDefined(); + + const disableIntent = { + createdAt: jobCreatedAt, + createdById: jobUserId, + createdByKind: "user" as const, + endedAt: null, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: jobNextRunAt, + externalJobId: null, + externalProvider: null, + id: jobDisableIntentId, + reason: "Paused during maintenance.", + scheduledJobId: jobScheduleId, + targetKind: "dashboard-schedule" as const, + }; + expect(v.parse(jobDisableIntentInsertSchema, disableIntent)).toBeDefined(); + expect(v.parse(jobDisableIntentSelectSchema, disableIntent)).toBeDefined(); + expect( + v.parse(jobDisableIntentCloseSchema, { + endedAt: jobUpdatedAt, + endedById: jobUserId, + endedByKind: "user", + endedReason: "re-enabled", + }) + ).toBeDefined(); + + expect(v.parse(jobRunInsertSchema, validJobRunInsert)).toBeDefined(); + expect(v.parse(jobRunSelectSchema, validJobRunRow)).toBeDefined(); + + const jobEvent = { + attempt: 0, + jobRunId: jobEventRunId, + kind: "queued" as const, + message: "Queued for execution.", + occurredAt: jobUpdatedAt, + progressJson: null, + sequence: 1, + workerInstanceId: null, + }; + expect(v.parse(jobRunEventInsertSchema, jobEvent)).toBeDefined(); + expect(v.parse(jobRunEventSelectSchema, jobEvent)).toBeDefined(); + + const worker = { + capacity: 2, + drainingAt: null, + heartbeatAt: jobUpdatedAt, + id: jobWorkerId, + pid: 1234, + releaseId: "b".repeat(40), + startedAt: jobCreatedAt, + state: "online" as const, + stoppedAt: null, + }; + expect(v.parse(workerInstanceInsertSchema, worker)).toBeDefined(); + expect(v.parse(workerInstanceSelectSchema, worker)).toBeDefined(); + + const lease = { + acquiredAt: jobCreatedAt, + expiresAt: jobNextRunAt, + jobRunId, + leaseToken: jobLeaseToken, + renewedAt: jobUpdatedAt, + resourceKey: "database", + workerInstanceId: jobWorkerId, + }; + expect(v.parse(resourceLeaseInsertSchema, lease)).toBeDefined(); + expect(v.parse(resourceLeaseSelectSchema, lease)).toBeDefined(); + + expect( + v.parse(jobWorkerControlSelectSchema, { + claimingPaused: false, + id: 1, + updatedAt: new Date(0), + updatedById: null, + updatedByKind: null, + version: 1, + }) + ).toBeDefined(); + expect( + v.parse(jobWorkerControlUpdateSchema, { + claimingPaused: true, + updatedAt: jobUpdatedAt, + updatedById: jobUserId, + updatedByKind: "user", + version: 2, + }) + ).toBeDefined(); + }); + + test("refines durable job identifiers, JSON roots, and bounded counters", () => { + expect(() => + v.parse(scheduledJobSelectSchema, { + ...validScheduledJobRow, + id: "System.Worker-Smoke", + }) + ).toThrow("Schedule id is invalid"); + expect(() => + v.parse(scheduledJobSelectSchema, { + ...validScheduledJobRow, + actionPayloadJson: "[]", + }) + ).toThrow("Stored job payload must contain a JSON object"); + expect(() => + v.parse(scheduledJobSelectSchema, { + ...validScheduledJobRow, + resourceKeysJson: '["database","database"]', + }) + ).toThrow("Stored job resource keys are not canonical"); + expect(() => + v.parse(scheduledJobSelectSchema, { + ...validScheduledJobRow, + resourceKeysJson: "{}", + }) + ).toThrow("Stored job resource keys are not canonical"); + expect(() => + v.parse(jobRunSelectSchema, { + ...validJobRunRow, + id: "550e8400-e29b-41d4-a716-446655440000", + }) + ).toThrow("Expected a lowercase UUIDv7 identifier"); + expect(() => + v.parse(jobRunInsertSchema, { + ...validJobRunInsert, + availableAt: new Date(1999), + }) + ).toThrow("New job run must be an internally consistent queued row"); + expect(() => + v.parse(jobRunSelectSchema, { + ...validJobRunRow, + eventCount: 1001, + }) + ).toThrow("Stored job event count is invalid"); + expect(() => + v.parse(jobRunSelectSchema, { + ...validJobRunRow, + eventCount: 1, + payloadEventCount: 2, + }) + ).toThrow("Stored job run is inconsistent"); + expect(() => + v.parse(resourceLeaseSelectSchema, { + acquiredAt: jobCreatedAt, + expiresAt: jobNextRunAt, + jobRunId, + leaseToken: jobLeaseToken, + renewedAt: jobUpdatedAt, + resourceKey: "Database", + workerInstanceId: jobWorkerId, + }) + ).toThrow("Job resource key is invalid"); + }); + + test("rejects inconsistent durable job lifecycle rows", () => { + expect( + v.parse(scheduledJobSelectSchema, { + ...validScheduledJobRow, + enabled: false, + }) + ).toMatchObject({ enabled: false, nextRunAt: jobNextRunAt }); + expect(() => + v.parse(scheduledJobSelectSchema, { + ...validScheduledJobRow, + nextRunAt: null, + }) + ).toThrow(); + expect(() => + v.parse(jobDisableIntentSelectSchema, { + createdAt: jobCreatedAt, + createdById: jobUserId, + createdByKind: "user", + endedAt: jobUpdatedAt, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: null, + externalJobId: null, + externalProvider: null, + id: jobDisableIntentId, + reason: "Incomplete closure.", + scheduledJobId: jobScheduleId, + targetKind: "dashboard-schedule", + }) + ).toThrow(); + expect(() => + v.parse(jobRunSelectSchema, { + ...validJobRunRow, + attemptCount: 1, + firstStartedAt: jobUpdatedAt, + lastAttemptStartedAt: jobUpdatedAt, + state: "running", + }) + ).toThrow(); + expect(() => + v.parse(jobRunEventSelectSchema, { + attempt: 0, + jobRunId, + kind: "progress", + message: null, + occurredAt: jobUpdatedAt, + progressJson: null, + sequence: 1, + workerInstanceId: null, + }) + ).toThrow(); + expect(() => + v.parse(workerInstanceSelectSchema, { + capacity: 1, + drainingAt: null, + heartbeatAt: jobUpdatedAt, + id: jobWorkerId, + pid: 1234, + releaseId: "b".repeat(40), + startedAt: jobCreatedAt, + state: "stopped", + stoppedAt: null, + }) + ).toThrow(); + expect(() => + v.parse(resourceLeaseSelectSchema, { + acquiredAt: jobCreatedAt, + expiresAt: jobUpdatedAt, + jobRunId, + leaseToken: jobLeaseToken, + renewedAt: jobUpdatedAt, + resourceKey: "database", + workerInstanceId: jobWorkerId, + }) + ).toThrow(); + expect(() => + v.parse(jobWorkerControlSelectSchema, { + claimingPaused: false, + id: 1, + updatedAt: new Date(0), + updatedById: null, + updatedByKind: null, + version: 2, + }) + ).toThrow(); + expect(() => v.parse(jobDisableIntentCloseSchema, {})).toThrow(); + expect(() => + v.parse(jobDisableIntentCloseSchema, { + endedAt: jobUpdatedAt, + endedById: jobUserId, + endedByKind: "user", + endedReason: "expired", + }) + ).toThrow(); + }); }); diff --git a/greenfield/src/server/database/validation/scalars.ts b/greenfield/src/server/database/validation/scalars.ts index fc29e3bcc..3b7fed97c 100644 --- a/greenfield/src/server/database/validation/scalars.ts +++ b/greenfield/src/server/database/validation/scalars.ts @@ -50,6 +50,8 @@ export function jsonObjectTextSchema(schema: v.StringSchema) { * @param schema Generated Drizzle Date schema. * @returns Refined nonnegative epoch Date schema. */ -export function nonnegativeDateSchema(schema: v.DateSchema) { +export function nonnegativeDateSchema( + schema: v.DateSchema | undefined> +) { return v.pipe(schema, nonnegativeDateAction()); } diff --git a/greenfield/src/server/database/validation/scheduledJobs.ts b/greenfield/src/server/database/validation/scheduledJobs.ts new file mode 100644 index 000000000..9878cb33a --- /dev/null +++ b/greenfield/src/server/database/validation/scheduledJobs.ts @@ -0,0 +1,182 @@ +import { createInsertSchema, createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { + jobActionKeySchema, + jobAttemptLimitSchema, + jobCancellationPolicySchema, + jobDescriptionSchema, + jobDisplayNameSchema, + jobPayloadMaximumBytes, + jobPayloadSchema, + jobPrioritySchema, + jobResourceClassSchema, + jobResourceKeysMaximumBytes, + jobResourceKeysSchema, + jobTimeoutSchema, + jobVersionSchema, + scheduleCronExpressionSchema, + scheduleIdSchema, + scheduleIntervalMaximumMilliseconds, + scheduleIntervalMinimumMilliseconds, + scheduleKindSchema, + scheduleTimeOfDaySchema, + scheduleTimeZoneSchema, +} from "../../../contracts/jobModel.ts"; +import { utf8ByteLength } from "../../../shared/encoding.ts"; +import { parseJsonText } from "../../../shared/json.ts"; +import { positiveSafeIntegerSchema } from "../../../shared/validation.ts"; +import { scheduledJobs } from "../schema/scheduledJobs.ts"; +import { nonnegativeDateSchema } from "./scalars.ts"; + +const scheduleIntervalSchema = v.pipe( + positiveSafeIntegerSchema("Schedule interval is invalid"), + v.minValue(scheduleIntervalMinimumMilliseconds, "Schedule interval is invalid"), + v.maxValue(scheduleIntervalMaximumMilliseconds, "Schedule interval is invalid") +); + +const storedCronExpressionSchema = v.pipe( + v.string("Stored schedule cron expression is invalid"), + v.check((value) => { + const parsed = v.safeParse(scheduleCronExpressionSchema, value); + return parsed.success && parsed.output === value; + }, "Stored schedule cron expression is not canonical") +); + +const actionPayloadJsonSchema = v.pipe( + v.string("Stored job payload is invalid"), + v.check( + (value) => utf8ByteLength(value) <= jobPayloadMaximumBytes, + "Stored job payload is outside its byte budget" + ), + v.check( + (value) => v.safeParse(jobPayloadSchema, parseJsonText(value)).success, + "Stored job payload must contain a JSON object" + ) +); + +const resourceKeysJsonSchema = v.pipe( + v.string("Stored job resource keys are invalid"), + v.check( + (value) => utf8ByteLength(value) <= jobResourceKeysMaximumBytes, + "Stored job resource keys are outside their byte budget" + ), + v.check( + (value) => v.safeParse(jobResourceKeysSchema, parseJsonText(value)).success, + "Stored job resource keys are not canonical" + ) +); + +interface StoredScheduleShape { + readonly cronExpression?: string | null; + readonly enabled: boolean; + readonly intervalMs?: number | null; + readonly nextRunAt?: Date | null; + readonly scheduleKind: "cron" | "daily" | "interval"; + readonly timeOfDay?: string | null; + readonly timeZone?: string | null; +} + +function scheduleShapeIsConsistent(schedule: StoredScheduleShape): boolean { + const cronExpression = schedule.cronExpression ?? null; + const intervalMs = schedule.intervalMs ?? null; + const nextRunAt = schedule.nextRunAt ?? null; + const timeOfDay = schedule.timeOfDay ?? null; + const timeZone = schedule.timeZone ?? null; + + if (schedule.enabled && !(nextRunAt instanceof Date)) return false; + if (schedule.scheduleKind === "interval") { + return ( + intervalMs !== null && + cronExpression === null && + timeOfDay === null && + timeZone === null + ); + } + if (schedule.scheduleKind === "daily") { + return ( + intervalMs === null && + cronExpression === null && + timeOfDay !== null && + timeZone !== null + ); + } + return ( + intervalMs === null && + cronExpression !== null && + timeOfDay === null && + timeZone !== null + ); +} + +function scheduleTimesAreConsistent(schedule: { + readonly createdAt: Date; + readonly updatedAt: Date; +}): boolean { + return schedule.updatedAt.getTime() >= schedule.createdAt.getTime(); +} + +const scheduleRefinements = { + actionKey: () => jobActionKeySchema, + actionPayloadJson: () => actionPayloadJsonSchema, + attemptLimit: () => jobAttemptLimitSchema, + cancellationPolicy: () => jobCancellationPolicySchema, + createdAt: nonnegativeDateSchema, + cronExpression: () => v.nullable(storedCronExpressionSchema), + description: () => jobDescriptionSchema, + id: () => scheduleIdSchema, + intervalMs: () => v.nullable(scheduleIntervalSchema), + name: () => jobDisplayNameSchema, + nextRunAt: nonnegativeDateSchema, + priority: () => jobPrioritySchema, + resourceClass: () => jobResourceClassSchema, + resourceKeysJson: () => resourceKeysJsonSchema, + scheduleKind: () => scheduleKindSchema, + timeOfDay: () => v.nullable(scheduleTimeOfDaySchema), + timeZone: () => v.nullable(scheduleTimeZoneSchema), + timeoutMs: () => jobTimeoutSchema, + updatedAt: nonnegativeDateSchema, + version: () => jobVersionSchema, +}; + +const generatedScheduledJobSelectSchema = createSelectSchema( + scheduledJobs, + scheduleRefinements +); +const scheduledJobSelectObjectSchema = v.strictObject( + generatedScheduledJobSelectSchema.entries +); + +/** Validates one complete schedule row read from SQLite. */ +export const scheduledJobSelectSchema = v.pipe( + scheduledJobSelectObjectSchema, + v.check( + (schedule) => scheduleShapeIsConsistent(schedule), + "Stored schedule shape is inconsistent" + ), + v.check( + (schedule) => scheduleTimesAreConsistent(schedule), + "Stored schedule timestamps are inconsistent" + ) +); + +const generatedScheduledJobInsertSchema = createInsertSchema( + scheduledJobs, + scheduleRefinements +); +const scheduledJobInsertObjectSchema = v.strictObject( + generatedScheduledJobInsertSchema.entries +); + +/** Validates one complete code-owned schedule before insertion. */ +export const scheduledJobInsertSchema = v.pipe( + scheduledJobInsertObjectSchema, + v.check( + (schedule) => scheduleShapeIsConsistent(schedule), + "Stored schedule shape is inconsistent" + ), + v.check( + (schedule) => scheduleTimesAreConsistent(schedule), + "Stored schedule timestamps are inconsistent" + ) +); diff --git a/greenfield/src/server/database/validation/workerInstances.ts b/greenfield/src/server/database/validation/workerInstances.ts new file mode 100644 index 000000000..ea5957eae --- /dev/null +++ b/greenfield/src/server/database/validation/workerInstances.ts @@ -0,0 +1,99 @@ +import { createInsertSchema, createSelectSchema } from "drizzle-orm/valibot"; +import * as v from "valibot"; + +import { + jobWorkerCapacityMaximum, + jobWorkerStateSchema, +} from "../../../contracts/jobModel.ts"; +import { + fullCommitShaSchema, + positiveSafeIntegerSchema, +} from "../../../shared/validation.ts"; +import { workerInstances } from "../schema/workerInstances.ts"; +import { nonnegativeDateSchema, uuidV7TextSchema } from "./scalars.ts"; + +const workerCapacitySchema = v.pipe( + positiveSafeIntegerSchema("Stored worker capacity is invalid"), + v.maxValue(jobWorkerCapacityMaximum, "Stored worker capacity is invalid") +); +const workerPidSchema = v.pipe( + positiveSafeIntegerSchema("Stored worker pid is invalid"), + v.maxValue(2_147_483_647, "Stored worker pid is invalid") +); + +interface StoredWorkerInstance { + readonly drainingAt?: Date | null; + readonly heartbeatAt: Date; + readonly startedAt: Date; + readonly state: "draining" | "online" | "stopped"; + readonly stoppedAt?: Date | null; +} + +function workerLifecycleIsConsistent(worker: StoredWorkerInstance): boolean { + const drainingAt = worker.drainingAt ?? null; + const stoppedAt = worker.stoppedAt ?? null; + if (worker.heartbeatAt.getTime() < worker.startedAt.getTime()) return false; + if ( + (worker.state === "online" && (drainingAt !== null || stoppedAt !== null)) || + (worker.state === "draining" && (drainingAt === null || stoppedAt !== null)) || + (worker.state === "stopped" && (drainingAt === null || stoppedAt === null)) + ) { + return false; + } + return ( + (drainingAt === null || drainingAt.getTime() >= worker.startedAt.getTime()) && + (stoppedAt === null || + stoppedAt.getTime() >= (drainingAt ?? worker.startedAt).getTime()) && + (stoppedAt === null || stoppedAt.getTime() >= worker.heartbeatAt.getTime()) + ); +} + +const workerRefinements = { + capacity: () => workerCapacitySchema, + drainingAt: nonnegativeDateSchema, + heartbeatAt: nonnegativeDateSchema, + id: uuidV7TextSchema, + pid: () => workerPidSchema, + releaseId: () => fullCommitShaSchema("Stored worker release id is invalid"), + startedAt: nonnegativeDateSchema, + state: () => jobWorkerStateSchema, + stoppedAt: nonnegativeDateSchema, +}; + +const generatedWorkerInstanceSelectSchema = createSelectSchema( + workerInstances, + workerRefinements +); +const workerInstanceSelectObjectSchema = v.strictObject( + generatedWorkerInstanceSelectSchema.entries +); + +/** Validates one complete worker registration read from SQLite. */ +export const workerInstanceSelectSchema = v.pipe( + workerInstanceSelectObjectSchema, + v.check( + (worker) => workerLifecycleIsConsistent(worker), + "Stored worker lifecycle is inconsistent" + ) +); + +const generatedWorkerInstanceInsertSchema = createInsertSchema( + workerInstances, + workerRefinements +); +const workerInstanceInsertObjectSchema = v.strictObject( + generatedWorkerInstanceInsertSchema.entries +); + +/** Validates one initially-online worker registration before insertion. */ +export const workerInstanceInsertSchema = v.pipe( + workerInstanceInsertObjectSchema, + v.check( + (worker) => + worker.state === "online" && + worker.drainingAt == null && + worker.stoppedAt == null && + workerLifecycleIsConsistent(worker), + "New worker registration must be online and consistent" + ) +); diff --git a/greenfield/src/server/domains/jobs/actionRegistry.test.ts b/greenfield/src/server/domains/jobs/actionRegistry.test.ts new file mode 100644 index 000000000..e25d68a6e --- /dev/null +++ b/greenfield/src/server/domains/jobs/actionRegistry.test.ts @@ -0,0 +1,80 @@ +import { describe, expect, test } from "bun:test"; + +import { + findJobActionRegistration, + isRegisteredJobSchedule, + parseJobActionOutputMessage, + parseJobActionProgress, + validateJobActionRegistration, +} from "./actionRegistry.ts"; + +describe("durable job action registry", () => { + test("exposes only the safe worker smoke action", () => { + const registration = findJobActionRegistration("system.worker-smoke"); + + expect(registration).toMatchObject({ + cancellationPolicy: "cooperative", + manualExposure: "jobs-write", + resourceClass: "light", + retrySafe: true, + }); + expect(findJobActionRegistration("system.shell")).toBeUndefined(); + expect( + isRegisteredJobSchedule("system.worker-smoke", "system.worker-smoke") + ).toBe(true); + expect( + isRegisteredJobSchedule("system.worker-smoke-renamed", "system.worker-smoke") + ).toBe(false); + expect(isRegisteredJobSchedule("system.worker-smoke", "system.shell")).toBe( + false + ); + }); + + test("retains the canonical schedule produced while validating a registration", () => { + const smoke = findJobActionRegistration("system.worker-smoke"); + if (smoke === undefined) throw new TypeError("Missing smoke registration"); + + const registration = validateJobActionRegistration({ + ...smoke, + defaultSchedule: { + expression: "0\t0 * JAN MON", + kind: "cron", + timeZone: "UTC", + }, + scheduleId: "system.normalized-cron-test", + }); + + expect(registration.defaultSchedule).toEqual({ + expression: "0 0 * 1 1", + kind: "cron", + timeZone: "UTC", + }); + expect(() => + validateJobActionRegistration({ + ...smoke, + manualExposure: "administrator" as never, + }) + ).toThrow("Job manual exposure is invalid"); + expect(() => + validateJobActionRegistration({ + ...smoke, + retrySafe: "yes" as never, + }) + ).toThrow("Job retry-safe flag is invalid"); + expect(() => + validateJobActionRegistration({ + ...smoke, + execute: null as never, + }) + ).toThrow("Job action executor is invalid"); + }); + + test("bounds progress and output before persistence", () => { + expect(parseJobActionProgress({ completed: 1 })).toEqual({ completed: 1 }); + expect(parseJobActionOutputMessage("safe output")).toBe("safe output"); + + expect(() => parseJobActionProgress({ value: "x".repeat(17 * 1024) })).toThrow(); + expect(() => parseJobActionOutputMessage("🙂".repeat(2000))).toThrow(); + expect(() => parseJobActionOutputMessage("line\nbreak")).toThrow(); + }); +}); diff --git a/greenfield/src/server/domains/jobs/actionRegistry.ts b/greenfield/src/server/domains/jobs/actionRegistry.ts new file mode 100644 index 000000000..855d33028 --- /dev/null +++ b/greenfield/src/server/domains/jobs/actionRegistry.ts @@ -0,0 +1,230 @@ +import { Effect } from "effect"; +import * as v from "valibot"; + +import { + type JobCancellationPolicy, + type JobResourceClass, + type JobRunResult, + type ScheduleConfiguration, + jobActionKeySchema, + jobAttemptLimitSchema, + jobCancellationPolicySchema, + jobDescriptionSchema, + jobDisplayNameSchema, + jobPayloadSchema, + jobPrioritySchema, + jobResourceClassSchema, + jobResourceKeysSchema, + jobRunEventProgressSchema, + jobTimeoutSchema, + scheduleConfigurationSchema, + scheduleIdSchema, +} from "../../../contracts/jobModel.ts"; +import { utf8ByteLength } from "../../../shared/encoding.ts"; +import type { JsonObject } from "../../../shared/json.ts"; +import { boundedControlSafeTextSchema } from "../../../shared/validation.ts"; + +export type JobManualExposure = "jobs-write" | "none"; +export type JobActionEventWriteResult = "appended" | "dropped" | "truncated"; + +const jobManualExposureSchema = v.picklist( + ["jobs-write", "none"], + "Job manual exposure is invalid" +); + +/** Explicit action-owned classification for failures safe to retry from scratch. */ +export class JobActionRetryableError extends Error { + constructor(cause?: unknown) { + super( + "The job action reported a retryable failure", + cause === undefined ? undefined : { cause } + ); + this.name = "JobActionRetryableError"; + } +} + +const jobActionOutputMessageSchema = v.pipe( + boundedControlSafeTextSchema(4096, "Job action output is invalid"), + v.check((message) => utf8ByteLength(message) <= 4096, "Job action output is invalid") +); + +/** Safe execution context supplied by the worker without host or shell authority. */ +export interface JobActionExecutionContext { + readonly databaseReleaseId: string; + readonly nowMs: () => number; + readonly reportProgress: ( + progress: JsonObject + ) => Effect.Effect; + readonly workerInstanceId: string; + readonly writeOutput: ( + kind: "stderr" | "stdout", + message: string + ) => Effect.Effect; +} + +/** Code-owned action metadata reconciled into schedules and captured into each run. */ +export interface JobActionRegistration { + readonly actionKey: string; + readonly actionPayload: JsonObject; + readonly attemptLimit: number; + readonly cancellationPolicy: JobCancellationPolicy; + readonly defaultEnabled: boolean; + readonly defaultSchedule: ScheduleConfiguration; + readonly description: string; + readonly displayName: string; + readonly execute: ( + context: JobActionExecutionContext, + payload: JsonObject + ) => Effect.Effect; + readonly manualExposure: JobManualExposure; + readonly priority: number; + readonly resourceClass: JobResourceClass; + readonly resourceKeys: readonly string[]; + readonly retrySafe: boolean; + readonly scheduleId: string; + readonly timeoutMs: number; +} + +const emptyPayloadSchema = v.strictObject({}); +const smokeResultSchema = v.strictObject({ + checkedAtMs: v.pipe( + v.number("Worker smoke timestamp is invalid"), + v.safeInteger("Worker smoke timestamp is invalid"), + v.minValue(0, "Worker smoke timestamp is invalid") + ), + databaseReleaseId: v.pipe( + v.string("Worker smoke release is invalid"), + v.length(40, "Worker smoke release is invalid"), + v.regex(/^[0-9a-f]{40}$/u, "Worker smoke release is invalid") + ), + status: v.literal("ok"), + workerInstanceId: v.pipe( + v.string("Worker smoke identity is invalid"), + v.uuid("Worker smoke identity is invalid") + ), +}); + +/** + * Validates one code-owned action and retains canonical schedule output. + * @param registration Candidate release-owned action metadata. + * @returns A frozen registration safe for reconciliation and execution. + */ +export function validateJobActionRegistration( + registration: JobActionRegistration +): JobActionRegistration { + v.parse(jobActionKeySchema, registration.actionKey); + const actionPayload = v.parse(jobPayloadSchema, registration.actionPayload); + v.parse(jobAttemptLimitSchema, registration.attemptLimit); + v.parse(jobCancellationPolicySchema, registration.cancellationPolicy); + v.parse( + v.boolean("Job default-enabled flag is invalid"), + registration.defaultEnabled + ); + const defaultSchedule = v.parse( + scheduleConfigurationSchema, + registration.defaultSchedule + ); + v.parse(jobDescriptionSchema, registration.description); + v.parse(jobDisplayNameSchema, registration.displayName); + v.parse(jobPrioritySchema, registration.priority); + v.parse(jobResourceClassSchema, registration.resourceClass); + const resourceKeys = v.parse(jobResourceKeysSchema, registration.resourceKeys); + v.parse(jobManualExposureSchema, registration.manualExposure); + v.parse(v.boolean("Job retry-safe flag is invalid"), registration.retrySafe); + v.parse(scheduleIdSchema, registration.scheduleId); + v.parse(jobTimeoutSchema, registration.timeoutMs); + v.parse(v.function("Job action executor is invalid"), registration.execute); + return Object.freeze({ + ...registration, + actionPayload: Object.freeze({ ...actionPayload }), + defaultSchedule: Object.freeze({ ...defaultSchedule }), + resourceKeys: Object.freeze([...resourceKeys]), + }); +} + +const workerSmokeRegistration = validateJobActionRegistration({ + actionKey: "system.worker-smoke", + actionPayload: Object.freeze({}), + attemptLimit: 3, + cancellationPolicy: "cooperative", + defaultEnabled: false, + defaultSchedule: Object.freeze({ + intervalMs: 86_400_000, + kind: "interval", + }), + description: + "Verifies the release, database, and durable worker without host mutation.", + displayName: "Worker smoke", + execute: (context, payload) => + Effect.sync(() => { + v.parse(emptyPayloadSchema, payload); + return v.parse(smokeResultSchema, { + checkedAtMs: context.nowMs(), + databaseReleaseId: context.databaseReleaseId, + status: "ok", + workerInstanceId: context.workerInstanceId, + }); + }), + manualExposure: "jobs-write", + priority: 0, + resourceClass: "light", + resourceKeys: Object.freeze(["database"]), + retrySafe: true, + scheduleId: "system.worker-smoke", + timeoutMs: 30_000, +}); + +/** Complete reviewed action registry for this slice. */ +export const jobActionRegistrations = Object.freeze([workerSmokeRegistration]); + +const registrationByKey = new Map( + jobActionRegistrations.map((registration) => [registration.actionKey, registration]) +); +if (registrationByKey.size !== jobActionRegistrations.length) { + throw new Error("Job action registry contains duplicate action keys"); +} +const registrationByScheduleId = new Map( + jobActionRegistrations.map((registration) => [registration.scheduleId, registration]) +); +if (registrationByScheduleId.size !== jobActionRegistrations.length) { + throw new Error("Job action registry contains duplicate schedule IDs"); +} + +/** + * Resolves one exact reviewed action. + * @param actionKey Durable action identity. + * @returns The action registration, when this release implements it. + */ +export function findJobActionRegistration( + actionKey: string +): JobActionRegistration | undefined { + return registrationByKey.get(actionKey); +} + +/** + * Checks that a durable schedule still belongs to the exact registered action. + * @param scheduleId Durable schedule identity. + * @param actionKey Durable action identity captured by the schedule. + * @returns Whether this release owns the exact schedule/action pair. + */ +export function isRegisteredJobSchedule(scheduleId: string, actionKey: string): boolean { + return registrationByScheduleId.get(scheduleId)?.actionKey === actionKey; +} + +/** + * Validates one action-owned progress payload before it reaches persistence. + * @param progress Structured progress candidate. + * @returns The bounded transport-safe payload. + */ +export function parseJobActionProgress(progress: JsonObject): JsonObject { + return v.parse(jobRunEventProgressSchema, progress); +} + +/** + * Validates one action-owned output fragment before it reaches persistence. + * @param message Human-readable output candidate. + * @returns The bounded control-safe message. + */ +export function parseJobActionOutputMessage(message: string): string { + return v.parse(jobActionOutputMessageSchema, message); +} diff --git a/greenfield/src/server/domains/jobs/coordinator.test.ts b/greenfield/src/server/domains/jobs/coordinator.test.ts new file mode 100644 index 000000000..09fe9268b --- /dev/null +++ b/greenfield/src/server/domains/jobs/coordinator.test.ts @@ -0,0 +1,2001 @@ +import { describe, expect, spyOn, test } from "bun:test"; + +import { Effect } from "effect"; + +import { testImmediateDatabaseWriteAdmission } from "../../test/support/databaseWriteAdmission.ts"; +import { openFreshMigratedDatabase } from "../../test/support/freshDatabase.ts"; +import { + type JobActionRegistration, + JobActionRetryableError, + jobActionRegistrations, +} from "./actionRegistry.ts"; +import { + createJobWorkerCoordinator, + jobSchedulePollScanLimit, + type JobWorkerCoordinatorOptions, + type JobWorkerSideEffectFactory, + type JobWorkerSideEffectInput, +} from "./coordinator.ts"; +import type { + JobDisableIntentRecord, + JobRunRecord, + ScheduledJobRecord, + WorkerInstanceRecord, +} from "./records.ts"; +import { + createJobRepository, + type DueScheduleEnqueueInput, + type ExpireDisableIntentResult, + type ExpireDisableIntentsInput, + type JobAppendEventResult, + type JobClaimResult, + type JobMutationSideEffects, + type JobRepository, + type JobRunInsert, + type JobSettlementResult, + type ListDueSchedulesInput, +} from "./repository.ts"; +import { createJobRealtimeSideEffects } from "./sideEffects.ts"; + +const releaseId = "a".repeat(40); +const at = new Date("2026-08-08T00:00:00.000Z"); + +const noSideEffects = Object.freeze({ + auditEvents: Object.freeze([]), + realtimeEvents: Object.freeze([]), +}); +const sideEffects: JobWorkerSideEffectFactory = Object.freeze({ + forQueue: () => noSideEffects, + forRun: () => noSideEffects, + forRunEvent: () => noSideEffects, + forSchedule: () => noSideEffects, + forScheduleEvent: () => noSideEffects, +}); + +function deferred() { + let resolveDeferred: ((value: T | PromiseLike) => void) | undefined; + const promise = new Promise((resolve) => { + resolveDeferred = resolve; + }); + return { + promise, + resolve(value: T) { + resolveDeferred?.(value); + }, + }; +} + +function workerRecord( + id: string, + state: "draining" | "online" | "stopped", + heartbeatAt = at +) { + return { + capacity: 1, + drainingAt: state === "online" ? null : heartbeatAt, + heartbeatAt, + id, + pid: 100, + releaseId, + startedAt: at, + state, + stoppedAt: state === "stopped" ? heartbeatAt : null, + } satisfies WorkerInstanceRecord; +} + +function claimedRun(workerId: string, actionKey = "system.worker-smoke"): JobRunRecord { + return { + actionKey, + attemptCount: 1, + attemptLimit: 3, + availableAt: at, + cancellationPolicy: "cooperative", + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: "Worker smoke", + enqueueSha256: "b".repeat(64), + eventBytes: 0, + eventCount: 2, + finishedAt: null, + firstStartedAt: at, + heartbeatAt: at, + id: Bun.randomUUIDv7(), + idempotencyKey: "c".repeat(64), + lastAttemptStartedAt: at, + leaseExpiresAt: new Date(at.getTime() + 120_000), + leaseOwnerId: workerId, + leaseToken: Bun.randomUUIDv7(), + payloadEventCount: 0, + payloadJson: "{}", + priority: 0, + queuedAt: at, + requestedById: "system.scheduler", + requestedByKind: "system", + resourceClass: "light", + resourceKeysJson: '["database"]', + resultJson: null, + retrySafe: true, + scheduledForAt: null, + scheduledJobId: null, + scheduledJobVersion: null, + state: "running", + stateVersion: 2, + terminalCode: null, + terminalMessage: null, + timeoutMs: 30_000, + triggerType: "system", + updatedAt: at, + }; +} + +function intervalSchedule( + overrides: Partial = {} +): ScheduledJobRecord { + return { + actionKey: "system.worker-smoke", + actionPayloadJson: "{}", + attemptLimit: 3, + cancellationPolicy: "cooperative", + createdAt: at, + cronExpression: null, + description: "Worker smoke", + enabled: true, + id: "system.worker-smoke", + intervalMs: 60_000, + name: "Worker smoke", + nextRunAt: new Date(at.getTime() - 120_000), + priority: 0, + resourceClass: "light", + resourceKeysJson: '["database"]', + retrySafe: true, + scheduleKind: "interval", + timeOfDay: null, + timeZone: null, + timeoutMs: 30_000, + updatedAt: at, + version: 1, + ...overrides, + }; +} + +function queuedScheduledRun( + schedule: ScheduledJobRecord, + overrides: Partial = {} +): JobRunInsert { + if (schedule.nextRunAt === null) { + throw new Error("Expected a durable schedule cursor"); + } + return { + actionKey: schedule.actionKey, + attemptLimit: schedule.attemptLimit, + availableAt: new Date(at.getTime() + 86_400_000), + cancellationPolicy: schedule.cancellationPolicy, + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: schedule.name, + enqueueSha256: "d".repeat(64), + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id: Bun.randomUUIDv7(), + idempotencyKey: "e".repeat(32), + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadJson: schedule.actionPayloadJson, + priority: schedule.priority, + queuedAt: at, + requestedById: "jobs-scheduler", + requestedByKind: "system", + resourceClass: schedule.resourceClass, + resourceKeysJson: schedule.resourceKeysJson, + resultJson: null, + retrySafe: schedule.retrySafe, + scheduledForAt: schedule.nextRunAt, + scheduledJobId: schedule.id, + scheduledJobVersion: schedule.version, + state: "queued", + terminalCode: null, + terminalMessage: null, + timeoutMs: schedule.timeoutMs, + triggerType: "schedule", + updatedAt: at, + ...overrides, + }; +} + +function expiredDisableIntent(scheduleId: string): JobDisableIntentRecord { + return { + createdAt: new Date(at.getTime() - 120_000), + createdById: "019fdf20-0000-7000-8000-000000000001", + createdByKind: "user", + endedAt: at, + endedById: "system.jobs-worker", + endedByKind: "system", + endedReason: "expired", + expiresAt: new Date(at.getTime() - 60_000), + externalJobId: null, + externalProvider: null, + id: "019fdf20-0000-7000-8000-000000000002", + reason: "Temporary operator pause", + scheduledJobId: scheduleId, + targetKind: "dashboard-schedule", + }; +} + +interface RepositoryFixtureOptions { + readonly appendEvent?: ( + input: Parameters[0] + ) => JobAppendEventResult; + readonly cancellationRequested?: boolean; + readonly claim?: JobClaimResult; + readonly claimGate?: Promise; + readonly claims?: readonly JobClaimResult[]; + readonly dueSchedules?: readonly ScheduledJobRecord[]; + readonly drainWorkerAt?: Date; + readonly expiringSchedule?: ScheduledJobRecord; + readonly expiryGate?: Promise; + readonly expiryFailure?: Error; + readonly expiryResults?: readonly ExpireDisableIntentResult[]; + readonly heartbeatFailure?: Error; + readonly reconciliationFailure?: Error; + readonly recoveredRuns?: readonly JobRunRecord[]; + readonly registrationFailure?: Error; + readonly settlementAt?: Date; + readonly settlementRun?: JobRunRecord; + readonly stopWorkerAt?: Date; +} + +function repositoryFixture(options: RepositoryFixtureOptions = {}) { + const events: string[] = []; + const claimSideEffects: JobMutationSideEffects[] = []; + const claimInputs: Array[0]> = []; + const enqueues: DueScheduleEnqueueInput[] = []; + const eventSideEffects: JobMutationSideEffects[] = []; + const expiryEligibility: boolean[] = []; + const expiryNextRuns: Date[] = []; + const recoverySideEffects: JobMutationSideEffects[] = []; + const reconciliationInputs: Array< + Parameters[0] + > = []; + const lifecycleSideEffects: Array<{ + readonly operation: "drain" | "stop"; + readonly sideEffects: JobMutationSideEffects; + }> = []; + const settlements: Array[0]> = []; + const settlementSideEffects: JobMutationSideEffects[] = []; + const claims = [ + ...(options.claims ?? (options.claim === undefined ? [] : [options.claim])), + ]; + let recoveredRuns = [...(options.recoveredRuns ?? [])]; + const eventRun = claims.find((result) => result.kind === "claimed")?.run; + let dueSchedules = [...(options.dueSchedules ?? [])]; + const repository = { + appendClaimEvent(input) { + events.push(`append:${input.kind}`); + const result = options.appendEvent?.(input) ?? { kind: "dropped" }; + if ( + eventRun !== undefined && + (result.kind === "appended" || result.kind === "truncated") + ) { + eventSideEffects.push( + input.sideEffectsForRun({ + ...eventRun, + updatedAt: result.event?.occurredAt ?? eventRun.updatedAt, + }) + ); + } + return Promise.resolve(result); + }, + beginWorkerDrain(input) { + events.push("drain"); + const worker = workerRecord( + input.workerId, + "draining", + options.drainWorkerAt + ); + lifecycleSideEffects.push({ + operation: "drain", + sideEffects: input.sideEffectsForWorker(worker), + }); + return Promise.resolve({ + kind: "updated" as const, + worker, + }); + }, + async claimNextRun(input) { + claimInputs.push(input); + const result = claims.shift() ?? ({ kind: "empty" } as const); + events.push(`claim:${result.kind}`); + if (result.kind === "claimed") { + claimSideEffects.push(input.sideEffectsForClaim(result.run)); + } + if (options.claimGate !== undefined) await options.claimGate; + return result; + }, + enqueueNextDueSchedule(input) { + enqueues.push(input); + events.push("enqueue-due"); + return Promise.resolve({ kind: "not-due" as const }); + }, + async expireDisableIntents(input: ExpireDisableIntentsInput) { + events.push("expire-disable-intents"); + await options.expiryGate; + if (options.expiryFailure !== undefined) { + throw options.expiryFailure; + } + if (options.expiringSchedule !== undefined) { + const canReenable = input.canReenableSchedule(options.expiringSchedule); + expiryEligibility.push(canReenable); + if (canReenable) { + const next = input.nextRunAt(options.expiringSchedule, input.at); + if (next !== undefined) expiryNextRuns.push(next); + } + } + return options.expiryResults ?? []; + }, + heartbeatWorker(input) { + events.push("heartbeat"); + if (options.heartbeatFailure) { + return Promise.reject(options.heartbeatFailure); + } + return Promise.resolve(workerRecord(input.workerId, "online")); + }, + listDueSchedules() { + events.push("list-due"); + const schedules = dueSchedules; + dueSchedules = []; + return schedules; + }, + readClaimCancellation() { + events.push("read-cancellation"); + return { + cancelRequested: options.cancellationRequested ?? false, + valid: true, + }; + }, + reconcileSchedules(input) { + events.push(`reconcile:${input.schedules.length}`); + reconciliationInputs.push(input); + return options.reconciliationFailure === undefined + ? Promise.resolve([]) + : Promise.reject(options.reconciliationFailure); + }, + recoverExpiredClaims(input) { + const recovered = recoveredRuns; + recoveredRuns = []; + recoverySideEffects.push( + ...recovered.map((run) => input.sideEffectsForRun(run)) + ); + return Promise.resolve(recovered); + }, + registerWorker(input) { + events.push("register"); + return options.registrationFailure === undefined + ? Promise.resolve(workerRecord(input.worker.id, "online")) + : Promise.reject(options.registrationFailure); + }, + renewClaim(input) { + events.push("renew"); + return Promise.resolve({ + kind: "renewed" as const, + run: claimedRun(input.workerId), + }); + }, + settleClaim(input) { + settlements.push(input); + events.push(`settle:${input.outcome.kind}`); + const settled = { + ...(options.settlementRun ?? claimedRun(input.workerId)), + updatedAt: options.settlementAt ?? at, + }; + settlementSideEffects.push(input.sideEffectsForRun(settled)); + return Promise.resolve({ + kind: "settled" as const, + run: settled, + } satisfies JobSettlementResult); + }, + stopWorker(input) { + events.push("stop"); + const worker = workerRecord(input.workerId, "stopped", options.stopWorkerAt); + lifecycleSideEffects.push({ + operation: "stop", + sideEffects: input.sideEffectsForWorker(worker), + }); + return Promise.resolve({ + kind: "updated" as const, + worker, + }); + }, + } satisfies JobWorkerCoordinatorOptions["repository"]; + return { + claimInputs, + claimSideEffects, + enqueues, + eventSideEffects, + events, + expiryEligibility, + expiryNextRuns, + lifecycleSideEffects, + reconciliationInputs, + recoverySideEffects, + repository, + settlements, + settlementSideEffects, + }; +} + +async function waitUntil(predicate: () => boolean): Promise { + const deadline = Date.now() + 2000; + while (!predicate()) { + if (Date.now() >= deadline) throw new Error("Test condition timed out"); + await Bun.sleep(1); + } +} + +function coordinatorOptions( + repository: JobWorkerCoordinatorOptions["repository"], + workerInstanceId: string +): JobWorkerCoordinatorOptions { + return { + databaseReleaseId: releaseId, + generateId: () => Bun.randomUUIDv7(), + nowMs: () => at.getTime(), + pid: 100, + repository, + sideEffects, + timings: { + cancellationPollMs: 2, + claimLeaseMs: 100, + claimRenewalMs: 20, + heartbeatMs: 20, + idlePollMs: 2, + schedulePollMs: 20, + workerFreshnessMs: 50, + }, + workerInstanceId, + }; +} + +describe("durable job worker coordinator", () => { + test("reconciles, registers, executes the safe smoke action, and drains in order", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toMatchObject({ + kind: "succeeded", + }); + expect(fixture.events.indexOf("register")).toBeGreaterThan( + fixture.events.indexOf("reconcile:1") + ); + expect(fixture.events.indexOf("drain")).toBeLessThan( + fixture.events.indexOf("stop") + ); + expect(await coordinator.completion).toBeUndefined(); + }); + + test("derives drain and stop queue effects inside durable worker callbacks", async () => { + const workerId = Bun.randomUUIDv7(); + const drainWorkerAt = new Date(at.getTime() + 20_000); + const stopWorkerAt = new Date(at.getTime() + 30_000); + const fixture = repositoryFixture({ drainWorkerAt, stopWorkerAt }); + const queueTransitions: JobWorkerSideEffectInput[] = []; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + sideEffects: { + ...sideEffects, + forQueue: (input) => { + queueTransitions.push(input); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { id: input.targetId, kind: "queue" }, + }); + }, + }, + }); + + await coordinator.initialize(); + await coordinator.dispose(); + + expect(queueTransitions).toEqual([ + { + action: "jobs.worker.register", + at, + outcome: "accepted", + targetId: workerId, + }, + { + action: "jobs.worker.drain", + at: drainWorkerAt, + outcome: "accepted", + targetId: workerId, + }, + { + action: "jobs.worker.stop", + at: stopWorkerAt, + outcome: "succeeded", + targetId: workerId, + }, + ]); + expect( + fixture.lifecycleSideEffects.map(({ operation, sideEffects }) => ({ + operation, + realtime: sideEffects.realtimeEvents.map( + ({ entityId, occurredAt, topic }) => ({ + entityId, + occurredAt, + topic, + }) + ), + })) + ).toEqual([ + { + operation: "drain", + realtime: [ + { + entityId: workerId, + occurredAt: drainWorkerAt, + topic: "jobs.runs", + }, + ], + }, + { + operation: "stop", + realtime: [ + { + entityId: workerId, + occurredAt: stopWorkerAt, + topic: "jobs.runs", + }, + ], + }, + ]); + }); + + test("supplies durable cancelled-run effects for retired schedules", async () => { + const workerId = Bun.randomUUIDv7(); + const scheduleId = "system.worker-smoke"; + const cancelledAt = new Date(at.getTime() + 20_000); + const cancelledRun: JobRunRecord = { + ...claimedRun(workerId), + cancelRequestedAt: cancelledAt, + cancelRequestedById: "system.jobs-worker", + cancelRequestedByKind: "system", + finishedAt: cancelledAt, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + scheduledForAt: at, + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + state: "cancelled", + stateVersion: 3, + terminalCode: "cancelled/schedule-retired", + terminalMessage: + "Cancelled because the schedule was retired from the action registry", + triggerType: "schedule", + updatedAt: cancelledAt, + }; + const fixture = repositoryFixture(); + const runTransitions: JobWorkerSideEffectInput[] = []; + const scheduleTransitions: JobWorkerSideEffectInput[] = []; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + sideEffects: { + ...sideEffects, + forRun: (input) => { + runTransitions.push(input); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "run", + operation: "updated", + }, + }); + }, + forScheduleEvent: (input) => { + scheduleTransitions.push(input); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + }); + }, + }, + }); + + await coordinator.initialize(); + const reconciliation = fixture.reconciliationInputs.at(0); + const retiredRunCancellation = reconciliation?.retiredRunCancellation; + if (retiredRunCancellation === undefined) { + throw new Error("Missing retired-run cancellation metadata"); + } + const cancellationSideEffects = + retiredRunCancellation.sideEffectsForRun(cancelledRun); + await coordinator.dispose(); + + expect(retiredRunCancellation).toMatchObject({ + actor: { id: "system.jobs-worker", kind: "system" }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: + "Cancelled because the schedule was retired from the action registry", + }); + expect(runTransitions).toEqual([ + { + action: "jobs.run.cancelled", + at: cancelledAt, + outcome: "cancelled", + targetId: cancelledRun.id, + }, + ]); + expect(scheduleTransitions).toEqual([ + { + action: "jobs.run.cancelled", + at: cancelledAt, + outcome: "cancelled", + targetId: scheduleId, + }, + ]); + expect( + cancellationSideEffects.realtimeEvents.map(({ entityId, topic }) => ({ + entityId, + topic, + })) + ).toEqual([ + { entityId: cancelledRun.id, topic: "jobs.runs" }, + { entityId: scheduleId, topic: "schedules.records" }, + ]); + }); + + test("starts after retiring a queued never-cancellable schedule run", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const workerId = Bun.randomUUIDv7(); + const scheduleId = "system.worker-smoke-never-retired"; + const scheduleTransitions: JobWorkerSideEffectInput[] = []; + const runTransitions: JobWorkerSideEffectInput[] = []; + + try { + const [registered] = await repository.reconcileSchedules({ + at, + schedules: [ + intervalSchedule({ + actionKey: "retired.action", + cancellationPolicy: "never", + createdAt: new Date(at.getTime() - 1000), + id: scheduleId, + nextRunAt: at, + updatedAt: new Date(at.getTime() - 1000), + }), + ], + sideEffectsForSchedule: () => noSideEffects, + }); + if (registered === undefined) { + throw new Error("Missing never-cancellable schedule fixture"); + } + const run = queuedScheduledRun(registered, { availableAt: at }); + expect( + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at, + nextRunAt: new Date(at.getTime() + 60_000), + observedNextRunAt: at, + run, + scheduleId, + }) + ).toMatchObject({ kind: "inserted" }); + + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(repository, workerId), + sideEffects: { + ...sideEffects, + forRun: (input) => { + runTransitions.push(input); + if (input.action === "jobs.run.cancelled") { + throw new Error( + "worker reconciliation cancelled never-cancellable work" + ); + } + return noSideEffects; + }, + forSchedule: (input) => { + scheduleTransitions.push(input); + return noSideEffects; + }, + forScheduleEvent: (input) => { + scheduleTransitions.push(input); + return noSideEffects; + }, + }, + }); + + await coordinator.initialize(); + await waitUntil(() => repository.findRun(run.id)?.state === "failed"); + expect(repository.findSchedule(scheduleId)?.schedule).toMatchObject({ + enabled: false, + updatedAt: at, + version: 2, + }); + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 3, + state: "failed", + terminalCode: "action-unavailable", + }); + expect(runTransitions).toEqual([ + { + action: "jobs.run.action-unavailable", + at, + outcome: "failed", + targetId: run.id, + }, + ]); + expect(scheduleTransitions).toContainEqual({ + action: "schedules.reconcile", + at, + outcome: "accepted", + targetId: scheduleId, + }); + expect(scheduleTransitions).toContainEqual({ + action: "jobs.run.action-unavailable", + at, + outcome: "failed", + targetId: scheduleId, + }); + expect(scheduleTransitions).not.toContainEqual( + expect.objectContaining({ action: "jobs.run.cancelled" }) + ); + + await coordinator.dispose(); + expect(await coordinator.completion).toBeUndefined(); + } finally { + database.sqlite.close(true); + } + }); + + test("continues bounded claim pages and resets the cursor after a claim", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId); + const cursor = { + availableAt: new Date(at.getTime() - 1000), + availableThrough: at, + id: Bun.randomUUIDv7(), + priority: 0, + queuedAt: new Date(at.getTime() - 1000), + } as const; + const fixture = repositoryFixture({ + claims: [ + { cursor, kind: "page-exhausted" }, + { kind: "claimed", run }, + ], + }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.claimInputs.length >= 3); + await coordinator.dispose(); + + expect(fixture.claimInputs[0]).not.toHaveProperty("cursor"); + expect(fixture.claimInputs[1]).toMatchObject({ cursor }); + expect(fixture.claimInputs[2]).not.toHaveProperty("cursor"); + expect(fixture.settlements).toHaveLength(1); + }); + + test("rejects completion when initialization fails before worker loops start", async () => { + for (const operation of ["reconcile", "register"] as const) { + const failure = new Error(`${operation} failed`); + const workerId = Bun.randomUUIDv7(); + const fixture = repositoryFixture( + operation === "reconcile" + ? { reconciliationFailure: failure } + : { registrationFailure: failure } + ); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + const completion = coordinator.completion.catch((error: unknown) => error); + const initialization = coordinator.initialize(); + + expect(coordinator.initialize()).toBe(initialization); + expect(await initialization.catch((error: unknown) => error)).toBe(failure); + expect(await completion).toBe(failure); + expect(await coordinator.dispose().catch((error: unknown) => error)).toBe( + failure + ); + } + }); + + test("stops the claim monitor immediately after a fast action finishes", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const options = coordinatorOptions(fixture.repository, workerId); + const coordinator = createJobWorkerCoordinator({ + ...options, + timings: { + ...options.timings, + cancellationPollMs: 60_000, + }, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlements).toHaveLength(1); + expect(fixture.settlements[0]?.outcome.kind).toBe("succeeded"); + expect(fixture.events).not.toContain("read-cancellation"); + expect(fixture.events).not.toContain("renew"); + }); + + test("anchors claim renewal to the durable clamped heartbeat", async () => { + const workerId = Bun.randomUUIDv7(); + const durableHeartbeat = new Date(at.getTime() + 20_000); + const run: JobRunRecord = { + ...claimedRun(workerId, "test.clock-regression"), + heartbeatAt: durableHeartbeat, + leaseExpiresAt: new Date(at.getTime() + 30_000), + updatedAt: durableHeartbeat, + }; + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + let logicalNowMs = at.getTime(); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => + Effect.tryPromise(async () => { + await Bun.sleep(5); + logicalNowMs = at.getTime() + 25; + await Bun.sleep(20); + return {}; + }), + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + nowMs: () => logicalNowMs, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.events).not.toContain("renew"); + expect(fixture.settlements[0]?.outcome.kind).toBe("succeeded"); + }); + + test("invalidates a claimed schedule projection at the durable claim time", async () => { + const workerId = Bun.randomUUIDv7(); + const scheduleId = "system.worker-smoke"; + const durableAt = new Date(at.getTime() + 20_000); + const run: JobRunRecord = { + ...claimedRun(workerId), + firstStartedAt: durableAt, + heartbeatAt: durableAt, + lastAttemptStartedAt: durableAt, + leaseExpiresAt: new Date(durableAt.getTime() + 30_000), + scheduledForAt: at, + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + triggerType: "schedule", + updatedAt: durableAt, + }; + const fixture = repositoryFixture({ + claim: { kind: "claimed", run }, + settlementAt: durableAt, + }); + const observed: Array<{ + readonly action: string; + readonly at: Date; + readonly target: "queue" | "run" | "run-event" | "schedule-event"; + }> = []; + const recordingSideEffects: JobWorkerSideEffectFactory = { + forQueue: (input) => { + observed.push({ action: input.action, at: input.at, target: "queue" }); + return input.action === "jobs.run.claim" + ? createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { id: "jobs.queue", kind: "queue" }, + }) + : noSideEffects; + }, + forRun: (input) => { + observed.push({ action: input.action, at: input.at, target: "run" }); + return noSideEffects; + }, + forRunEvent: (input) => { + observed.push({ + action: input.action, + at: input.at, + target: "run-event", + }); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "run", + operation: "updated", + }, + }); + }, + forSchedule: () => noSideEffects, + forScheduleEvent: (input) => { + observed.push({ + action: input.action, + at: input.at, + target: "schedule-event", + }); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + }); + }, + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + sideEffects: recordingSideEffects, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect( + observed.filter(({ action }) => + ["jobs.run.claim", "jobs.run.succeeded"].includes(action) + ) + ).toEqual([ + { action: "jobs.run.claim", at: durableAt, target: "queue" }, + { action: "jobs.run.claim", at: durableAt, target: "run-event" }, + { + action: "jobs.run.claim", + at: durableAt, + target: "schedule-event", + }, + { action: "jobs.run.succeeded", at: durableAt, target: "run" }, + ]); + expect(fixture.claimSideEffects).toEqual([ + { + auditEvents: [], + realtimeEvents: [ + expect.objectContaining({ + entityId: "jobs.queue", + entityType: "job-queue", + operation: "snapshot-required", + topic: "jobs.runs", + }), + expect.objectContaining({ + entityId: run.id, + entityType: "job-run", + operation: "updated", + topic: "jobs.runs", + }), + expect.objectContaining({ + entityId: scheduleId, + entityType: "schedule", + operation: "updated", + topic: "schedules.records", + }), + ], + }, + ]); + }); + + test("fails the claim transaction callback when schedule invalidation fails", async () => { + const failure = new Error("schedule invalidation failed"); + const workerId = Bun.randomUUIDv7(); + const run: JobRunRecord = { + ...claimedRun(workerId), + scheduledForAt: at, + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + triggerType: "schedule", + }; + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + sideEffects: { + ...sideEffects, + forScheduleEvent: () => { + throw failure; + }, + }, + }); + const completion = coordinator.completion.catch((error: unknown) => error); + + await coordinator.initialize(); + + expect(await completion).toBe(failure); + await coordinator.dispose(); + expect(fixture.claimSideEffects).toEqual([]); + expect(fixture.settlements).toEqual([]); + }); + + test("invalidates the schedule projection in the settlement transaction", async () => { + const workerId = Bun.randomUUIDv7(); + const scheduleId = "system.worker-smoke"; + const run: JobRunRecord = { + ...claimedRun(workerId), + scheduledForAt: at, + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + triggerType: "schedule", + }; + const settlementRun: JobRunRecord = { + ...run, + finishedAt: at, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + resultJson: "{}", + state: "succeeded", + stateVersion: run.stateVersion + 1, + }; + const fixture = repositoryFixture({ + claim: { kind: "claimed", run }, + settlementRun, + }); + const recordingSideEffects: JobWorkerSideEffectFactory = { + ...sideEffects, + forRun: (input) => + createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "run", + operation: "updated", + }, + }), + forScheduleEvent: (input) => + createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + }), + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + sideEffects: recordingSideEffects, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlementSideEffects).toEqual([ + { + auditEvents: [], + realtimeEvents: [ + expect.objectContaining({ + entityId: run.id, + topic: "jobs.runs", + }), + expect.objectContaining({ + entityId: scheduleId, + topic: "schedules.records", + }), + ], + }, + ]); + }); + + test("invalidates scheduled retry and cancellation recoveries atomically", async () => { + const workerId = Bun.randomUUIDv7(); + const scheduleId = "system.worker-smoke"; + const retryRun: JobRunRecord = { + ...claimedRun(workerId), + availableAt: new Date(at.getTime() + 1000), + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + scheduledForAt: at, + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + state: "queued", + stateVersion: 3, + triggerType: "schedule", + }; + const cancelledAt = new Date(at.getTime() + 2000); + const cancelledRun: JobRunRecord = { + ...retryRun, + availableAt: at, + cancelRequestedAt: cancelledAt, + cancelRequestedById: Bun.randomUUIDv7(), + cancelRequestedByKind: "user", + finishedAt: cancelledAt, + id: Bun.randomUUIDv7(), + state: "cancelled", + stateVersion: 4, + terminalCode: "job/cancel-requested", + terminalMessage: "The run was cancelled after its worker lease expired.", + updatedAt: cancelledAt, + }; + const fixture = repositoryFixture({ recoveredRuns: [retryRun, cancelledRun] }); + const runTransitions: JobWorkerSideEffectInput[] = []; + const scheduleTransitions: JobWorkerSideEffectInput[] = []; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + sideEffects: { + ...sideEffects, + forRun: (input) => { + runTransitions.push(input); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "run", + operation: "updated", + }, + }); + }, + forScheduleEvent: (input) => { + scheduleTransitions.push(input); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + }); + }, + }, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.recoverySideEffects.length === 2); + await coordinator.dispose(); + + expect(runTransitions).toEqual([ + { + action: "jobs.run.lease-expired", + at: retryRun.updatedAt, + outcome: "failed", + targetId: retryRun.id, + }, + { + action: "jobs.run.cancelled", + at: cancelledAt, + outcome: "cancelled", + targetId: cancelledRun.id, + }, + ]); + expect(scheduleTransitions).toEqual([ + { + action: "jobs.run.lease-expired", + at: retryRun.updatedAt, + outcome: "failed", + targetId: scheduleId, + }, + { + action: "jobs.run.cancelled", + at: cancelledAt, + outcome: "cancelled", + targetId: scheduleId, + }, + ]); + expect( + fixture.recoverySideEffects.map(({ realtimeEvents }) => + realtimeEvents.map(({ entityId, topic }) => ({ entityId, topic })) + ) + ).toEqual([ + [ + { entityId: retryRun.id, topic: "jobs.runs" }, + { entityId: scheduleId, topic: "schedules.records" }, + ], + [ + { entityId: cancelledRun.id, topic: "jobs.runs" }, + { entityId: scheduleId, topic: "schedules.records" }, + ], + ]); + }); + + test("classifies a repository-normalized shutdown cancellation", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.shutdown-cancellation-race"); + const cancelledAt = new Date(at.getTime() + 1000); + const settlementRun: JobRunRecord = { + ...run, + cancelRequestedAt: cancelledAt, + cancelRequestedById: Bun.randomUUIDv7(), + cancelRequestedByKind: "user", + finishedAt: cancelledAt, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + state: "cancelled", + stateVersion: run.stateVersion + 2, + terminalCode: "cancel-requested", + terminalMessage: "The job action was cancelled.", + updatedAt: cancelledAt, + }; + const fixture = repositoryFixture({ + claim: { kind: "claimed", run }, + settlementAt: cancelledAt, + settlementRun, + }); + const observed: JobWorkerSideEffectInput[] = []; + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === run.actionKey + ? { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => Effect.never, + } + : undefined, + sideEffects: { + ...sideEffects, + forRun: (input) => { + observed.push(input); + return noSideEffects; + }, + }, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.events.includes("read-cancellation")); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toMatchObject({ + kind: "failed", + terminalCode: "worker-shutdown", + }); + expect(observed).toEqual([ + { + action: "jobs.run.cancelled", + at: cancelledAt, + outcome: "cancelled", + targetId: run.id, + }, + ]); + }); + + test("fails an unknown action closed without retrying it", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "unknown.action"); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toEqual({ + kind: "failed", + terminalCode: "action-unavailable", + terminalMessage: "This release does not implement the queued action.", + }); + }); + + test("coalesces one due interval run and advances its cadence", async () => { + const workerId = Bun.randomUUIDv7(); + const schedule = intervalSchedule(); + const fixture = repositoryFixture({ dueSchedules: [schedule] }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.enqueues.length === 1); + await coordinator.dispose(); + + const enqueue = fixture.enqueues[0]; + expect(enqueue?.observedNextRunAt).toEqual(schedule.nextRunAt); + expect(enqueue?.nextRunAt).toEqual(new Date(at.getTime() + 60_000)); + expect(enqueue?.run).toMatchObject({ + requestedById: "system.scheduler", + requestedByKind: "system", + scheduledForAt: schedule.nextRunAt, + scheduledJobId: schedule.id, + scheduledJobVersion: schedule.version, + triggerType: "schedule", + }); + }); + + test("pages past active schedules across bounded polling passes", async () => { + const workerId = Bun.randomUUIDv7(); + const dueAt = new Date(at.getTime() - 30_000); + const schedules = Array.from( + { length: jobSchedulePollScanLimit + 2 }, + (_, index) => + intervalSchedule({ + id: `system.worker-smoke-${String(index).padStart(3, "0")}`, + nextRunAt: dueAt, + }) + ); + const runnableSchedule = schedules.at(-1); + if (runnableSchedule === undefined) throw new Error("Missing runnable schedule"); + const newlyDueSchedule = intervalSchedule({ + id: "system.worker-smoke-newly-due", + nextRunAt: new Date(at.getTime() + 30_000), + }); + const allSchedules = [...schedules, newlyDueSchedule]; + const fixture = repositoryFixture(); + const listInputs: ListDueSchedulesInput[] = []; + const listedScheduleIds: string[][] = []; + const enqueueAttempts: DueScheduleEnqueueInput[] = []; + let clockMs = at.getTime(); + const activeRun = claimedRun(workerId); + const repository = { + ...fixture.repository, + enqueueNextDueSchedule(input: DueScheduleEnqueueInput) { + enqueueAttempts.push(input); + return Promise.resolve( + input.scheduleId === runnableSchedule.id + ? { kind: "inserted" as const, run: activeRun } + : { kind: "active" as const, run: activeRun } + ); + }, + listDueSchedules(input: ListDueSchedulesInput) { + listInputs.push(input); + const afterCursor = allSchedules.filter((schedule) => { + if (schedule.nextRunAt === null) return false; + if (schedule.nextRunAt.getTime() > input.at.getTime()) return false; + if (input.cursor === undefined) return true; + const timeDifference = + schedule.nextRunAt.getTime() - input.cursor.nextRunAt.getTime(); + return ( + timeDifference > 0 || + (timeDifference === 0 && schedule.id > input.cursor.id) + ); + }); + const page = afterCursor.slice(0, input.limit); + listedScheduleIds.push(page.map(({ id }) => id)); + if (listInputs.length === 8) clockMs = at.getTime() - 60_000; + return page; + }, + } satisfies JobWorkerCoordinatorOptions["repository"]; + const baseOptions = coordinatorOptions(repository, workerId); + const coordinator = createJobWorkerCoordinator({ + ...baseOptions, + nowMs: () => clockMs, + timings: { ...baseOptions.timings, schedulePollMs: 1 }, + }); + + await coordinator.initialize(); + await waitUntil(() => + enqueueAttempts.some(({ scheduleId }) => scheduleId === runnableSchedule.id) + ); + await coordinator.dispose(); + + expect(listInputs.length).toBeGreaterThanOrEqual(9); + expect(listInputs[0]?.cursor).toBeUndefined(); + expect(listInputs[1]?.cursor).toEqual({ + id: schedules[31]?.id, + nextRunAt: dueAt, + }); + expect(listInputs[8]?.cursor).toEqual({ + id: schedules[jobSchedulePollScanLimit - 1]?.id, + nextRunAt: dueAt, + }); + expect(listInputs[8]?.at).toEqual(at); + expect(listedScheduleIds[8]).not.toContain(newlyDueSchedule.id); + const firstTraversal = enqueueAttempts.slice(0, schedules.length); + expect(firstTraversal.map(({ scheduleId }) => scheduleId)).toEqual( + schedules.map(({ id }) => id) + ); + expect(new Set(firstTraversal.map(({ scheduleId }) => scheduleId)).size).toBe( + schedules.length + ); + expect( + fixture.events.filter((event) => event === "expire-disable-intents").length + ).toBeGreaterThanOrEqual(2); + expect(firstTraversal.at(-1)?.observedNextRunAt).toEqual(dueAt); + expect(firstTraversal.at(-1)).toMatchObject({ + at, + run: { queuedAt: at }, + }); + }); + + test("atomically invalidates a manual run's schedule for durable action events", async () => { + const workerId = Bun.randomUUIDv7(); + const scheduleId = "system.worker-smoke"; + const run: JobRunRecord = { + ...claimedRun(workerId, "test.progress"), + requestedById: Bun.randomUUIDv7(), + requestedByKind: "user", + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + triggerType: "manual", + }; + const durableEventAt = new Date(at.getTime() + 20_000); + const durableEventKinds: string[] = []; + let appendCount = 0; + const fixture = repositoryFixture({ + appendEvent: (input) => { + appendCount += 1; + if (appendCount === 5) return { kind: "dropped" }; + const event = { + attempt: run.attemptCount, + jobRunId: run.id, + kind: appendCount === 4 ? "output-truncated" : input.kind, + message: appendCount === 4 ? null : (input.message ?? null), + occurredAt: durableEventAt, + progressJson: appendCount === 4 ? null : (input.progressJson ?? null), + sequence: appendCount, + workerInstanceId: workerId, + } as const; + durableEventKinds.push(event.kind); + return appendCount === 4 + ? { event, kind: "truncated" } + : { event, kind: "appended" }; + }, + claim: { kind: "claimed", run }, + }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: "test.progress", + execute: ( + context: Parameters<(typeof jobActionRegistrations)[0]["execute"]>[0] + ) => + Effect.gen(function* () { + yield* context.reportProgress({ completed: 1 }); + yield* context.writeOutput("stdout", "safe output"); + yield* context.writeOutput("stderr", "safe diagnostic"); + yield* context.writeOutput("stdout", "truncated output"); + yield* context.writeOutput("stdout", "dropped output"); + return {}; + }), + }; + const eventInvalidations: Array<{ + readonly action: string; + readonly at: Date; + readonly targetId: string; + }> = []; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + sideEffects: { + ...sideEffects, + forRunEvent: (input) => { + eventInvalidations.push({ + action: input.action, + at: input.at, + targetId: input.targetId, + }); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "run", + operation: "updated", + }, + }); + }, + forScheduleEvent: (input) => { + eventInvalidations.push({ + action: input.action, + at: input.at, + targetId: input.targetId, + }); + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + }); + }, + }, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.events.filter((event) => event.startsWith("append:"))).toEqual([ + "append:progress", + "append:stdout", + "append:stderr", + "append:stdout", + "append:stdout", + ]); + expect(durableEventKinds).toEqual([ + "progress", + "stdout", + "stderr", + "output-truncated", + ]); + expect(eventInvalidations).toEqual([ + { action: "jobs.run.claim", at, targetId: run.id }, + { action: "jobs.run.claim", at, targetId: scheduleId }, + ...Array.from({ length: 4 }, () => [ + { action: "jobs.run.event", at: durableEventAt, targetId: run.id }, + { + action: "jobs.run.event", + at: durableEventAt, + targetId: scheduleId, + }, + ]).flat(), + ]); + expect( + fixture.eventSideEffects.map(({ auditEvents, realtimeEvents }) => ({ + auditEventCount: auditEvents.length, + realtime: realtimeEvents.map(({ entityId, topic }) => ({ + entityId, + topic, + })), + })) + ).toEqual( + Array.from({ length: 4 }, () => ({ + auditEventCount: 0, + realtime: [ + { entityId: run.id, topic: "jobs.runs" }, + { entityId: scheduleId, topic: "schedules.records" }, + ], + })) + ); + expect(fixture.settlements[0]?.outcome.kind).toBe("succeeded"); + }); + + test("settles persisted cooperative cancellation as cancelled", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.cancel"); + const fixture = repositoryFixture({ + cancellationRequested: true, + claim: { kind: "claimed", run }, + }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => Effect.never, + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toMatchObject({ + kind: "cancelled", + terminalCode: "cancel-requested", + }); + }); + + test("settles an action timeout without retry", async () => { + const workerId = Bun.randomUUIDv7(); + const run: JobRunRecord = { + ...claimedRun(workerId, "test.timeout"), + timeoutMs: 5, + }; + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => Effect.never, + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toMatchObject({ + kind: "timed-out", + terminalCode: "action-timeout", + }); + }); + + test("schedules retry only for retry-safe failed actions with attempts left", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.retry"); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => + Effect.fail(new JobActionRetryableError(new Error("private failure"))), + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + const outcome = fixture.settlements[0]?.outcome; + expect(outcome).toMatchObject({ + kind: "failed", + terminalCode: "action-failed", + terminalMessage: "The job action failed.", + }); + expect(outcome?.kind === "failed" ? outcome.retryAt : undefined).toEqual( + new Date(at.getTime() + 1000) + ); + }); + + test("does not retry permanent action failures", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.permanent-failure"); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => Effect.fail(new Error("private permanent failure")), + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.settlements.length === 1); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toEqual({ + kind: "failed", + terminalCode: "action-failed", + terminalMessage: "The job action failed.", + }); + }); + + test("interrupts and retry-safely settles active work before worker stop", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.shutdown"); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => Effect.never, + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.events.includes("read-cancellation")); + await coordinator.dispose(); + + expect(fixture.settlements[0]?.outcome).toMatchObject({ + kind: "failed", + terminalCode: "worker-shutdown", + }); + expect( + fixture.settlements[0]?.outcome.kind === "failed" + ? fixture.settlements[0].outcome.retryAt + : undefined + ).toBeInstanceOf(Date); + expect(fixture.events.indexOf("drain")).toBeLessThan( + fixture.events.indexOf("settle:failed") + ); + expect(fixture.events.indexOf("settle:failed")).toBeLessThan( + fixture.events.indexOf("stop") + ); + }); + + test("clears the forced-drain timer when interrupted work finishes", async () => { + const forceDrainMs = 123_456; + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.forced-drain-completes"); + const actionGate = deferred(); + const actionStarted = deferred(); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => + Effect.uninterruptible( + Effect.promise(() => { + actionStarted.resolve(); + return actionGate.promise; + }).pipe(Effect.as({})) + ), + }; + const options = coordinatorOptions(fixture.repository, workerId); + const coordinator = createJobWorkerCoordinator({ + ...options, + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + timings: { ...options.timings, forceDrainMs }, + }); + const setTimeoutSpy = spyOn(globalThis, "setTimeout"); + const clearTimeoutSpy = spyOn(globalThis, "clearTimeout"); + let forcedTimer: ReturnType | undefined; + try { + await coordinator.initialize(); + await actionStarted.promise; + const force = new AbortController(); + const disposal = coordinator.dispose(force.signal); + await waitUntil(() => fixture.events.includes("drain")); + + force.abort(); + await waitUntil(() => + setTimeoutSpy.mock.calls.some( + ([, milliseconds]) => milliseconds === forceDrainMs + ) + ); + const timerCall = setTimeoutSpy.mock.calls.findIndex( + ([, milliseconds]) => milliseconds === forceDrainMs + ); + const timerResult = setTimeoutSpy.mock.results[timerCall]; + if (timerResult?.type !== "return") { + throw new Error("Forced-drain timer was not created"); + } + forcedTimer = timerResult.value; + actionGate.resolve(); + await disposal; + + expect( + clearTimeoutSpy.mock.calls.some(([timer]) => timer === forcedTimer) + ).toBeTrue(); + } finally { + actionGate.resolve(); + if (forcedTimer !== undefined) clearTimeout(forcedTimer); + clearTimeoutSpy.mockRestore(); + setTimeoutSpy.mockRestore(); + } + }); + + test("fails a forced drain after its bounded timeout", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.forced-drain-timeout"); + const actionGate = deferred(); + const actionStarted = deferred(); + const fixture = repositoryFixture({ claim: { kind: "claimed", run } }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => + Effect.uninterruptible( + Effect.promise(() => { + actionStarted.resolve(); + return actionGate.promise; + }).pipe(Effect.as({})) + ), + }; + const options = coordinatorOptions(fixture.repository, workerId); + const coordinator = createJobWorkerCoordinator({ + ...options, + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + timings: { ...options.timings, forceDrainMs: 5 }, + }); + + try { + await coordinator.initialize(); + await actionStarted.promise; + const force = new AbortController(); + const disposal = coordinator + .dispose(force.signal) + .catch((error: unknown) => error); + await waitUntil(() => fixture.events.includes("drain")); + force.abort(); + + expect(await disposal).toEqual( + new Error("Durable job action exceeded forced-drain timeout") + ); + expect(fixture.events).toContain("stop"); + } finally { + actionGate.resolve(); + await waitUntil(() => fixture.settlements.length === 1); + } + }); + + test("settles a claim that resolves after worker draining begins", async () => { + const workerId = Bun.randomUUIDv7(); + const run = claimedRun(workerId, "test.deferred-claim"); + const claimGate = deferred(); + const fixture = repositoryFixture({ + claim: { kind: "claimed", run }, + claimGate: claimGate.promise, + }); + const baseRegistration = jobActionRegistrations.at(0); + if (baseRegistration === undefined) throw new Error("Missing smoke action"); + let executions = 0; + const registration: JobActionRegistration = { + ...baseRegistration, + actionKey: run.actionKey, + execute: () => { + executions += 1; + return Effect.succeed({}); + }, + }; + const coordinator = createJobWorkerCoordinator({ + ...coordinatorOptions(fixture.repository, workerId), + findAction: (actionKey) => + actionKey === registration.actionKey ? registration : undefined, + }); + + await coordinator.initialize(); + await waitUntil(() => fixture.events.includes("claim:claimed")); + const disposal = coordinator.dispose(); + await waitUntil(() => fixture.events.includes("drain")); + + expect(fixture.settlements).toHaveLength(0); + expect(fixture.events).not.toContain("stop"); + claimGate.resolve(); + await disposal; + + expect(fixture.settlements).toHaveLength(1); + expect(executions).toBe(0); + expect(fixture.settlements[0]?.outcome).toMatchObject({ + kind: "failed", + terminalCode: "worker-shutdown", + }); + expect(fixture.events.indexOf("drain")).toBeLessThan( + fixture.events.indexOf("settle:failed") + ); + expect(fixture.events.indexOf("settle:failed")).toBeLessThan( + fixture.events.indexOf("stop") + ); + }); + + test("waits for an interrupted infrastructure pass before stopping", async () => { + const workerId = Bun.randomUUIDv7(); + const expiryGate = deferred(); + const fixture = repositoryFixture({ expiryGate: expiryGate.promise }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.events.includes("expire-disable-intents")); + const disposal = coordinator.dispose(); + await waitUntil(() => fixture.events.includes("drain")); + await Bun.sleep(5); + + expect(fixture.events).not.toContain("stop"); + expiryGate.resolve(); + await disposal; + expect(fixture.events.indexOf("expire-disable-intents")).toBeLessThan( + fixture.events.indexOf("stop") + ); + }); + + test("rejects completion when a coordinator loop fails", async () => { + const workerId = Bun.randomUUIDv7(); + const failure = new Error("heartbeat failed"); + const fixture = repositoryFixture({ heartbeatFailure: failure }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + expect(await coordinator.completion.catch((error: unknown) => error)).toBe( + failure + ); + await coordinator.dispose(); + }); + + test("fails completion when bounded disable-intent expiry fails", async () => { + const workerId = Bun.randomUUIDv7(); + const failure = new Error("expiry failed"); + const fixture = repositoryFixture({ expiryFailure: failure }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + expect(await coordinator.completion.catch((error: unknown) => error)).toBe( + failure + ); + expect(fixture.events).toContain("expire-disable-intents"); + await coordinator.dispose(); + }); + + test("resumes an expired interval at its retained future dormant cursor", async () => { + const workerId = Bun.randomUUIDv7(); + const schedule = intervalSchedule({ + enabled: false, + nextRunAt: new Date(at.getTime() + 60_000), + }); + const fixture = repositoryFixture({ expiringSchedule: schedule }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.expiryNextRuns.length === 1); + await coordinator.dispose(); + + expect(fixture.expiryNextRuns).toEqual([new Date(at.getTime() + 60_000)]); + expect(fixture.expiryEligibility).toEqual([true]); + }); + + test("does not resume an expired schedule outside the exact registry pair", async () => { + const workerId = Bun.randomUUIDv7(); + const schedule = intervalSchedule({ + enabled: false, + id: "system.worker-smoke-retired", + nextRunAt: new Date(at.getTime() + 60_000), + }); + const fixture = repositoryFixture({ + dueSchedules: [schedule], + expiringSchedule: schedule, + expiryResults: [ + { + intent: expiredDisableIntent(schedule.id), + kind: "left-disabled", + schedule, + }, + ], + }); + const coordinator = createJobWorkerCoordinator( + coordinatorOptions(fixture.repository, workerId) + ); + + await coordinator.initialize(); + await waitUntil(() => fixture.events.includes("list-due")); + await coordinator.dispose(); + + expect(fixture.expiryEligibility.length).toBeGreaterThanOrEqual(1); + expect(fixture.expiryEligibility.every((eligible) => !eligible)).toBe(true); + expect(fixture.expiryNextRuns).toEqual([]); + expect(fixture.enqueues).toEqual([]); + expect(fixture.events.indexOf("expire-disable-intents")).toBeLessThan( + fixture.events.indexOf("list-due") + ); + }); +}); diff --git a/greenfield/src/server/domains/jobs/coordinator.ts b/greenfield/src/server/domains/jobs/coordinator.ts new file mode 100644 index 000000000..bdc319f14 --- /dev/null +++ b/greenfield/src/server/domains/jobs/coordinator.ts @@ -0,0 +1,1179 @@ +import { addMilliseconds, subMilliseconds } from "date-fns"; +import { Effect } from "effect"; +import * as v from "valibot"; + +import { + type JobRunResult, + jobPayloadSchema, + jobRunResultSchema, + jobWorkerFreshnessMs, +} from "../../../contracts/jobModel.ts"; +import type { JsonObject } from "../../../shared/json.ts"; +import { parseJsonText } from "../../../shared/json.ts"; +import { sha256Hex } from "../../shared/crypto.ts"; +import { + type JobActionEventWriteResult, + type JobActionRegistration, + JobActionRetryableError, + findJobActionRegistration, + jobActionRegistrations, + parseJobActionOutputMessage, + parseJobActionProgress, +} from "./actionRegistry.ts"; +import type { JobRunRecord, ScheduledJobRecord } from "./records.ts"; +import { toScheduleConfiguration } from "./records.ts"; +import { buildRegisteredSchedule } from "./registeredSchedule.ts"; +import { + type ClaimNextRunInput, + type JobMutationSideEffects, + type JobClaimOutcome, + type JobRepository, + type JobRunInsert, + type ListDueSchedulesInput, + type ScheduledJobInsert, + type WorkerLifecycleResult, + type WorkerInstanceInsert, +} from "./repository.ts"; +import { nextScheduleOccurrence } from "./scheduleTime.ts"; + +export const jobWorkerCapacity = 1; +export const jobWorkerHeartbeatIntervalMs = 10_000; +export const jobClaimLeaseMs = 120_000; +export const jobClaimRenewalIntervalMs = 30_000; +export const jobClaimCancellationPollIntervalMs = 1000; +export const jobWorkerIdlePollIntervalMs = 1000; +export const jobWorkerForceDrainMs = 5000; +export const jobSchedulePollIntervalMs = 1000; +export const jobSchedulePollLimit = 32; +export const jobSchedulePollScanLimit = jobSchedulePollLimit * 8; +export const jobDisableIntentExpiryLimit = 32; +export const jobExpiredClaimRecoveryLimit = 32; + +type JobWorkerRepository = Pick< + JobRepository, + | "appendClaimEvent" + | "beginWorkerDrain" + | "claimNextRun" + | "enqueueNextDueSchedule" + | "expireDisableIntents" + | "heartbeatWorker" + | "listDueSchedules" + | "readClaimCancellation" + | "reconcileSchedules" + | "recoverExpiredClaims" + | "registerWorker" + | "renewClaim" + | "settleClaim" + | "stopWorker" +>; + +export type JobWorkerMutationOutcome = "accepted" | "cancelled" | "failed" | "succeeded"; + +export interface JobWorkerSideEffectInput { + readonly action: string; + readonly at: Date; + readonly outcome: JobWorkerMutationOutcome; + readonly targetId: string; +} + +/** Required atomic audit/realtime rows for worker-owned durable mutations. */ +export interface JobWorkerSideEffectFactory { + forQueue(input: JobWorkerSideEffectInput): JobMutationSideEffects; + forRun(input: JobWorkerSideEffectInput): JobMutationSideEffects; + forRunEvent(input: JobWorkerSideEffectInput): JobMutationSideEffects; + forSchedule(input: JobWorkerSideEffectInput): JobMutationSideEffects; + forScheduleEvent(input: JobWorkerSideEffectInput): JobMutationSideEffects; +} + +export interface JobWorkerCoordinatorTimings { + readonly cancellationPollMs: number; + readonly claimLeaseMs: number; + readonly claimRenewalMs: number; + readonly heartbeatMs: number; + readonly idlePollMs: number; + readonly forceDrainMs: number; + readonly schedulePollMs: number; + readonly workerFreshnessMs: number; +} + +export interface JobWorkerCoordinatorOptions { + readonly databaseReleaseId: string; + readonly findAction?: (actionKey: string) => JobActionRegistration | undefined; + readonly generateId?: () => string; + readonly nowMs?: () => number; + readonly pid: number; + readonly repository: JobWorkerRepository; + readonly sideEffects: JobWorkerSideEffectFactory; + readonly timings?: Partial; + readonly workerInstanceId: string; +} + +/** Process-owned lifecycle for one single-capacity durable job coordinator. */ +export interface JobWorkerCoordinator { + readonly completion: Promise; + dispose(forceSignal?: AbortSignal): Promise; + initialize(): Promise; +} + +class JobClaimLostError extends Error { + constructor() { + super("Durable job claim was lost"); + this.name = "JobClaimLostError"; + } +} + +class JobActionCancelledError extends Error { + constructor() { + super("Durable job cancellation was requested"); + this.name = "JobActionCancelledError"; + } +} + +class JobActionTimedOutError extends Error { + constructor() { + super("Durable job action timed out"); + this.name = "JobActionTimedOutError"; + } +} + +class JobCoordinatorShutdownError extends Error { + constructor() { + super("Durable job worker is shutting down"); + this.name = "JobCoordinatorShutdownError"; + } +} + +class JobActionFinishedError extends Error { + constructor() { + super("Durable job action finished"); + this.name = "JobActionFinishedError"; + } +} + +const defaultTimings: JobWorkerCoordinatorTimings = Object.freeze({ + cancellationPollMs: jobClaimCancellationPollIntervalMs, + claimLeaseMs: jobClaimLeaseMs, + claimRenewalMs: jobClaimRenewalIntervalMs, + heartbeatMs: jobWorkerHeartbeatIntervalMs, + idlePollMs: jobWorkerIdlePollIntervalMs, + forceDrainMs: jobWorkerForceDrainMs, + schedulePollMs: jobSchedulePollIntervalMs, + workerFreshnessMs: jobWorkerFreshnessMs, +}); + +function parsePositiveDuration(value: number, name: string): number { + if (!Number.isSafeInteger(value) || value <= 0) { + throw new RangeError(`${name} is invalid`); + } + return value; +} + +function resolveTimings( + input: Partial | undefined +): JobWorkerCoordinatorTimings { + const values = { ...defaultTimings, ...input }; + return Object.freeze({ + cancellationPollMs: parsePositiveDuration( + values.cancellationPollMs, + "Job cancellation polling interval" + ), + claimLeaseMs: parsePositiveDuration(values.claimLeaseMs, "Job claim lease"), + claimRenewalMs: parsePositiveDuration( + values.claimRenewalMs, + "Job claim renewal interval" + ), + heartbeatMs: parsePositiveDuration( + values.heartbeatMs, + "Job worker heartbeat interval" + ), + idlePollMs: parsePositiveDuration( + values.idlePollMs, + "Job worker idle polling interval" + ), + forceDrainMs: parsePositiveDuration( + values.forceDrainMs, + "Job worker forced-drain timeout" + ), + schedulePollMs: parsePositiveDuration( + values.schedulePollMs, + "Job schedule polling interval" + ), + workerFreshnessMs: parsePositiveDuration( + values.workerFreshnessMs, + "Job worker freshness window" + ), + }); +} + +function waitFor(milliseconds: number, signal?: AbortSignal): Promise { + if (signal?.aborted === true) { + return Promise.reject( + signal.reason instanceof Error + ? signal.reason + : new JobCoordinatorShutdownError() + ); + } + return new Promise((resolve, reject) => { + const finish = (): void => { + signal?.removeEventListener("abort", abort); + resolve(); + }; + const timeout = setTimeout(finish, milliseconds); + const abort = (): void => { + clearTimeout(timeout); + signal?.removeEventListener("abort", abort); + reject( + signal?.reason instanceof Error + ? signal.reason + : new JobCoordinatorShutdownError() + ); + }; + signal?.addEventListener("abort", abort, { once: true }); + }); +} + +function throwIfAborted(signal: AbortSignal): void { + if (!signal.aborted) return; + throw signal.reason instanceof Error + ? signal.reason + : new JobCoordinatorShutdownError(); +} + +function mergeSideEffects( + effects: readonly JobMutationSideEffects[] +): JobMutationSideEffects { + return Object.freeze({ + auditEvents: Object.freeze(effects.flatMap((effect) => effect.auditEvents)), + realtimeEvents: Object.freeze(effects.flatMap((effect) => effect.realtimeEvents)), + }); +} + +function settlementMutationOutcome(run: JobRunRecord): JobWorkerMutationOutcome { + if (run.state === "cancelled") return "cancelled"; + if (run.state === "succeeded") return "succeeded"; + return "failed"; +} + +function durableRunTransitionSideEffects( + factory: JobWorkerSideEffectFactory, + requestedAction: string, + settled: JobRunRecord +): JobMutationSideEffects { + const outcome = settlementMutationOutcome(settled); + const action = settled.state === "cancelled" ? "jobs.run.cancelled" : requestedAction; + return mergeSideEffects([ + factory.forRun({ + action, + at: settled.updatedAt, + outcome, + targetId: settled.id, + }), + ...(settled.scheduledJobId === null + ? [] + : [ + factory.forScheduleEvent({ + action, + at: settled.updatedAt, + outcome, + targetId: settled.scheduledJobId, + }), + ]), + ]); +} + +function durableRunEventSideEffects( + factory: JobWorkerSideEffectFactory, + action: string, + run: JobRunRecord +): JobMutationSideEffects { + return mergeSideEffects([ + factory.forRunEvent({ + action, + at: run.updatedAt, + outcome: "accepted", + targetId: run.id, + }), + ...(run.scheduledJobId === null + ? [] + : [ + factory.forScheduleEvent({ + action, + at: run.updatedAt, + outcome: "accepted", + targetId: run.scheduledJobId, + }), + ]), + ]); +} + +function scheduleInsert( + registration: JobActionRegistration, + at: Date +): ScheduledJobInsert { + const schedule = buildRegisteredSchedule(registration, at); + if (schedule === undefined) { + throw new RangeError("Default job schedule has no representable occurrence"); + } + return schedule; +} + +function scheduledRunIdentity( + scheduleId: string, + scheduledForAtMs: number +): { + readonly enqueueSha256: string; + readonly idempotencyKey: string; +} { + const idempotencySource = + `mira-dashboard:schedules.run:scheduled:v1:${scheduleId}:` + + String(scheduledForAtMs); + const enqueueSource = JSON.stringify({ + procedure: "schedules.run", + scheduleId, + scheduledForAtMs, + triggerType: "schedule", + version: 1, + }); + return Object.freeze({ + enqueueSha256: sha256Hex(enqueueSource), + idempotencyKey: sha256Hex(idempotencySource), + }); +} + +function scheduledRunInsert( + schedule: ScheduledJobRecord, + at: Date, + generateId: () => string +): JobRunInsert { + if (schedule.nextRunAt === null) { + throw new Error("Due schedule is missing its durable cursor"); + } + const identity = scheduledRunIdentity(schedule.id, schedule.nextRunAt.getTime()); + return { + actionKey: schedule.actionKey, + attemptLimit: schedule.attemptLimit, + availableAt: at, + cancellationPolicy: schedule.cancellationPolicy, + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: schedule.name, + enqueueSha256: identity.enqueueSha256, + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id: generateId(), + idempotencyKey: identity.idempotencyKey, + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadJson: schedule.actionPayloadJson, + priority: schedule.priority, + queuedAt: at, + requestedById: "system.scheduler", + requestedByKind: "system", + resourceClass: schedule.resourceClass, + resourceKeysJson: schedule.resourceKeysJson, + resultJson: null, + retrySafe: schedule.retrySafe, + scheduledForAt: schedule.nextRunAt, + scheduledJobId: schedule.id, + scheduledJobVersion: schedule.version, + state: "queued", + terminalCode: null, + terminalMessage: null, + timeoutMs: schedule.timeoutMs, + triggerType: "schedule", + updatedAt: at, + }; +} + +/** + * Capped retry delay after one already-recorded failed attempt. + * @param attemptCount Current positive durable attempt count. + * @returns Delay before another retry-safe claim. + */ +export function jobRetryDelayMs(attemptCount: number): number { + if (!Number.isSafeInteger(attemptCount) || attemptCount < 1) { + throw new RangeError("Job attempt count is invalid"); + } + return Math.min(60_000, 1000 * 2 ** Math.min(attemptCount - 1, 16)); +} + +function retryAt(run: JobRunRecord, at: Date): Date { + return addMilliseconds(at, jobRetryDelayMs(run.attemptCount)); +} + +function actionFailureOutcome( + run: JobRunRecord, + at: Date, + retryable: boolean, + terminalCode: string, + terminalMessage: string +) { + return { + kind: "failed" as const, + ...(retryable && run.retrySafe && run.attemptCount < run.attemptLimit + ? { retryAt: retryAt(run, at) } + : {}), + terminalCode, + terminalMessage, + }; +} + +function executionOutcome( + run: JobRunRecord, + at: Date, + result: JobRunResult | undefined, + abortReason: unknown, + actionFailure: unknown +): JobClaimOutcome { + if (result !== undefined) { + return { kind: "succeeded", resultJson: JSON.stringify(result) }; + } + if (abortReason instanceof JobActionTimedOutError) { + return { + kind: "timed-out", + terminalCode: "action-timeout", + terminalMessage: "The job action exceeded its execution timeout.", + }; + } + if (abortReason instanceof JobActionCancelledError) { + return { + kind: "cancelled", + terminalCode: "cancel-requested", + terminalMessage: "The job action was cancelled.", + }; + } + const shutdown = abortReason instanceof JobCoordinatorShutdownError; + return actionFailureOutcome( + run, + at, + shutdown || actionFailure instanceof JobActionRetryableError, + shutdown ? "worker-shutdown" : "action-failed", + shutdown + ? "The worker stopped before the action completed." + : "The job action failed." + ); +} + +function normalizeCoordinatorFailure(error: unknown): Error { + return error instanceof Error + ? error + : new Error("Durable job coordinator failed", { cause: error }); +} + +function workerReachedState( + result: WorkerLifecycleResult, + state: "draining" | "stopped" +): boolean { + return ( + (result.kind === "updated" || result.kind === "state-changed") && + result.worker.state === state + ); +} + +async function waitForActiveExecution( + execution: Promise, + forceSignal: AbortSignal | undefined, + forceDrainMs: number +): Promise { + if (forceSignal === undefined) { + await execution; + return; + } + if (!forceSignal.aborted) { + let requestForce: (() => void) | undefined; + const forced = new Promise<"forced">((resolve) => { + requestForce = () => resolve("forced"); + }); + const onForce = (): void => requestForce?.(); + forceSignal.addEventListener("abort", onForce, { once: true }); + try { + const result = await Promise.race([ + execution.then(() => "completed" as const), + forced, + ]); + if (result === "completed") return; + } finally { + forceSignal.removeEventListener("abort", onForce); + } + } + const drainController = new AbortController(); + try { + await Promise.race([ + execution, + waitFor(forceDrainMs, drainController.signal).then(() => { + throw new Error("Durable job action exceeded forced-drain timeout"); + }), + ]); + } finally { + drainController.abort(new JobCoordinatorShutdownError()); + } +} + +interface ExecuteClaimOptions { + readonly databaseReleaseId: string; + readonly findAction: (actionKey: string) => JobActionRegistration | undefined; + readonly lifecycleSignal: AbortSignal; + readonly nowMs: () => number; + readonly repository: JobWorkerRepository; + readonly run: JobRunRecord; + readonly sideEffects: JobWorkerSideEffectFactory; + readonly timings: JobWorkerCoordinatorTimings; + readonly workerInstanceId: string; +} + +async function executeClaim(options: ExecuteClaimOptions): Promise { + const run = options.run; + const leaseToken = run.leaseToken; + const claimHeartbeatAtMs = run.heartbeatAt?.getTime(); + if ( + leaseToken === null || + claimHeartbeatAtMs === undefined || + run.leaseOwnerId !== options.workerInstanceId + ) { + throw new JobClaimLostError(); + } + const registration = options.findAction(run.actionKey); + if (registration === undefined) { + const at = new Date(options.nowMs()); + const outcome = { + kind: "failed", + terminalCode: "action-unavailable", + terminalMessage: "This release does not implement the queued action.", + } as const satisfies JobClaimOutcome; + await options.repository.settleClaim({ + at, + leaseToken, + outcome, + runId: run.id, + sideEffectsForRun: (settled) => + durableRunTransitionSideEffects( + options.sideEffects, + "jobs.run.action-unavailable", + settled + ), + workerId: options.workerInstanceId, + }); + return; + } + + const actionController = new AbortController(); + let monitorFailure: unknown; + const stopAction = (reason: unknown): void => { + if (!actionController.signal.aborted) actionController.abort(reason); + }; + const lifecycleAbort = (): void => stopAction(new JobCoordinatorShutdownError()); + options.lifecycleSignal.addEventListener("abort", lifecycleAbort, { once: true }); + if (options.lifecycleSignal.aborted) lifecycleAbort(); + const timeout = setTimeout( + () => stopAction(new JobActionTimedOutError()), + run.timeoutMs + ); + + const monitor = async (): Promise => { + let renewalDueAt = claimHeartbeatAtMs + options.timings.claimRenewalMs; + while (!actionController.signal.aborted) { + await waitFor(options.timings.cancellationPollMs, actionController.signal); + if (actionController.signal.aborted) return; + try { + const at = new Date(options.nowMs()); + const cancellation = options.repository.readClaimCancellation({ + at, + leaseToken, + runId: run.id, + workerId: options.workerInstanceId, + }); + if (!cancellation.valid) { + stopAction(new JobClaimLostError()); + return; + } + if (cancellation.cancelRequested) { + stopAction(new JobActionCancelledError()); + return; + } + if (at.getTime() < renewalDueAt) continue; + const renewal = await options.repository.renewClaim({ + at, + leaseExpiresAt: addMilliseconds(at, options.timings.claimLeaseMs), + leaseToken, + runId: run.id, + workerId: options.workerInstanceId, + }); + if (renewal.kind === "lost-claim") { + stopAction(new JobClaimLostError()); + return; + } + renewalDueAt = at.getTime() + options.timings.claimRenewalMs; + } catch (error) { + monitorFailure = error; + stopAction(error); + return; + } + } + }; + + const appendEvent = async ( + kind: "progress" | "stderr" | "stdout", + value: JsonObject | string + ): Promise => { + const at = new Date(options.nowMs()); + const result = await options.repository.appendClaimEvent({ + at, + kind, + leaseToken, + ...(kind === "progress" + ? { + progressJson: JSON.stringify( + parseJobActionProgress(value as JsonObject) + ), + } + : { message: parseJobActionOutputMessage(value as string) }), + runId: run.id, + sideEffectsForRun: (updatedRun) => + durableRunEventSideEffects( + options.sideEffects, + "jobs.run.event", + updatedRun + ), + workerId: options.workerInstanceId, + }); + if (result.kind === "lost-claim") throw new JobClaimLostError(); + return result.kind; + }; + + const action = registration.execute( + Object.freeze({ + databaseReleaseId: options.databaseReleaseId, + nowMs: options.nowMs, + reportProgress: (progress: JsonObject) => + Effect.tryPromise(() => appendEvent("progress", progress)), + workerInstanceId: options.workerInstanceId, + writeOutput: (kind: "stderr" | "stdout", message: string) => + Effect.tryPromise(() => appendEvent(kind, message)), + }), + v.parse(jobPayloadSchema, parseJsonText(run.payloadJson)) + ); + const monitorPromise = monitor().catch((error: unknown) => { + if (!actionController.signal.aborted) { + monitorFailure = error; + stopAction(error); + } + }); + + let result: JobRunResult | undefined; + let actionFailure: unknown; + try { + result = v.parse( + jobRunResultSchema, + await Effect.runPromise(action, { signal: actionController.signal }) + ); + } catch (error) { + // The public settlement below deliberately redacts action defects. + actionFailure = error; + } finally { + clearTimeout(timeout); + options.lifecycleSignal.removeEventListener("abort", lifecycleAbort); + stopAction(new JobActionFinishedError()); + await monitorPromise; + } + + if (monitorFailure !== undefined) { + throw monitorFailure instanceof Error + ? monitorFailure + : new Error("Durable job claim monitor failed", { + cause: monitorFailure, + }); + } + const abortReason: unknown = actionController.signal.reason; + if (abortReason instanceof JobClaimLostError) return; + const at = new Date(options.nowMs()); + const outcome = executionOutcome(run, at, result, abortReason, actionFailure); + await options.repository.settleClaim({ + at, + leaseToken, + outcome, + runId: run.id, + sideEffectsForRun: (settled) => + durableRunTransitionSideEffects( + options.sideEffects, + `jobs.run.${outcome.kind}`, + settled + ), + workerId: options.workerInstanceId, + }); +} + +/** + * Creates the Effect-owned coordinator for schedule polling, claims, and execution. + * @param options Repository, action registry, clock, and process identity. + * @returns Idempotent process lifecycle with an observable unexpected-failure promise. + */ +export function createJobWorkerCoordinator( + options: JobWorkerCoordinatorOptions +): JobWorkerCoordinator { + const timings = resolveTimings(options.timings); + const nowMs = options.nowMs ?? Date.now; + const generateId = options.generateId ?? (() => Bun.randomUUIDv7()); + const findAction = options.findAction ?? findJobActionRegistration; + const abortController = new AbortController(); + let activeExecution: Promise | undefined; + let initializePromise: Promise | undefined; + let disposePromise: Promise | undefined; + let programPromise: Promise | undefined; + let claimCursor: ClaimNextRunInput["cursor"]; + let dueScheduleAvailableThrough: Date | undefined; + let dueScheduleCursor: ListDueSchedulesInput["cursor"]; + const activePasses = { + claim: new Set>(), + heartbeat: new Set>(), + schedule: new Set>(), + }; + let resolveCompletion: (() => void) | undefined; + let rejectCompletion: ((error: unknown) => void) | undefined; + const completion = new Promise((resolve, reject) => { + resolveCompletion = resolve; + rejectCompletion = reject; + }); + + const trackPass = ( + kind: keyof typeof activePasses, + operation: () => Promise + ): Promise => { + const execution = operation(); + activePasses[kind].add(execution); + void execution.then( + () => activePasses[kind].delete(execution), + () => activePasses[kind].delete(execution) + ); + return execution; + }; + + const heartbeatPass = async (signal: AbortSignal): Promise => { + throwIfAborted(signal); + const worker = await options.repository.heartbeatWorker({ + at: new Date(nowMs()), + workerId: options.workerInstanceId, + }); + throwIfAborted(signal); + if (worker === undefined) throw new Error("Job worker registration was lost"); + }; + const heartbeatLoop = Effect.tryPromise({ + catch: (error) => error, + try: (signal) => trackPass("heartbeat", () => heartbeatPass(signal)), + }).pipe(Effect.andThen(Effect.sleep(timings.heartbeatMs)), Effect.forever); + + const schedulePass = async (signal: AbortSignal): Promise => { + throwIfAborted(signal); + const at = new Date(nowMs()); + const expired = await options.repository.expireDisableIntents({ + at, + canReenableSchedule: (schedule) => + findAction(schedule.actionKey)?.scheduleId === schedule.id, + limit: jobDisableIntentExpiryLimit, + nextRunAt: (schedule, after) => { + const next = nextScheduleOccurrence( + toScheduleConfiguration(schedule), + after.getTime(), + (schedule.nextRunAt ?? schedule.createdAt).getTime() + ); + return next === undefined ? undefined : new Date(next); + }, + sideEffectsForSchedule: (schedule, intent) => + options.sideEffects.forSchedule({ + action: "schedules.disable-intent-expired", + at: intent.endedAt ?? schedule.updatedAt, + outcome: "accepted", + targetId: schedule.id, + }), + systemActorId: "system.jobs-worker", + }); + throwIfAborted(signal); + const leftDisabledScheduleIds = new Set(); + for (const result of expired) { + switch (result.kind) { + case "left-disabled": { + if (result.schedule.enabled) { + throw new Error( + "Retired schedule was enabled after disable-intent expiry" + ); + } + leftDisabledScheduleIds.add(result.schedule.id); + break; + } + case "next-occurrence-unavailable": { + throw new RangeError( + "Expired disable intent has no representable next occurrence" + ); + } + case "re-enabled": { + if (!result.schedule.enabled) { + throw new Error( + "Registered schedule remained disabled after disable-intent expiry" + ); + } + break; + } + default: { + result satisfies never; + } + } + } + let enqueuedScheduleCount = 0; + let scannedScheduleCount = 0; + const availableThrough = + dueScheduleCursor === undefined ? at : (dueScheduleAvailableThrough ?? at); + dueScheduleAvailableThrough = availableThrough; + const effectiveAt = new Date(Math.max(at.getTime(), availableThrough.getTime())); + while ( + enqueuedScheduleCount < jobSchedulePollLimit && + scannedScheduleCount < jobSchedulePollScanLimit + ) { + throwIfAborted(signal); + const pageLimit = Math.min( + jobSchedulePollLimit, + jobSchedulePollScanLimit - scannedScheduleCount + ); + const schedules = options.repository.listDueSchedules({ + at: availableThrough, + ...(dueScheduleCursor === undefined ? {} : { cursor: dueScheduleCursor }), + limit: pageLimit, + }); + if (schedules.length === 0) { + dueScheduleAvailableThrough = undefined; + dueScheduleCursor = undefined; + break; + } + for (const schedule of schedules) { + throwIfAborted(signal); + if (schedule.nextRunAt === null) { + throw new Error("Due schedule is missing its keyset cursor"); + } + const cursor = { + id: schedule.id, + nextRunAt: schedule.nextRunAt, + } as const; + if (!leftDisabledScheduleIds.has(schedule.id)) { + const nextRunAtMs = nextScheduleOccurrence( + toScheduleConfiguration(schedule), + effectiveAt.getTime(), + schedule.nextRunAt.getTime() + ); + if (nextRunAtMs === undefined) { + throw new RangeError( + "Due schedule has no representable next occurrence" + ); + } + const run = scheduledRunInsert(schedule, effectiveAt, generateId); + const sideEffects = mergeSideEffects([ + options.sideEffects.forSchedule({ + action: "schedules.enqueue-due", + at: effectiveAt, + outcome: "accepted", + targetId: schedule.id, + }), + options.sideEffects.forRun({ + action: "jobs.run.enqueue-scheduled", + at: effectiveAt, + outcome: "accepted", + targetId: run.id, + }), + ]); + const result = await options.repository.enqueueNextDueSchedule({ + ...sideEffects, + at: effectiveAt, + nextRunAt: new Date(nextRunAtMs), + observedNextRunAt: schedule.nextRunAt, + run, + scheduleId: schedule.id, + }); + if (result.kind === "inserted") enqueuedScheduleCount += 1; + throwIfAborted(signal); + } + dueScheduleCursor = cursor; + scannedScheduleCount += 1; + if (enqueuedScheduleCount >= jobSchedulePollLimit) return; + } + if (schedules.length < pageLimit) { + dueScheduleAvailableThrough = undefined; + dueScheduleCursor = undefined; + break; + } + } + }; + const scheduleLoop = Effect.tryPromise({ + catch: (error) => error, + try: (signal) => trackPass("schedule", () => schedulePass(signal)), + }).pipe(Effect.andThen(Effect.sleep(timings.schedulePollMs)), Effect.forever); + + const claimPass = async (signal: AbortSignal): Promise => { + const at = new Date(nowMs()); + await options.repository.recoverExpiredClaims({ + at, + limit: jobExpiredClaimRecoveryLimit, + retryAt: (run) => retryAt(run, at), + sideEffectsForRun: (run) => + durableRunTransitionSideEffects( + options.sideEffects, + "jobs.run.lease-expired", + run + ), + }); + throwIfAborted(signal); + const leaseToken = generateId(); + const claim = await options.repository.claimNextRun({ + at, + ...(claimCursor === undefined ? {} : { cursor: claimCursor }), + leaseExpiresAt: addMilliseconds(at, timings.claimLeaseMs), + leaseToken, + minimumHeartbeatAt: subMilliseconds(at, timings.workerFreshnessMs), + sideEffectsForClaim: (run) => + mergeSideEffects([ + options.sideEffects.forQueue({ + action: "jobs.run.claim", + at: run.updatedAt, + outcome: "accepted", + targetId: options.workerInstanceId, + }), + durableRunEventSideEffects( + options.sideEffects, + "jobs.run.claim", + run + ), + ]), + workerId: options.workerInstanceId, + }); + if (claim.kind === "page-exhausted") { + claimCursor = claim.cursor; + throwIfAborted(signal); + return; + } + claimCursor = undefined; + if (claim.kind !== "claimed") { + throwIfAborted(signal); + if (claim.kind === "worker-unavailable") { + throw new Error("Job worker cannot claim durable work"); + } + await waitFor(timings.idlePollMs, signal); + return; + } + if (signal.aborted) { + const shutdownAt = new Date(nowMs()); + const outcome = actionFailureOutcome( + claim.run, + shutdownAt, + true, + "worker-shutdown", + "The worker stopped before the action completed." + ); + await options.repository.settleClaim({ + at: shutdownAt, + leaseToken, + outcome, + runId: claim.run.id, + sideEffectsForRun: (settled) => + durableRunTransitionSideEffects( + options.sideEffects, + `jobs.run.${outcome.kind}`, + settled + ), + workerId: options.workerInstanceId, + }); + return; + } + activeExecution = executeClaim({ + databaseReleaseId: options.databaseReleaseId, + findAction, + lifecycleSignal: signal, + nowMs, + repository: options.repository, + run: claim.run, + sideEffects: options.sideEffects, + timings, + workerInstanceId: options.workerInstanceId, + }); + try { + await activeExecution; + } finally { + activeExecution = undefined; + } + }; + const claimLoop = Effect.tryPromise({ + catch: (error) => error, + try: (signal) => trackPass("claim", () => claimPass(signal)), + }).pipe(Effect.forever); + + const program = Effect.all([heartbeatLoop, scheduleLoop, claimLoop], { + concurrency: "unbounded", + discard: true, + }); + + const initialize = async (): Promise => { + const at = new Date(nowMs()); + const schedules = jobActionRegistrations.map((registration) => + scheduleInsert(registration, at) + ); + await options.repository.reconcileSchedules({ + at, + retiredRunCancellation: { + actor: { id: "system.jobs-worker", kind: "system" }, + sideEffectsForRun: (run) => + durableRunTransitionSideEffects( + options.sideEffects, + "jobs.run.cancelled", + run + ), + terminalCode: "cancelled/schedule-retired", + terminalMessage: + "Cancelled because the schedule was retired from the action registry", + }, + schedules, + sideEffectsForSchedule: (schedule) => + options.sideEffects.forSchedule({ + action: "schedules.reconcile", + at: schedule.updatedAt, + outcome: "accepted", + targetId: schedule.id, + }), + }); + const worker: WorkerInstanceInsert = { + capacity: jobWorkerCapacity, + drainingAt: null, + heartbeatAt: at, + id: options.workerInstanceId, + pid: options.pid, + releaseId: options.databaseReleaseId, + startedAt: at, + state: "online", + stoppedAt: null, + }; + await options.repository.registerWorker({ + ...options.sideEffects.forQueue({ + action: "jobs.worker.register", + at, + outcome: "accepted", + targetId: options.workerInstanceId, + }), + worker, + }); + programPromise = Effect.runPromise(program, { + signal: abortController.signal, + }); + void programPromise.then( + () => { + if (disposePromise === undefined) { + rejectCompletion?.( + new Error("Durable job coordinator stopped unexpectedly") + ); + } else { + resolveCompletion?.(); + } + return; + }, + (error: unknown) => { + if (disposePromise === undefined) rejectCompletion?.(error); + else resolveCompletion?.(); + return; + } + ); + }; + + const dispose = async (forceSignal?: AbortSignal): Promise => { + if (initializePromise === undefined) { + abortController.abort(new JobCoordinatorShutdownError()); + resolveCompletion?.(); + return; + } + try { + await initializePromise; + } catch (error) { + abortController.abort(new JobCoordinatorShutdownError()); + throw normalizeCoordinatorFailure(error); + } + let failure: Error | undefined; + const drainingAt = new Date(nowMs()); + try { + const result = await options.repository.beginWorkerDrain({ + at: drainingAt, + sideEffectsForWorker: (worker) => + options.sideEffects.forQueue({ + action: "jobs.worker.drain", + at: worker.heartbeatAt, + outcome: "accepted", + targetId: worker.id, + }), + workerId: options.workerInstanceId, + }); + if (!workerReachedState(result, "draining")) { + throw new Error("Durable job worker could not enter draining state"); + } + } catch (error) { + failure = normalizeCoordinatorFailure(error); + } + abortController.abort(new JobCoordinatorShutdownError()); + let activeExecutionDrained = true; + if (activeExecution !== undefined) { + try { + await waitForActiveExecution( + activeExecution, + forceSignal, + timings.forceDrainMs + ); + } catch (error) { + activeExecutionDrained = false; + failure ??= normalizeCoordinatorFailure(error); + } + } + if (activeExecutionDrained && programPromise !== undefined) { + await programPromise.catch(() => {}); + } + const passes = [ + ...activePasses.heartbeat, + ...activePasses.schedule, + ...(activeExecutionDrained ? activePasses.claim : []), + ]; + if (passes.length > 0) await Promise.allSettled(passes); + const stoppedAt = new Date(nowMs()); + try { + const result = await options.repository.stopWorker({ + at: stoppedAt, + sideEffectsForWorker: (worker) => + options.sideEffects.forQueue({ + action: "jobs.worker.stop", + at: worker.heartbeatAt, + outcome: "succeeded", + targetId: worker.id, + }), + workerId: options.workerInstanceId, + }); + if (!workerReachedState(result, "stopped")) { + throw new Error("Durable job worker could not enter stopped state"); + } + } catch (error) { + failure ??= normalizeCoordinatorFailure(error); + } + resolveCompletion?.(); + if (failure !== undefined) throw failure; + }; + + return Object.freeze({ + completion, + dispose(forceSignal?: AbortSignal) { + disposePromise ??= dispose(forceSignal); + return disposePromise; + }, + initialize() { + if (disposePromise !== undefined) { + return Promise.reject(new Error("Durable job coordinator is disposed")); + } + initializePromise ??= initialize().catch((error: unknown) => { + const failure = normalizeCoordinatorFailure(error); + rejectCompletion?.(failure); + throw failure; + }); + return initializePromise; + }, + }); +} diff --git a/greenfield/src/server/domains/jobs/errors.ts b/greenfield/src/server/domains/jobs/errors.ts new file mode 100644 index 000000000..7e72eb631 --- /dev/null +++ b/greenfield/src/server/domains/jobs/errors.ts @@ -0,0 +1,42 @@ +import { Data } from "effect"; + +import type { DatabaseRuntimeWriteUnavailableError } from "../../database/runtime/databaseErrors.ts"; + +export type JobResourceKind = "job-run" | "schedule" | "worker-control"; + +/** Expected exact-record lookup failure in the durable jobs domain. */ +export class JobNotFoundError extends Data.TaggedError("JobNotFoundError")<{ + readonly id: string; + readonly resource: JobResourceKind; +}> {} + +/** Expected optimistic, idempotency, lifecycle, or action-policy conflict. */ +export class JobConflictError extends Data.TaggedError("JobConflictError")<{ + readonly id: string; + readonly reason: + | "action-not-manually-exposed" + | "action-unavailable" + | "cancellation-not-supported" + | "idempotency-mismatch" + | "run-already-active" + | "state-changed" + | "version-changed"; + readonly resource: JobResourceKind; +}> {} + +/** Expected time-dependent schedule input failure after write admission. */ +export class JobValidationError extends Data.TaggedError("JobValidationError")<{ + readonly id: string; + readonly reason: + | "disable-intent-expired" + | "enabled-state-unchanged" + | "next-occurrence-unavailable" + | "schedule-unchanged"; + readonly resource: "schedule"; +}> {} + +export type JobOperationError = + | DatabaseRuntimeWriteUnavailableError + | JobConflictError + | JobNotFoundError + | JobValidationError; diff --git a/greenfield/src/server/domains/jobs/procedures.test.ts b/greenfield/src/server/domains/jobs/procedures.test.ts new file mode 100644 index 000000000..b557dc914 --- /dev/null +++ b/greenfield/src/server/domains/jobs/procedures.test.ts @@ -0,0 +1,152 @@ +import { describe, expect, test } from "bun:test"; + +import { TRPCError } from "@trpc/server"; +import { Effect } from "effect"; + +import { captureFailure } from "../../test/support/promise.ts"; +import { + createTestApplicationRuntime, + createTestAutomationAuthentication, + createTestRequestContext, + createTestSessionAuthentication, +} from "../../test/support/requestContext.ts"; +import { appRouter } from "../../trpc/appRouter.ts"; +import { JobConflictError, JobNotFoundError, JobValidationError } from "./errors.ts"; +import { createTestJobService } from "./testSupport/service.ts"; + +const runId = "018f6f50-6a9e-7b88-8000-000000000001"; +const scheduleId = "system.worker-smoke"; + +const queuedRun = Object.freeze({ + actionKey: "system.worker-smoke", + attemptCount: 0, + attemptLimit: 3, + availableAtMs: 1000, + cancellationPolicy: "cooperative" as const, + displayName: "Worker smoke", + eventCount: 1, + id: runId, + priority: 0, + queuedAtMs: 1000, + resourceClass: "light" as const, + resourceKeys: ["database"], + retrySafe: true, + scheduledJobId: scheduleId, + scheduledJobVersion: 1, + state: "queued" as const, + stateVersion: 1, + timeoutMs: 30_000, + triggerType: "manual" as const, + updatedAtMs: 1000, +}); + +async function expectTrpcCode( + operation: () => Promise, + code: TRPCError["code"] +): Promise { + const failure = await captureFailure(operation); + expect(failure).toBeInstanceOf(TRPCError); + expect((failure as TRPCError).code).toBe(code); +} + +describe("durable jobs procedures", () => { + test("enforces capabilities and session-only operator mutations", async () => { + const anonymous = appRouter.createCaller(await createTestRequestContext()); + await expectTrpcCode( + () => anonymous.jobs.listRuns({ limit: 10 }), + "UNAUTHORIZED" + ); + + const missingCapability = appRouter.createCaller( + await createTestRequestContext( + createTestSessionAuthentication(["tasks:read"]) + ) + ); + await expectTrpcCode( + () => missingCapability.schedules.list({ limit: 10 }), + "FORBIDDEN" + ); + + const automationService = createTestJobService({ + runSchedule: () => Effect.succeed(queuedRun), + }); + const automation = appRouter.createCaller( + await createTestRequestContext( + createTestAutomationAuthentication(["jobs:write"]), + createTestApplicationRuntime(), + { jobService: automationService } + ) + ); + await expectTrpcCode(() => automation.jobs.cancelRun({ id: runId }), "FORBIDDEN"); + await expectTrpcCode( + () => + automation.jobs.setClaimingPaused({ + expectedVersion: 1, + paused: true, + }), + "FORBIDDEN" + ); + await expectTrpcCode( + () => + automation.schedules.update({ + expectedVersion: 1, + id: scheduleId, + patch: { + disableIntent: { + reason: "Operator-only", + }, + enabled: false, + }, + }), + "FORBIDDEN" + ); + expect( + await automation.schedules.run({ + id: scheduleId, + idempotencyKey: "A".repeat(32), + }) + ).toEqual(queuedRun); + }); + + test("maps declared domain failures without exposing implementation errors", async () => { + const service = createTestJobService({ + cancelRun: () => + Effect.fail( + new JobConflictError({ + id: runId, + reason: "state-changed", + resource: "job-run", + }) + ), + getRun: () => + Effect.fail(new JobNotFoundError({ id: runId, resource: "job-run" })), + updateSchedule: () => + Effect.fail( + new JobValidationError({ + id: scheduleId, + reason: "disable-intent-expired", + resource: "schedule", + }) + ), + }); + const caller = appRouter.createCaller( + await createTestRequestContext( + createTestSessionAuthentication(["jobs:read", "jobs:write"]), + createTestApplicationRuntime(), + { jobService: service } + ) + ); + + await expectTrpcCode(() => caller.jobs.getRun({ id: runId }), "NOT_FOUND"); + await expectTrpcCode(() => caller.jobs.cancelRun({ id: runId }), "CONFLICT"); + await expectTrpcCode( + () => + caller.schedules.update({ + expectedVersion: 1, + id: scheduleId, + patch: { schedule: { intervalMs: 60_000, kind: "interval" } }, + }), + "BAD_REQUEST" + ); + }); +}); diff --git a/greenfield/src/server/domains/jobs/procedures.ts b/greenfield/src/server/domains/jobs/procedures.ts new file mode 100644 index 000000000..7330bdcbe --- /dev/null +++ b/greenfield/src/server/domains/jobs/procedures.ts @@ -0,0 +1,14 @@ +import { router } from "../../trpc/trpc.ts"; +import { jobRoutes, scheduleRoutes } from "./routes.ts"; + +/** Leaf procedure names owned by the durable-job router. */ +export const jobProcedureNames = Object.freeze(Object.keys(jobRoutes)); + +/** Durable run inventory, cancellation, and worker-control router. */ +export const jobRouter = router(jobRoutes); + +/** Leaf procedure names owned by the Dashboard-local schedule router. */ +export const scheduleProcedureNames = Object.freeze(Object.keys(scheduleRoutes)); + +/** Dashboard-local schedule inventory, update, and run router. */ +export const scheduleRouter = router(scheduleRoutes); diff --git a/greenfield/src/server/domains/jobs/records.ts b/greenfield/src/server/domains/jobs/records.ts new file mode 100644 index 000000000..4528729bb --- /dev/null +++ b/greenfield/src/server/domains/jobs/records.ts @@ -0,0 +1,254 @@ +import { getTime } from "date-fns"; +import * as v from "valibot"; + +import { + type ActiveJobDisableIntent, + type JobRunEvent, + type JobRunResult, + type JobRunSummary, + type JobWorkerControl, + type JobWorkerSummary, + type ScheduleConfiguration, + type ScheduleSummary, + activeJobDisableIntentSchema, + jobResourceKeysSchema, + jobRunEventSchema, + jobRunResultSchema, + jobRunSummarySchema, + jobWorkerControlSchema, + jobWorkerSummarySchema, + scheduleConfigurationSchema, + scheduleSummarySchema, +} from "../../../contracts/jobModel.ts"; +import { parseJsonText } from "../../../shared/json.ts"; +import { jobDisableIntentSelectSchema } from "../../database/validation/jobDisableIntents.ts"; +import { jobRunEventSelectSchema } from "../../database/validation/jobRunEvents.ts"; +import { jobRunSelectSchema } from "../../database/validation/jobRuns.ts"; +import { jobWorkerControlSelectSchema } from "../../database/validation/jobWorkerControl.ts"; +import { scheduledJobSelectSchema } from "../../database/validation/scheduledJobs.ts"; +import { workerInstanceSelectSchema } from "../../database/validation/workerInstances.ts"; + +export type JobDisableIntentRecord = v.InferOutput; +export type JobRunEventRecord = v.InferOutput; +export type JobRunRecord = v.InferOutput; +export type JobWorkerControlRecord = v.InferOutput; +export type ScheduledJobRecord = v.InferOutput; +export type WorkerInstanceRecord = v.InferOutput; + +/** + * Converts one validated persistence run into its redacted public projection. + * @returns Contract-validated public run summary. + */ +export function toJobRunSummary(record: JobRunRecord): JobRunSummary { + return v.parse(jobRunSummarySchema, { + actionKey: record.actionKey, + attemptCount: record.attemptCount, + attemptLimit: record.attemptLimit, + availableAtMs: getTime(record.availableAt), + cancellationPolicy: record.cancellationPolicy, + ...(record.cancelRequestedAt === null + ? {} + : { cancelRequestedAtMs: getTime(record.cancelRequestedAt) }), + displayName: record.displayName, + eventCount: record.eventCount, + ...(record.finishedAt === null + ? {} + : { finishedAtMs: getTime(record.finishedAt) }), + ...(record.firstStartedAt === null + ? {} + : { firstStartedAtMs: getTime(record.firstStartedAt) }), + id: record.id, + ...(record.lastAttemptStartedAt === null + ? {} + : { lastAttemptStartedAtMs: getTime(record.lastAttemptStartedAt) }), + priority: record.priority, + queuedAtMs: getTime(record.queuedAt), + resourceClass: record.resourceClass, + resourceKeys: v.parse( + jobResourceKeysSchema, + parseJsonText(record.resourceKeysJson) + ), + retrySafe: record.retrySafe, + ...(record.scheduledForAt === null + ? {} + : { scheduledForAtMs: getTime(record.scheduledForAt) }), + ...(record.scheduledJobId === null + ? {} + : { scheduledJobId: record.scheduledJobId }), + ...(record.scheduledJobVersion === null + ? {} + : { scheduledJobVersion: record.scheduledJobVersion }), + state: record.state, + stateVersion: record.stateVersion, + ...(record.terminalCode === null ? {} : { terminalCode: record.terminalCode }), + ...(record.terminalMessage === null + ? {} + : { terminalMessage: record.terminalMessage }), + timeoutMs: record.timeoutMs, + triggerType: record.triggerType, + updatedAtMs: getTime(record.updatedAt), + }); +} + +/** + * Parses the bounded terminal result of a successful run. + * @returns The structured result when the run succeeded. + */ +export function toJobRunResult(record: JobRunRecord): JobRunResult | undefined { + return record.resultJson === null + ? undefined + : v.parse(jobRunResultSchema, parseJsonText(record.resultJson)); +} + +/** + * Converts one immutable event row into its bounded public projection. + * @returns Contract-validated public event. + */ +export function toJobRunEvent(record: JobRunEventRecord): JobRunEvent { + return v.parse(jobRunEventSchema, { + attempt: record.attempt, + kind: record.kind, + ...(record.message === null ? {} : { message: record.message }), + occurredAtMs: getTime(record.occurredAt), + ...(record.progressJson === null + ? {} + : { progress: parseJsonText(record.progressJson) }), + sequence: record.sequence, + ...(record.workerInstanceId === null + ? {} + : { workerInstanceId: record.workerInstanceId }), + }); +} + +/** + * Restores the complete mutually exclusive schedule variant from one row. + * @returns Contract-validated schedule configuration. + */ +export function toScheduleConfiguration( + record: ScheduledJobRecord +): ScheduleConfiguration { + if (record.scheduleKind === "interval") { + return v.parse(scheduleConfigurationSchema, { + intervalMs: record.intervalMs, + kind: "interval", + }); + } + if (record.scheduleKind === "daily") { + return v.parse(scheduleConfigurationSchema, { + kind: "daily", + timeOfDay: record.timeOfDay, + timeZone: record.timeZone, + }); + } + return v.parse(scheduleConfigurationSchema, { + expression: record.cronExpression, + kind: "cron", + timeZone: record.timeZone, + }); +} + +/** + * Converts one still-open operator disable intent to its public shape. + * @returns Contract-validated active disable intent. + */ +export function toActiveDisableIntent( + record: JobDisableIntentRecord +): ActiveJobDisableIntent { + if (record.endedAt !== null || record.scheduledJobId === null) { + throw new Error("Expected an active Dashboard schedule disable intent"); + } + return v.parse(activeJobDisableIntentSchema, { + createdAtMs: getTime(record.createdAt), + ...(record.expiresAt === null ? {} : { expiresAtMs: getTime(record.expiresAt) }), + id: record.id, + reason: record.reason, + }); +} + +export interface ScheduleSummaryRelations { + readonly activeDisableIntent?: JobDisableIntentRecord; + readonly activeRun?: JobRunRecord; + readonly latestRun?: JobRunRecord; +} + +/** + * Builds one contract-validated schedule projection with related run state. + * @returns Contract-validated public schedule summary. + */ +export function toScheduleSummary( + record: ScheduledJobRecord, + relations: ScheduleSummaryRelations = {} +): ScheduleSummary { + return v.parse(scheduleSummarySchema, { + actionKey: record.actionKey, + ...(relations.activeDisableIntent === undefined + ? {} + : { + activeDisableIntent: toActiveDisableIntent( + relations.activeDisableIntent + ), + }), + ...(relations.activeRun === undefined + ? {} + : { activeRun: toJobRunSummary(relations.activeRun) }), + attemptLimit: record.attemptLimit, + cancellationPolicy: record.cancellationPolicy, + createdAtMs: getTime(record.createdAt), + description: record.description, + enabled: record.enabled, + id: record.id, + ...(relations.latestRun === undefined + ? {} + : { latestRun: toJobRunSummary(relations.latestRun) }), + name: record.name, + ...(record.enabled && record.nextRunAt !== null + ? { nextRunAtMs: getTime(record.nextRunAt) } + : {}), + priority: record.priority, + resourceClass: record.resourceClass, + resourceKeys: v.parse( + jobResourceKeysSchema, + parseJsonText(record.resourceKeysJson) + ), + retrySafe: record.retrySafe, + schedule: toScheduleConfiguration(record), + timeoutMs: record.timeoutMs, + updatedAtMs: getTime(record.updatedAt), + version: record.version, + }); +} + +/** + * Converts the required singleton row to the public worker-control state. + * @returns Contract-validated public worker control. + */ +export function toJobWorkerControl(record: JobWorkerControlRecord): JobWorkerControl { + return v.parse(jobWorkerControlSchema, { + claimingPaused: record.claimingPaused, + updatedAtMs: getTime(record.updatedAt), + version: record.version, + }); +} + +/** + * Builds one bounded worker summary with a separately counted active workload. + * @returns Contract-validated public worker summary. + */ +export function toJobWorkerSummary( + record: WorkerInstanceRecord, + activeRunCount: number +): JobWorkerSummary { + return v.parse(jobWorkerSummarySchema, { + activeRunCount, + capacity: record.capacity, + ...(record.drainingAt === null + ? {} + : { drainingAtMs: getTime(record.drainingAt) }), + heartbeatAtMs: getTime(record.heartbeatAt), + id: record.id, + releaseId: record.releaseId, + startedAtMs: getTime(record.startedAt), + state: record.state, + ...(record.stoppedAt === null ? {} : { stoppedAtMs: getTime(record.stoppedAt) }), + }); +} diff --git a/greenfield/src/server/domains/jobs/registeredSchedule.ts b/greenfield/src/server/domains/jobs/registeredSchedule.ts new file mode 100644 index 000000000..bad4e35e1 --- /dev/null +++ b/greenfield/src/server/domains/jobs/registeredSchedule.ts @@ -0,0 +1,43 @@ +import type { JobActionRegistration } from "./actionRegistry.ts"; +import type { ScheduledJobInsert } from "./repository.ts"; +import { nextScheduleOccurrence } from "./scheduleTime.ts"; + +/** + * Builds the durable row for one code-owned action registration. + * @param registration Reviewed action metadata and default cadence. + * @param at Registration timestamp and next-occurrence boundary. + * @returns The complete row, or undefined when no occurrence is representable. + */ +export function buildRegisteredSchedule( + registration: JobActionRegistration, + at: Date +): ScheduledJobInsert | undefined { + const schedule = registration.defaultSchedule; + const nextRunAtMs = nextScheduleOccurrence(schedule, at.getTime()); + if (nextRunAtMs === undefined) return undefined; + + return { + actionKey: registration.actionKey, + actionPayloadJson: JSON.stringify(registration.actionPayload), + attemptLimit: registration.attemptLimit, + cancellationPolicy: registration.cancellationPolicy, + createdAt: at, + cronExpression: schedule.kind === "cron" ? schedule.expression : null, + description: registration.description, + enabled: registration.defaultEnabled, + id: registration.scheduleId, + intervalMs: schedule.kind === "interval" ? schedule.intervalMs : null, + name: registration.displayName, + nextRunAt: new Date(nextRunAtMs), + priority: registration.priority, + resourceClass: registration.resourceClass, + resourceKeysJson: JSON.stringify(registration.resourceKeys), + retrySafe: registration.retrySafe, + scheduleKind: schedule.kind, + timeOfDay: schedule.kind === "daily" ? schedule.timeOfDay : null, + timeZone: schedule.kind === "interval" ? null : schedule.timeZone, + timeoutMs: registration.timeoutMs, + updatedAt: at, + version: 1, + }; +} diff --git a/greenfield/src/server/domains/jobs/repository.test.ts b/greenfield/src/server/domains/jobs/repository.test.ts new file mode 100644 index 000000000..4471d388c --- /dev/null +++ b/greenfield/src/server/domains/jobs/repository.test.ts @@ -0,0 +1,2741 @@ +import { describe, expect, test } from "bun:test"; + +import { asc, count, eq } from "drizzle-orm"; + +import { jobWorkerSummaryMaximum } from "../../../contracts/jobModel.ts"; +import { jobRunEvents } from "../../database/schema/jobRunEvents.ts"; +import { jobRuns } from "../../database/schema/jobRuns.ts"; +import { realtimeEvents } from "../../database/schema/realtime.ts"; +import { resourceLeases } from "../../database/schema/resourceLeases.ts"; +import { scheduledJobs } from "../../database/schema/scheduledJobs.ts"; +import { workerInstances } from "../../database/schema/workerInstances.ts"; +import { testImmediateDatabaseWriteAdmission } from "../../test/support/databaseWriteAdmission.ts"; +import { openFreshMigratedDatabase } from "../../test/support/freshDatabase.ts"; +import type { WorkerInstanceRecord } from "./records.ts"; +import { + createJobRepository, + type JobMutationSideEffects, + type JobRunEventInsert, + type JobRunInsert, + type ScheduledJobInsert, + type WorkerInstanceInsert, +} from "./repository.ts"; +import { createJobRealtimeSideEffects } from "./sideEffects.ts"; + +const userId = "019fdf10-0000-7000-8000-000000000001"; +const workerOneId = "019fdf10-0000-7000-8000-000000000002"; +const workerTwoId = "019fdf10-0000-7000-8000-000000000003"; +const noSideEffects: JobMutationSideEffects = Object.freeze({ + auditEvents: Object.freeze([]), + realtimeEvents: Object.freeze([]), +}); + +function uuid(index: number): string { + return `019fdf10-0000-7000-8000-${String(index).padStart(12, "0")}`; +} + +function idempotencyKey(index: number): string { + return index.toString(16).padStart(32, "0"); +} + +function schedule(overrides: Partial = {}): ScheduledJobInsert { + return { + actionKey: "system.worker-smoke", + actionPayloadJson: "{}", + attemptLimit: 3, + cancellationPolicy: "cooperative", + createdAt: new Date(1000), + cronExpression: null, + description: "Safe worker smoke check", + enabled: true, + id: "system.worker-smoke", + intervalMs: 60_000, + name: "Worker smoke", + nextRunAt: new Date(61_000), + priority: 0, + resourceClass: "light", + resourceKeysJson: '["database"]', + retrySafe: true, + scheduleKind: "interval", + timeOfDay: null, + timeZone: null, + timeoutMs: 10_000, + updatedAt: new Date(1000), + version: 1, + ...overrides, + }; +} + +function queuedRun(index: number, overrides: Partial = {}): JobRunInsert { + const queuedAt = new Date(1000 + index); + return { + actionKey: "system.worker-smoke", + attemptLimit: 3, + availableAt: queuedAt, + cancellationPolicy: "cooperative", + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: "Worker smoke", + enqueueSha256: index.toString(16).padStart(64, "0"), + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id: uuid(index), + idempotencyKey: idempotencyKey(index), + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadJson: "{}", + priority: 0, + queuedAt, + requestedById: userId, + requestedByKind: "user", + resourceClass: "light", + resourceKeysJson: '["database"]', + resultJson: null, + retrySafe: true, + scheduledForAt: null, + scheduledJobId: "system.worker-smoke", + scheduledJobVersion: 1, + state: "queued", + terminalCode: null, + terminalMessage: null, + timeoutMs: 10_000, + triggerType: "manual", + updatedAt: queuedAt, + ...overrides, + }; +} + +function queuedEvent(run: JobRunInsert): JobRunEventInsert { + return { + attempt: 0, + jobRunId: run.id, + kind: "queued", + message: null, + occurredAt: run.queuedAt, + progressJson: null, + sequence: 1, + workerInstanceId: null, + }; +} + +function worker(id: string, capacity = 1): WorkerInstanceInsert { + return { + capacity, + drainingAt: null, + heartbeatAt: new Date(2000), + id, + pid: 1234, + releaseId: "a".repeat(40), + startedAt: new Date(2000), + state: "online", + stoppedAt: null, + }; +} + +describe("durable jobs repository", () => { + test("reconciles code metadata and enforces caller-scoped manual idempotency", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + const [created] = await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule()], + sideEffectsForSchedule: () => noSideEffects, + }); + expect(created).toMatchObject({ + enabled: true, + id: "system.worker-smoke", + version: 1, + }); + + const [reconciled] = await repository.reconcileSchedules({ + at: new Date(2000), + schedules: [ + schedule({ + description: "Updated code-owned description", + enabled: false, + nextRunAt: null, + updatedAt: new Date(2000), + }), + ], + sideEffectsForSchedule: () => noSideEffects, + }); + expect(reconciled).toMatchObject({ + description: "Updated code-owned description", + enabled: true, + nextRunAt: new Date(61_000), + version: 2, + }); + + const rejectedRun = queuedRun(9, { scheduledJobVersion: 2 }); + const rejected = repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: { + ...queuedEvent(rejectedRun), + jobRunId: uuid(99), + }, + run: rejectedRun, + }); + expect(rejected).rejects.toThrow( + "Queued event does not belong to the inserted manual run" + ); + await rejected.catch(() => {}); + expect(repository.findRun(rejectedRun.id)).toBeUndefined(); + + const run = queuedRun(10, { scheduledJobVersion: 2 }); + const inserted = await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }); + expect(inserted).toMatchObject({ kind: "inserted", run: { eventCount: 1 } }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }) + ).toMatchObject({ kind: "replayed", run: { id: run.id } }); + + const mismatch = await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent({ ...run, enqueueSha256: "f".repeat(64) }), + run: { ...run, enqueueSha256: "f".repeat(64) }, + }); + expect(mismatch.kind).toBe("idempotency-mismatch"); + const active = await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(queuedRun(11, { scheduledJobVersion: 2 })), + run: queuedRun(11, { scheduledJobVersion: 2 }), + }); + expect(active).toMatchObject({ kind: "active", run: { id: run.id } }); + + const cancelled = await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(3000), + id: run.id, + sideEffectsForRun: () => noSideEffects, + terminalCode: "job/cancelled", + terminalMessage: "Cancelled by the operator.", + }); + expect(cancelled).toMatchObject({ + kind: "cancelled", + run: { eventCount: 3, state: "cancelled" }, + }); + + const [updatedSchedule] = await repository.reconcileSchedules({ + at: new Date(4000), + schedules: [schedule({ name: "Updated worker smoke" })], + sideEffectsForSchedule: () => noSideEffects, + }); + expect(updatedSchedule).toMatchObject({ + name: "Updated worker smoke", + version: 3, + }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }) + ).toMatchObject({ kind: "replayed", run: { id: run.id } }); + + const staleRun = queuedRun(12, { scheduledJobVersion: 2 }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(staleRun), + run: staleRun, + }) + ).toEqual({ kind: "action-unavailable" }); + expect(repository.findRun(staleRun.id)).toBeUndefined(); + + const malformedCurrentRun = queuedRun(13, { + displayName: "Stale worker smoke", + scheduledJobVersion: 3, + }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(malformedCurrentRun), + run: malformedCurrentRun, + }) + ).toEqual({ kind: "action-unavailable" }); + expect(repository.findRun(malformedCurrentRun.id)).toBeUndefined(); + } finally { + database.sqlite.close(true); + } + }); + + test("pages due schedules by next occurrence and ID", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [ + schedule({ + id: "system.worker-smoke-a", + nextRunAt: new Date(60_000), + }), + schedule({ + id: "system.worker-smoke-b", + nextRunAt: new Date(60_000), + }), + schedule({ + id: "system.worker-smoke-c", + nextRunAt: new Date(65_000), + }), + schedule({ + id: "system.worker-smoke-future", + nextRunAt: new Date(70_001), + }), + ], + sideEffectsForSchedule: () => noSideEffects, + }); + + const firstPage = repository.listDueSchedules({ + at: new Date(70_000), + limit: 2, + }); + expect(firstPage.map(({ id }) => id)).toEqual([ + "system.worker-smoke-a", + "system.worker-smoke-b", + ]); + const lastSchedule = firstPage.at(-1); + if (lastSchedule === undefined || lastSchedule.nextRunAt === null) { + throw new Error("Expected a due schedule cursor"); + } + expect( + repository + .listDueSchedules({ + at: new Date(70_000), + cursor: { + id: lastSchedule.id, + nextRunAt: lastSchedule.nextRunAt, + }, + limit: 2, + }) + .map(({ id }) => id) + ).toEqual(["system.worker-smoke-c"]); + } finally { + database.sqlite.close(true); + } + }); + + test("rolls back cancellation when durable-run side effects fail", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const run = queuedRun(4000); + let sideEffectRun: ReturnType; + + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule()], + sideEffectsForSchedule: () => noSideEffects, + }); + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }); + + const rejected = repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(2000), + id: run.id, + sideEffectsForRun: (cancelled) => { + sideEffectRun = cancelled; + throw new Error("reject cancellation side effects"); + }, + terminalCode: "job/cancelled", + terminalMessage: "Cancelled by the operator.", + }); + + expect(rejected).rejects.toThrow("reject cancellation side effects"); + await rejected.catch(() => {}); + expect(sideEffectRun).toMatchObject({ + eventCount: 3, + finishedAt: new Date(5000), + state: "cancelled", + updatedAt: new Date(5000), + }); + expect(repository.findRun(run.id)).toMatchObject({ + eventCount: 1, + finishedAt: null, + state: "queued", + updatedAt: new Date(5000), + }); + expect( + repository + .listRunEvents({ limit: 10, runId: run.id }) + .map(({ kind }) => kind) + ).toEqual(["queued"]); + let successfulSideEffectCount = 0; + expect( + await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(2000), + id: run.id, + sideEffectsForRun: (cancelled) => { + successfulSideEffectCount += 1; + expect(cancelled).toMatchObject({ + state: "cancelled", + updatedAt: new Date(5000), + }); + return noSideEffects; + }, + terminalCode: "job/cancelled", + terminalMessage: "Cancelled by the operator.", + }) + ).toMatchObject({ kind: "cancelled" }); + expect( + await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(6000), + id: run.id, + sideEffectsForRun: () => { + throw new Error("terminal cancellation emitted side effects"); + }, + terminalCode: "job/cancelled", + terminalMessage: "Cancelled by the operator.", + }) + ).toMatchObject({ kind: "terminal" }); + + const [neverCancellableSchedule] = await repository.reconcileSchedules({ + at: new Date(6000), + schedules: [ + schedule({ + cancellationPolicy: "never", + updatedAt: new Date(6000), + }), + ], + sideEffectsForSchedule: () => noSideEffects, + }); + if (neverCancellableSchedule === undefined) { + throw new Error("Missing never-cancellable schedule fixture"); + } + const unsupportedRun = queuedRun(4001, { + cancellationPolicy: "never", + scheduledJobVersion: neverCancellableSchedule.version, + }); + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(unsupportedRun), + run: unsupportedRun, + }); + expect( + await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(6000), + id: unsupportedRun.id, + sideEffectsForRun: () => { + throw new Error("unsupported cancellation emitted side effects"); + }, + terminalCode: "job/cancelled", + terminalMessage: "Cancelled by the operator.", + }) + ).toMatchObject({ kind: "unsupported" }); + expect(successfulSideEffectCount).toBe(1); + expect(database.orm.select().from(realtimeEvents).all()).toEqual([]); + } finally { + database.sqlite.close(true); + } + }); + + test("retires schedules removed from the code-owned action registry", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const replacement = schedule({ + actionKey: "system.worker-smoke-v2", + createdAt: new Date(2000), + id: "system.worker-smoke-v2", + nextRunAt: new Date(62_000), + updatedAt: new Date(2000), + }); + const original = schedule({ + createdAt: new Date(10_000), + nextRunAt: new Date(70_000), + updatedAt: new Date(10_000), + }); + try { + await repository.reconcileSchedules({ + at: new Date(10_000), + schedules: [original], + sideEffectsForSchedule: () => noSideEffects, + }); + const rejectedRetirement = repository.reconcileSchedules({ + at: new Date(9000), + schedules: [], + sideEffectsForSchedule: (retired) => ({ + auditEvents: [], + realtimeEvents: [ + { + entityId: retired.id, + entityType: "schedule", + expiresAt: retired.updatedAt, + occurredAt: retired.updatedAt, + operation: "updated", + payloadJson: JSON.stringify({ id: retired.id }), + topic: "schedules.records", + }, + ], + }), + }); + expect(rejectedRetirement).rejects.toThrow(); + await rejectedRetirement.catch(() => {}); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + enabled: true, + updatedAt: new Date(10_000), + version: 1, + }); + const queuedScheduleRun = queuedRun(42, { + availableAt: new Date(100_000), + queuedAt: new Date(100_000), + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledForAt: new Date(70_000), + triggerType: "schedule", + updatedAt: new Date(100_000), + }); + expect( + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: new Date(100_000), + nextRunAt: new Date(130_000), + observedNextRunAt: new Date(70_000), + run: queuedScheduleRun, + scheduleId: "system.worker-smoke", + }) + ).toMatchObject({ kind: "inserted" }); + let rolledBackCancellation: ReturnType; + const rejectedQueuedRetirement = repository.reconcileSchedules({ + at: new Date(9000), + retiredRunCancellation: { + actor: { id: "job-scheduler", kind: "system" }, + sideEffectsForRun: (cancelled) => { + rolledBackCancellation = cancelled; + throw new Error("reject retirement cancellation side effects"); + }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: "Cancelled because the schedule was retired", + }, + schedules: [replacement], + sideEffectsForSchedule: () => noSideEffects, + }); + expect(rejectedQueuedRetirement).rejects.toThrow( + "reject retirement cancellation side effects" + ); + await rejectedQueuedRetirement.catch(() => {}); + expect(rolledBackCancellation).toMatchObject({ + eventCount: 3, + state: "cancelled", + updatedAt: new Date(100_000), + }); + expect(repository.findRun(queuedScheduleRun.id)).toMatchObject({ + eventCount: 1, + state: "queued", + updatedAt: new Date(100_000), + }); + expect(repository.findSchedule(replacement.id)).toBeUndefined(); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + enabled: true, + nextRunAt: new Date(130_000), + updatedAt: new Date(10_000), + version: 1, + }); + const reconciledRows: Array<{ + readonly id: string; + readonly updatedAt: Date; + }> = []; + let cancellationSideEffectAt: Date | undefined; + const [registered] = await repository.reconcileSchedules({ + at: new Date(9000), + retiredRunCancellation: { + actor: { id: "job-scheduler", kind: "system" }, + sideEffectsForRun: (cancelled) => { + cancellationSideEffectAt = cancelled.updatedAt; + return noSideEffects; + }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: "Cancelled because the schedule was retired", + }, + schedules: [replacement], + sideEffectsForSchedule: (changedSchedule) => { + reconciledRows.push({ + id: changedSchedule.id, + updatedAt: changedSchedule.updatedAt, + }); + return noSideEffects; + }, + }); + + expect(registered).toMatchObject({ + enabled: true, + id: "system.worker-smoke-v2", + version: 1, + }); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + enabled: false, + nextRunAt: new Date(70_000), + updatedAt: new Date(10_000), + version: 2, + }); + expect(reconciledRows).toEqual([ + { id: "system.worker-smoke-v2", updatedAt: new Date(2000) }, + { id: "system.worker-smoke", updatedAt: new Date(10_000) }, + ]); + expect(cancellationSideEffectAt).toEqual(new Date(100_000)); + expect(repository.findRun(queuedScheduleRun.id)).toMatchObject({ + state: "cancelled", + terminalCode: "cancelled/schedule-retired", + updatedAt: new Date(100_000), + }); + expect( + repository.listDueSchedules({ at: new Date(70_000) }).map(({ id }) => id) + ).toEqual(["system.worker-smoke-v2"]); + + await repository.reconcileSchedules({ + at: new Date(3000), + schedules: [replacement], + sideEffectsForSchedule: () => noSideEffects, + }); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + enabled: false, + updatedAt: new Date(10_000), + version: 2, + }); + + const [reintroduced] = await repository.reconcileSchedules({ + at: new Date(11_000), + schedules: [original, replacement], + sideEffectsForSchedule: () => noSideEffects, + }); + expect(reintroduced).toMatchObject({ + enabled: false, + id: "system.worker-smoke", + nextRunAt: new Date(70_000), + version: 2, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("expires a removed schedule intent without re-enabling its schedule", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const disableIntentId = uuid(23); + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule()], + sideEffectsForSchedule: () => noSideEffects, + }); + await repository.updateSchedule({ + ...noSideEffects, + at: new Date(2000), + expectedActiveDisableIntentId: null, + expectedVersion: 1, + id: "system.worker-smoke", + insertDisableIntent: { + createdAt: new Date(2000), + createdById: userId, + createdByKind: "user", + endedAt: null, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: new Date(3000), + externalJobId: null, + externalProvider: null, + id: disableIntentId, + reason: "Pause until after the action is retired", + scheduledJobId: "system.worker-smoke", + targetKind: "dashboard-schedule", + }, + patch: { enabled: false }, + }); + await repository.reconcileSchedules({ + at: new Date(2500), + schedules: [], + sideEffectsForSchedule: () => noSideEffects, + }); + + let nextRunCalls = 0; + let sideEffectAt: Date | undefined; + const [expired] = await repository.expireDisableIntents({ + at: new Date(4000), + canReenableSchedule: () => false, + nextRunAt: () => { + nextRunCalls += 1; + return new Date(64_000); + }, + sideEffectsForSchedule: (disabledSchedule, closedIntent) => { + expect(disabledSchedule.enabled).toBe(false); + sideEffectAt = closedIntent.endedAt ?? undefined; + return noSideEffects; + }, + systemActorId: "job-scheduler", + }); + + expect(expired).toMatchObject({ + intent: { + endedAt: new Date(4000), + endedByKind: "system", + endedReason: "expired", + id: disableIntentId, + }, + kind: "left-disabled", + schedule: { + enabled: false, + nextRunAt: new Date(61_000), + version: 2, + }, + }); + expect(nextRunCalls).toBe(0); + expect(sideEffectAt).toEqual(new Date(4000)); + expect( + repository.findActiveDisableIntent("system.worker-smoke") + ).toBeUndefined(); + expect( + await repository.updateSchedule({ + ...noSideEffects, + at: new Date(5000), + expectedActiveDisableIntentId: disableIntentId, + expectedVersion: 2, + id: "system.worker-smoke", + patch: { enabled: true, nextRunAt: new Date(65_000) }, + }) + ).toMatchObject({ kind: "version-changed" }); + } finally { + database.sqlite.close(true); + } + }); + + test("enqueues one due schedule occurrence without changing operator version", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule()], + sideEffectsForSchedule: () => noSideEffects, + }); + const run = queuedRun(20, { + availableAt: new Date(100_000), + queuedAt: new Date(100_000), + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledForAt: new Date(61_000), + triggerType: "schedule", + updatedAt: new Date(100_000), + }); + const result = await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: new Date(100_000), + nextRunAt: new Date(121_000), + observedNextRunAt: new Date(61_000), + run, + scheduleId: "system.worker-smoke", + }); + expect(result).toMatchObject({ kind: "inserted", run: { eventCount: 1 } }); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + nextRunAt: new Date(121_000), + updatedAt: new Date(1000), + version: 1, + }); + + const active = await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: new Date(200_000), + nextRunAt: new Date(241_000), + observedNextRunAt: new Date(121_000), + run: queuedRun(21, { + availableAt: new Date(200_000), + queuedAt: new Date(200_000), + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledForAt: new Date(121_000), + triggerType: "schedule", + updatedAt: new Date(200_000), + }), + scheduleId: "system.worker-smoke", + }); + expect(active).toMatchObject({ kind: "active", run: { id: run.id } }); + expect( + repository.findSchedule("system.worker-smoke")?.schedule.nextRunAt + ).toEqual(new Date(121_000)); + + const disableAt = new Date(50_000); + const disableIntent = { + createdAt: disableAt, + createdById: userId, + createdByKind: "user" as const, + endedAt: null, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: null, + externalJobId: null, + externalProvider: null, + id: uuid(22), + reason: "Operator disabled the recurring smoke check", + scheduledJobId: "system.worker-smoke", + targetKind: "dashboard-schedule" as const, + }; + let rolledBackCancellation: ReturnType; + const rejectedDisable = repository.updateSchedule({ + ...noSideEffects, + at: disableAt, + expectedActiveDisableIntentId: null, + expectedVersion: 1, + id: "system.worker-smoke", + insertDisableIntent: disableIntent, + patch: { enabled: false }, + queuedCancellation: { + at: disableAt, + terminalCode: "schedule/disabled", + terminalMessage: "The schedule was disabled before execution.", + }, + queuedCancellationSideEffects: (cancelled) => { + rolledBackCancellation = cancelled; + throw new Error("reject schedule cancellation side effects"); + }, + }); + expect(rejectedDisable).rejects.toThrow( + "reject schedule cancellation side effects" + ); + await rejectedDisable.catch(() => {}); + expect(rolledBackCancellation).toMatchObject({ + eventCount: 3, + state: "cancelled", + updatedAt: new Date(100_000), + }); + expect(repository.findRun(run.id)).toMatchObject({ + eventCount: 1, + state: "queued", + updatedAt: new Date(100_000), + }); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ enabled: true, updatedAt: new Date(1000), version: 1 }); + expect( + repository.findActiveDisableIntent("system.worker-smoke") + ).toBeUndefined(); + + let cancellationSideEffectAt: Date | undefined; + const disabled = await repository.updateSchedule({ + ...noSideEffects, + at: disableAt, + expectedActiveDisableIntentId: null, + expectedVersion: 1, + id: "system.worker-smoke", + insertDisableIntent: disableIntent, + patch: { enabled: false }, + queuedCancellation: { + at: disableAt, + terminalCode: "schedule/disabled", + terminalMessage: "The schedule was disabled before execution.", + }, + queuedCancellationSideEffects: (cancelled) => { + cancellationSideEffectAt = cancelled.updatedAt; + return noSideEffects; + }, + }); + expect(disabled).toMatchObject({ + kind: "updated", + schedule: { + enabled: false, + nextRunAt: new Date(121_000), + updatedAt: disableAt, + version: 2, + }, + }); + expect(cancellationSideEffectAt).toEqual(new Date(100_000)); + expect(repository.findRun(run.id)).toMatchObject({ + eventCount: 3, + state: "cancelled", + terminalCode: "schedule/disabled", + }); + } finally { + database.sqlite.close(true); + } + }); + + test("rejects a stale due execution snapshot before enqueueing", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule()], + sideEffectsForSchedule: () => noSideEffects, + }); + const staleRun = queuedRun(23, { + availableAt: new Date(100_000), + queuedAt: new Date(100_000), + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledForAt: new Date(61_000), + triggerType: "schedule", + updatedAt: new Date(100_000), + }); + const [updatedSchedule] = await repository.reconcileSchedules({ + at: new Date(2000), + schedules: [ + schedule({ + actionKey: "system.worker-smoke-v2", + name: "Updated worker smoke", + updatedAt: new Date(2000), + }), + ], + sideEffectsForSchedule: () => noSideEffects, + }); + expect(updatedSchedule).toMatchObject({ + actionKey: "system.worker-smoke-v2", + nextRunAt: new Date(61_000), + version: 2, + }); + + expect( + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: new Date(100_000), + nextRunAt: new Date(121_000), + observedNextRunAt: new Date(61_000), + run: staleRun, + scheduleId: "system.worker-smoke", + }) + ).toMatchObject({ kind: "state-changed", schedule: { version: 2 } }); + expect(repository.findRun(staleRun.id)).toBeUndefined(); + expect( + repository.findSchedule("system.worker-smoke")?.schedule.nextRunAt + ).toEqual(new Date(61_000)); + + const currentRun = { + ...staleRun, + actionKey: "system.worker-smoke-v2", + displayName: "Updated worker smoke", + id: uuid(24), + idempotencyKey: idempotencyKey(24), + scheduledJobVersion: 2, + }; + const currentInput = { + ...noSideEffects, + at: new Date(100_000), + nextRunAt: new Date(121_000), + observedNextRunAt: new Date(61_000), + run: currentRun, + scheduleId: "system.worker-smoke", + }; + expect(await repository.enqueueNextDueSchedule(currentInput)).toMatchObject({ + kind: "inserted", + run: { id: currentRun.id }, + }); + expect(await repository.enqueueNextDueSchedule(currentInput)).toMatchObject({ + kind: "inserted", + run: { id: currentRun.id }, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("expires disable intents atomically and resumes cadence without touching an active run", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule(), schedule({ id: "system.worker-smoke-rollback" })], + sideEffectsForSchedule: () => noSideEffects, + }); + const disableIntent = { + createdAt: new Date(2000), + createdById: userId, + createdByKind: "user" as const, + endedAt: null, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: new Date(3000), + externalJobId: null, + externalProvider: null, + id: uuid(70), + reason: "Pause while maintenance is active", + scheduledJobId: "system.worker-smoke", + targetKind: "dashboard-schedule" as const, + }; + expect( + await repository.updateSchedule({ + ...noSideEffects, + at: new Date(20_000), + expectedActiveDisableIntentId: null, + expectedVersion: 1, + id: "system.worker-smoke", + insertDisableIntent: disableIntent, + patch: { enabled: false }, + }) + ).toMatchObject({ + kind: "updated", + schedule: { enabled: false, version: 2 }, + }); + + const activeRun = queuedRun(71, { scheduledJobVersion: 2 }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(activeRun), + run: activeRun, + }) + ).toMatchObject({ kind: "inserted" }); + let expirySideEffectAt: Date | undefined; + const [expired] = await repository.expireDisableIntents({ + at: new Date(4000), + canReenableSchedule: () => true, + nextRunAt: (disabledSchedule) => { + expect(disabledSchedule.nextRunAt).toEqual(new Date(61_000)); + return new Date(64_000); + }, + sideEffectsForSchedule: (resumed) => { + expirySideEffectAt = resumed.updatedAt; + return noSideEffects; + }, + systemActorId: "job-scheduler", + }); + expect(expired).toMatchObject({ + intent: { endedByKind: "system", endedReason: "expired" }, + kind: "re-enabled", + schedule: { enabled: true, nextRunAt: new Date(64_000), version: 3 }, + }); + expect(expirySideEffectAt).toEqual(new Date(20_000)); + expect(repository.findActiveRunForSchedule("system.worker-smoke")?.id).toBe( + activeRun.id + ); + + const rollbackIntent = { + ...disableIntent, + id: uuid(72), + scheduledJobId: "system.worker-smoke-rollback", + }; + await repository.updateSchedule({ + ...noSideEffects, + at: new Date(2000), + expectedActiveDisableIntentId: null, + expectedVersion: 1, + id: "system.worker-smoke-rollback", + insertDisableIntent: rollbackIntent, + patch: { enabled: false }, + }); + const rejectedExpiry = repository.expireDisableIntents({ + at: new Date(4000), + canReenableSchedule: () => true, + nextRunAt: () => new Date(64_000), + sideEffectsForSchedule: (resumed) => ({ + auditEvents: [], + realtimeEvents: [ + { + entityId: resumed.id, + entityType: "schedule", + expiresAt: new Date(4000), + occurredAt: new Date(4000), + operation: "updated", + payloadJson: JSON.stringify({ id: resumed.id }), + topic: "schedules.records", + }, + ], + }), + systemActorId: "job-scheduler", + }); + expect(rejectedExpiry).rejects.toThrow(); + await rejectedExpiry.catch(() => {}); + expect( + repository.findSchedule("system.worker-smoke-rollback")?.schedule + ).toMatchObject({ enabled: false, version: 2 }); + expect( + repository.findActiveDisableIntent("system.worker-smoke-rollback") + ).toMatchObject({ endedAt: null, id: rollbackIntent.id }); + } finally { + database.sqlite.close(true); + } + }); + + test("rejects operator disabling but retires around a queued never-cancellable run", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [schedule({ cancellationPolicy: "never" })], + sideEffectsForSchedule: () => noSideEffects, + }); + const run = queuedRun(73, { + availableAt: new Date(100_000), + cancellationPolicy: "never", + queuedAt: new Date(100_000), + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledForAt: new Date(61_000), + triggerType: "schedule", + updatedAt: new Date(100_000), + }); + expect( + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: new Date(100_000), + nextRunAt: new Date(121_000), + observedNextRunAt: new Date(61_000), + run, + scheduleId: "system.worker-smoke", + }) + ).toMatchObject({ kind: "inserted" }); + + const result = await repository.updateSchedule({ + ...noSideEffects, + at: new Date(110_000), + expectedActiveDisableIntentId: null, + expectedVersion: 1, + id: "system.worker-smoke", + insertDisableIntent: { + createdAt: new Date(110_000), + createdById: userId, + createdByKind: "user", + endedAt: null, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: null, + externalJobId: null, + externalProvider: null, + id: uuid(74), + reason: "Maintenance must wait for the queued run", + scheduledJobId: "system.worker-smoke", + targetKind: "dashboard-schedule", + }, + patch: { enabled: false }, + queuedCancellation: { + at: new Date(110_000), + terminalCode: "schedule/disabled", + terminalMessage: "The schedule was disabled before execution.", + }, + queuedCancellationSideEffects: () => noSideEffects, + }); + + expect(result).toMatchObject({ + kind: "cancellation-not-supported", + run: { id: run.id, state: "queued" }, + }); + const unchangedSchedule = repository.findSchedule("system.worker-smoke"); + expect(unchangedSchedule?.activeDisableIntent).toBeUndefined(); + expect(unchangedSchedule?.schedule).toMatchObject({ + enabled: true, + nextRunAt: new Date(121_000), + version: 1, + }); + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 1, + state: "queued", + }); + const rejectedRetirement = repository.reconcileSchedules({ + at: new Date(900), + retiredRunCancellation: { + actor: { id: "job-scheduler", kind: "system" }, + sideEffectsForRun: () => { + throw new Error("retirement cancelled never-cancellable work"); + }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: "Cancelled because the schedule was retired", + }, + schedules: [], + sideEffectsForSchedule: () => { + throw new Error("reject retirement schedule side effects"); + }, + }); + expect(rejectedRetirement).rejects.toThrow( + "reject retirement schedule side effects" + ); + await rejectedRetirement.catch(() => {}); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + enabled: true, + version: 1, + }); + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 1, + state: "queued", + }); + + const retiredSchedules: Array<{ + readonly id: string; + readonly updatedAt: Date; + readonly version: number; + }> = []; + await repository.reconcileSchedules({ + at: new Date(900), + retiredRunCancellation: { + actor: { id: "job-scheduler", kind: "system" }, + sideEffectsForRun: () => { + throw new Error("retirement cancelled never-cancellable work"); + }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: "Cancelled because the schedule was retired", + }, + schedules: [], + sideEffectsForSchedule: (retired) => { + retiredSchedules.push({ + id: retired.id, + updatedAt: retired.updatedAt, + version: retired.version, + }); + return noSideEffects; + }, + }); + + expect(retiredSchedules).toEqual([ + { + id: "system.worker-smoke", + updatedAt: new Date(1000), + version: 2, + }, + ]); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + enabled: false, + updatedAt: new Date(1000), + version: 2, + }); + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 1, + state: "queued", + updatedAt: new Date(100_000), + }); + + await repository.reconcileSchedules({ + at: new Date(130_000), + retiredRunCancellation: { + actor: { id: "job-scheduler", kind: "system" }, + sideEffectsForRun: () => { + throw new Error("idempotent retirement touched the run"); + }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: "Cancelled because the schedule was retired", + }, + schedules: [], + sideEffectsForSchedule: () => { + throw new Error("idempotent retirement emitted schedule effects"); + }, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("retires schedules without cancelling running or manual work", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const runningScheduleId = "system.worker-smoke-running"; + const manualScheduleId = "system.worker-smoke-manual"; + const runningRun = queuedRun(75, { + availableAt: new Date(100_000), + queuedAt: new Date(100_000), + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledForAt: new Date(61_000), + scheduledJobId: runningScheduleId, + triggerType: "schedule", + updatedAt: new Date(100_000), + }); + const manualRun = queuedRun(76, { scheduledJobId: manualScheduleId }); + const leaseToken = uuid(176); + + try { + await repository.reconcileSchedules({ + at: new Date(1000), + schedules: [ + schedule({ id: runningScheduleId }), + schedule({ id: manualScheduleId }), + ], + sideEffectsForSchedule: () => noSideEffects, + }); + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: new Date(100_000), + nextRunAt: new Date(121_000), + observedNextRunAt: new Date(61_000), + run: runningRun, + scheduleId: runningScheduleId, + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + expect( + await repository.claimNextRun({ + at: new Date(101_000), + leaseExpiresAt: new Date(130_000), + leaseToken, + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "claimed", run: { id: runningRun.id } }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(manualRun), + run: manualRun, + }) + ).toMatchObject({ kind: "inserted" }); + + await repository.reconcileSchedules({ + at: new Date(50_000), + retiredRunCancellation: { + actor: { id: "job-scheduler", kind: "system" }, + sideEffectsForRun: () => { + throw new Error("retirement touched preserved work"); + }, + terminalCode: "cancelled/schedule-retired", + terminalMessage: "Cancelled because the schedule was retired", + }, + schedules: [], + sideEffectsForSchedule: () => noSideEffects, + }); + + expect(repository.findRun(runningRun.id)).toMatchObject({ + cancelRequestedAt: null, + leaseToken, + state: "running", + updatedAt: new Date(101_000), + }); + expect(repository.findRun(manualRun.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 1, + state: "queued", + }); + expect(repository.findSchedule(runningScheduleId)?.schedule).toMatchObject({ + enabled: false, + updatedAt: new Date(50_000), + version: 2, + }); + expect(repository.findSchedule(manualScheduleId)?.schedule).toMatchObject({ + enabled: false, + updatedAt: new Date(50_000), + version: 2, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("claims in total order, skips occupied resources, renews, reports, and settles", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + const runs = [ + queuedRun(30, { + requestedById: "job-scheduler", + requestedByKind: "system", + resourceKeysJson: '["database"]', + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + updatedAt: new Date(20_000), + }), + queuedRun(31, { + requestedById: "job-scheduler", + requestedByKind: "system", + resourceKeysJson: '["database"]', + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }), + queuedRun(32, { + requestedById: "job-scheduler", + requestedByKind: "system", + resourceKeysJson: '["network"]', + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }), + ]; + for (const run of runs) { + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }) + ).toMatchObject({ kind: "inserted" }); + } + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerTwoId), + }); + + let claimSideEffectAt: Date | undefined; + const firstClaim = await repository.claimNextRun({ + sideEffectsForClaim: (claimed) => { + claimSideEffectAt = claimed.updatedAt; + return noSideEffects; + }, + at: new Date(3000), + leaseExpiresAt: new Date(13_000), + leaseToken: uuid(100), + minimumHeartbeatAt: new Date(1000), + workerId: workerOneId, + }); + expect(firstClaim).toMatchObject({ + kind: "claimed", + run: { + heartbeatAt: new Date(20_000), + id: runs[0]?.id, + leaseExpiresAt: new Date(30_000), + }, + }); + expect(claimSideEffectAt).toEqual(new Date(20_000)); + expect( + await repository.appendClaimEvent({ + at: new Date(25_000), + kind: "progress", + leaseToken: uuid(100), + progressJson: '{"percent":25}', + runId: runs[0]?.id ?? "", + sideEffectsForRun: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "appended" }); + expect( + await repository.renewClaim({ + at: new Date(20_000), + leaseExpiresAt: new Date(30_000), + leaseToken: uuid(100), + runId: runs[0]?.id ?? "", + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "renewed", + run: { + heartbeatAt: new Date(25_000), + leaseExpiresAt: new Date(35_000), + updatedAt: new Date(25_000), + }, + }); + expect( + database.orm + .select() + .from(resourceLeases) + .where(eq(resourceLeases.jobRunId, runs[0]?.id ?? "")) + .get() + ).toMatchObject({ + expiresAt: new Date(35_000), + renewedAt: new Date(25_000), + }); + const secondClaim = await repository.claimNextRun({ + sideEffectsForClaim: () => noSideEffects, + at: new Date(3000), + leaseExpiresAt: new Date(13_000), + leaseToken: uuid(101), + minimumHeartbeatAt: new Date(1000), + workerId: workerTwoId, + }); + expect(secondClaim).toMatchObject({ + kind: "claimed", + run: { id: runs[2]?.id }, + }); + + expect( + await repository.renewClaim({ + at: new Date(4000), + leaseExpiresAt: new Date(20_000), + leaseToken: uuid(101), + runId: runs[2]?.id ?? "", + workerId: workerTwoId, + }) + ).toMatchObject({ kind: "renewed" }); + let eventSideEffectAt: Date | undefined; + expect( + await repository.appendClaimEvent({ + at: new Date(5000), + kind: "progress", + leaseToken: uuid(101), + progressJson: '{"percent":50}', + runId: runs[2]?.id ?? "", + sideEffectsForRun: (updated) => { + eventSideEffectAt = updated.updatedAt; + return createJobRealtimeSideEffects({ + occurredAt: updated.updatedAt, + realtime: { + id: updated.id, + kind: "run", + operation: "updated", + }, + }); + }, + workerId: workerTwoId, + }) + ).toMatchObject({ kind: "appended" }); + expect(eventSideEffectAt).toEqual(new Date(5000)); + expect(database.orm.select().from(realtimeEvents).all()).toEqual([ + expect.objectContaining({ + entityId: runs[2]?.id, + entityType: "job-run", + occurredAt: new Date(5000), + operation: "updated", + payloadJson: JSON.stringify({ id: runs[2]?.id }), + topic: "jobs.runs", + }), + ]); + expect( + await repository.settleClaim({ + sideEffectsForRun: () => noSideEffects, + at: new Date(6000), + leaseToken: uuid(101), + outcome: { kind: "succeeded", resultJson: '{"status":"ok"}' }, + runId: runs[2]?.id ?? "", + workerId: workerTwoId, + }) + ).toMatchObject({ kind: "settled", run: { state: "succeeded" } }); + expect( + database.orm + .select({ value: count() }) + .from(resourceLeases) + .where(eq(resourceLeases.jobRunId, runs[2]?.id ?? "")) + .get()?.value + ).toBe(0); + + let settlementSideEffectAt: Date | undefined; + expect( + await repository.settleClaim({ + sideEffectsForRun: (settled) => { + settlementSideEffectAt = settled.updatedAt; + return noSideEffects; + }, + at: new Date(6500), + leaseToken: uuid(100), + outcome: { kind: "succeeded", resultJson: '{"status":"ok"}' }, + runId: runs[0]?.id ?? "", + workerId: workerOneId, + }) + ).toMatchObject({ kind: "settled", run: { state: "succeeded" } }); + expect(settlementSideEffectAt).toEqual(new Date(25_000)); + + const nowUnblocked = await repository.claimNextRun({ + sideEffectsForClaim: () => noSideEffects, + at: new Date(7000), + leaseExpiresAt: new Date(17_000), + leaseToken: uuid(102), + minimumHeartbeatAt: new Date(1000), + workerId: workerTwoId, + }); + expect(nowUnblocked).toMatchObject({ + kind: "claimed", + run: { id: runs[1]?.id }, + }); + expect( + await repository.settleClaim({ + sideEffectsForRun: () => noSideEffects, + at: new Date(8000), + leaseToken: uuid(102), + outcome: { + kind: "failed", + retryAt: new Date(9000), + terminalCode: "job/retryable", + terminalMessage: "Transient dependency failure.", + }, + runId: runs[1]?.id ?? "", + workerId: workerTwoId, + }) + ).toMatchObject({ + kind: "retry-scheduled", + run: { availableAt: new Date(9000), eventCount: 4, state: "queued" }, + }); + expect( + repository + .listRunEvents({ limit: 10, runId: runs[1]?.id ?? "" }) + .map(({ kind }) => kind) + ).toEqual(["retry-scheduled", "failed", "claimed", "queued"]); + } finally { + database.sqlite.close(true); + } + }); + + test("settles a durable cancellation that races worker shutdown as cancelled", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const run = queuedRun(34, { + requestedById: "job-scheduler", + requestedByKind: "system", + resourceKeysJson: '["database"]', + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }); + const leaseToken = uuid(134); + const shutdownOutcome = { + kind: "failed" as const, + retryAt: new Date(21_000), + terminalCode: "worker-shutdown", + terminalMessage: "The worker stopped before the action completed.", + }; + try { + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + expect( + await repository.claimNextRun({ + at: new Date(3000), + leaseExpiresAt: new Date(30_000), + leaseToken, + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "claimed", run: { id: run.id } }); + let rejectedCancelRun: ReturnType; + const rejectedCancel = repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(20_000), + id: run.id, + sideEffectsForRun: (requested) => { + rejectedCancelRun = requested; + throw new Error("reject cancel-request side effects"); + }, + terminalCode: "cancelled/operator-request", + terminalMessage: "Cancelled by the operator.", + }); + expect(rejectedCancel).rejects.toThrow("reject cancel-request side effects"); + await rejectedCancel.catch(() => {}); + expect(rejectedCancelRun).toMatchObject({ + cancelRequestedAt: new Date(20_000), + eventCount: 3, + state: "running", + updatedAt: new Date(20_000), + }); + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 2, + state: "running", + updatedAt: new Date(3000), + }); + expect( + await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(20_000), + id: run.id, + sideEffectsForRun: () => noSideEffects, + terminalCode: "cancelled/operator-request", + terminalMessage: "Cancelled by the operator.", + }) + ).toMatchObject({ kind: "requested" }); + + let rolledBackSideEffectRun: + | NonNullable> + | undefined; + const rejectedSettlement = repository.settleClaim({ + at: new Date(6000), + leaseToken, + outcome: shutdownOutcome, + runId: run.id, + sideEffectsForRun: (settled) => { + rolledBackSideEffectRun = settled; + throw new Error("reject settlement side effects"); + }, + workerId: workerOneId, + }); + expect(rejectedSettlement).rejects.toThrow("reject settlement side effects"); + await rejectedSettlement.catch(() => {}); + expect(rolledBackSideEffectRun).toMatchObject({ + finishedAt: new Date(20_000), + state: "cancelled", + terminalCode: "cancel-requested", + terminalMessage: "The job action was cancelled.", + updatedAt: new Date(20_000), + }); + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: new Date(20_000), + eventCount: 3, + finishedAt: null, + leaseToken, + state: "running", + updatedAt: new Date(20_000), + }); + expect( + database.orm + .select({ value: count() }) + .from(resourceLeases) + .where(eq(resourceLeases.jobRunId, run.id)) + .get()?.value + ).toBe(1); + + let committedSideEffectRun: + | NonNullable> + | undefined; + expect( + await repository.settleClaim({ + at: new Date(6000), + leaseToken, + outcome: shutdownOutcome, + runId: run.id, + sideEffectsForRun: (settled) => { + committedSideEffectRun = settled; + return noSideEffects; + }, + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "settled", + run: { + eventCount: 4, + finishedAt: new Date(20_000), + state: "cancelled", + terminalCode: "cancel-requested", + terminalMessage: "The job action was cancelled.", + updatedAt: new Date(20_000), + }, + }); + expect(committedSideEffectRun).toMatchObject({ state: "cancelled" }); + expect( + repository + .listRunEvents({ limit: 10, runId: run.id }) + .map(({ kind, message, occurredAt }) => ({ + kind, + message, + occurredAt, + })) + ).toEqual([ + { + kind: "cancelled", + message: "The job action was cancelled.", + occurredAt: new Date(20_000), + }, + { + kind: "cancel-requested", + message: null, + occurredAt: new Date(20_000), + }, + { kind: "claimed", message: null, occurredAt: new Date(3000) }, + { + kind: "queued", + message: null, + occurredAt: run.queuedAt, + }, + ]); + expect( + database.orm + .select({ value: count() }) + .from(resourceLeases) + .where(eq(resourceLeases.jobRunId, run.id)) + .get()?.value + ).toBe(0); + + let staleSideEffectCalled = false; + expect( + await repository.settleClaim({ + at: new Date(20_001), + leaseToken, + outcome: shutdownOutcome, + runId: run.id, + sideEffectsForRun: () => { + staleSideEffectCalled = true; + return noSideEffects; + }, + workerId: workerOneId, + }) + ).toEqual({ kind: "lost-claim" }); + expect(staleSideEffectCalled).toBe(false); + } finally { + database.sqlite.close(true); + } + }); + + test("claims runnable work after a full page of resource-conflicted runs", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const systemRun = { + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + } satisfies Partial; + const resourceHolder = queuedRun(40, { + ...systemRun, + resourceKeysJson: '["database"]', + }); + try { + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(resourceHolder), + run: resourceHolder, + }) + ).toMatchObject({ kind: "inserted" }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerTwoId, 2), + }); + expect( + await repository.claimNextRun({ + at: new Date(10_000), + leaseExpiresAt: new Date(30_000), + leaseToken: uuid(140), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "claimed", + run: { id: resourceHolder.id }, + }); + + const conflictedRuns = Array.from({ length: 32 }, (_, index) => + queuedRun(100 + index, { + ...systemRun, + resourceKeysJson: '["database"]', + }) + ); + const runnableRun = queuedRun(132, { + ...systemRun, + resourceKeysJson: '["network"]', + }); + for (const run of [...conflictedRuns, runnableRun]) { + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }) + ).toMatchObject({ kind: "inserted" }); + } + + const firstPage = await repository.claimNextRun({ + at: new Date(11_000), + leaseExpiresAt: new Date(31_000), + leaseToken: uuid(141), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerTwoId, + }); + expect(firstPage).toMatchObject({ + cursor: { id: conflictedRuns.at(-1)?.id }, + kind: "page-exhausted", + }); + expect(repository.findRun(runnableRun.id)).toMatchObject({ + state: "queued", + }); + if (firstPage.kind !== "page-exhausted") { + throw new Error("Expected the bounded claim page to be exhausted"); + } + expect(firstPage.cursor.availableThrough).toEqual(new Date(11_000)); + const futureTail = queuedRun(10_500, { + ...systemRun, + resourceKeysJson: '["network.future"]', + }); + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(futureTail), + run: futureTail, + }) + ).toMatchObject({ kind: "inserted" }); + expect( + await repository.claimNextRun({ + at: new Date(12_000), + cursor: firstPage.cursor, + leaseExpiresAt: new Date(32_000), + leaseToken: uuid(142), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerTwoId, + }) + ).toMatchObject({ + kind: "claimed", + run: { id: runnableRun.id }, + }); + expect( + await repository.claimNextRun({ + at: new Date(12_000), + cursor: firstPage.cursor, + leaseExpiresAt: new Date(32_000), + leaseToken: uuid(143), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerTwoId, + }) + ).toEqual({ kind: "empty" }); + expect(repository.findRun(futureTail.id)).toMatchObject({ state: "queued" }); + expect(repository.findRun(conflictedRuns[0]?.id ?? "")).toMatchObject({ + state: "queued", + }); + expect(repository.findRun(conflictedRuns.at(-1)?.id ?? "")).toMatchObject({ + state: "queued", + }); + expect( + database.orm + .select({ + jobRunId: resourceLeases.jobRunId, + resourceKey: resourceLeases.resourceKey, + }) + .from(resourceLeases) + .all() + ).toEqual([ + { jobRunId: resourceHolder.id, resourceKey: "database" }, + { jobRunId: runnableRun.id, resourceKey: "network" }, + ]); + } finally { + database.sqlite.close(true); + } + }); + + test("concatenates every mixed-order claim cursor range", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const systemRun = { + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + } satisfies Partial; + const runAt = ( + index: number, + availableAtMs: number, + priority: number, + queuedAtMs: number, + resourceKey: string + ): JobRunInsert => + queuedRun(index, { + ...systemRun, + availableAt: new Date(availableAtMs), + priority, + queuedAt: new Date(queuedAtMs), + resourceKeysJson: JSON.stringify([resourceKey]), + updatedAt: new Date(queuedAtMs), + }); + const resourceHolder = runAt(350, 1500, 100, 1500, "database"); + const cursor = { + availableAt: new Date(5000), + availableThrough: new Date(6000), + id: uuid(400), + priority: 10, + queuedAt: new Date(4000), + } as const; + const candidates = [ + runAt(401, 5000, 10, 4000, "database"), + runAt(402, 5000, 10, 4001, "database"), + runAt(403, 5000, 9, 3000, "database"), + runAt(404, 5001, 100, 3000, "network"), + ]; + try { + for (const run of [resourceHolder, ...candidates]) { + expect( + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }) + ).toMatchObject({ kind: "inserted" }); + } + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerTwoId), + }); + expect( + await repository.claimNextRun({ + at: new Date(3000), + leaseExpiresAt: new Date(30_000), + leaseToken: uuid(450), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "claimed", + run: { id: resourceHolder.id }, + }); + + expect( + await repository.claimNextRun({ + at: new Date(6000), + cursor, + leaseExpiresAt: new Date(36_000), + leaseToken: uuid(451), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerTwoId, + }) + ).toMatchObject({ + kind: "claimed", + run: { id: candidates[3]?.id }, + }); + expect( + candidates.slice(0, 3).map((run) => repository.findRun(run.id)?.state) + ).toEqual(["queued", "queued", "queued"]); + } finally { + database.sqlite.close(true); + } + }); + + test("filters stale workers before bounding the queue summary", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const minimumHeartbeatAt = new Date(10_000); + const boundaryWorkerId = uuid(10_000); + try { + for (let index = 0; index < jobWorkerSummaryMaximum + 1; index += 1) { + await repository.registerWorker({ + ...noSideEffects, + worker: { + ...worker(uuid(200 + index)), + heartbeatAt: new Date(minimumHeartbeatAt.getTime() - 1), + }, + }); + } + await repository.registerWorker({ + ...noSideEffects, + worker: { + ...worker(boundaryWorkerId), + heartbeatAt: minimumHeartbeatAt, + }, + }); + + expect(repository.readQueueState({ minimumHeartbeatAt }).workers).toEqual([ + { + activeRunCount: 0, + worker: expect.objectContaining({ + heartbeatAt: minimumHeartbeatAt, + id: boundaryWorkerId, + state: "online", + }), + }, + ]); + } finally { + database.sqlite.close(true); + } + }); + + test("reserves the terminal event when payload consumes the byte budget", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const run = queuedRun(60, { + attemptLimit: 10, + requestedById: "job-scheduler", + requestedByKind: "system", + resourceKeysJson: "[]", + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }); + let leaseToken = uuid(160); + const fullProgressJson = JSON.stringify({ value: "x".repeat(16_372) }); + const remainingProgressJson = JSON.stringify({ value: "x".repeat(8180) }); + const terminalMessage = "😀".repeat(2000); + expect(fullProgressJson.length).toBe(16_384); + expect(remainingProgressJson.length).toBe(8192); + + try { + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + expect( + await repository.claimNextRun({ + sideEffectsForClaim: () => noSideEffects, + at: new Date(3000), + leaseExpiresAt: new Date(30_000), + leaseToken, + minimumHeartbeatAt: new Date(1000), + workerId: workerOneId, + }) + ).toMatchObject({ kind: "claimed", run: { id: run.id } }); + + for (let index = 0; index < 62; index += 1) { + expect( + await repository.appendClaimEvent({ + at: new Date(4000 + index), + kind: "progress", + leaseToken, + progressJson: + index === 61 ? remainingProgressJson : fullProgressJson, + runId: run.id, + sideEffectsForRun: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "appended" }); + } + + let truncatedSideEffects = 0; + const sideEffectsForTruncation = ( + updated: NonNullable> + ) => { + truncatedSideEffects += 1; + return createJobRealtimeSideEffects({ + occurredAt: updated.updatedAt, + realtime: { + id: updated.id, + kind: "run", + operation: "updated", + }, + }); + }; + expect( + await repository.appendClaimEvent({ + at: new Date(4100), + kind: "stdout", + leaseToken, + message: "budget exhausted", + runId: run.id, + sideEffectsForRun: sideEffectsForTruncation, + workerId: workerOneId, + }) + ).toMatchObject({ + event: { kind: "output-truncated" }, + kind: "truncated", + }); + expect( + await repository.appendClaimEvent({ + at: new Date(4101), + kind: "stdout", + leaseToken, + message: "still exhausted", + runId: run.id, + sideEffectsForRun: sideEffectsForTruncation, + workerId: workerOneId, + }) + ).toEqual({ kind: "dropped" }); + expect(truncatedSideEffects).toBe(1); + expect(database.orm.select().from(realtimeEvents).all()).toHaveLength(1); + + for (let attempt = 1; attempt < 10; attempt += 1) { + const at = new Date(5000 + attempt * 100); + const retryAt = new Date(at.getTime() + 1); + expect( + await repository.settleClaim({ + sideEffectsForRun: () => noSideEffects, + at, + leaseToken, + outcome: { + kind: "failed", + retryAt, + terminalCode: "job/retryable", + terminalMessage, + }, + runId: run.id, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "retry-scheduled" }); + leaseToken = uuid(160 + attempt); + expect( + await repository.claimNextRun({ + sideEffectsForClaim: () => noSideEffects, + at: retryAt, + leaseExpiresAt: new Date(retryAt.getTime() + 30_000), + leaseToken, + minimumHeartbeatAt: new Date(1000), + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "claimed", + run: { attemptCount: attempt + 1, id: run.id }, + }); + } + + expect( + await repository.settleClaim({ + sideEffectsForRun: () => noSideEffects, + at: new Date(7000), + leaseToken, + outcome: { + kind: "failed", + terminalCode: "job/failed", + terminalMessage, + }, + runId: run.id, + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "settled", + run: { + eventBytes: 1_048_576, + eventCount: 93, + state: "failed", + terminalMessage, + }, + }); + expect( + repository.listRunEvents({ limit: 1, runId: run.id })[0] + ).toMatchObject({ kind: "failed", message: "😀".repeat(1024) }); + } finally { + database.sqlite.close(true); + } + }); + + test("derives worker lifecycle side effects from clock-clamped durable state", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const drainAt = new Date(10_000); + const stopAt = new Date(15_000); + const callbackWorkers: Array<{ + readonly heartbeatAt: Date; + readonly state: "draining" | "stopped"; + }> = []; + try { + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + await repository.heartbeatWorker({ at: drainAt, workerId: workerOneId }); + + expect( + await repository.beginWorkerDrain({ + at: new Date(7000), + sideEffectsForWorker: (durableWorker) => { + callbackWorkers.push({ + heartbeatAt: durableWorker.heartbeatAt, + state: "draining", + }); + expect(durableWorker).toMatchObject({ + drainingAt: drainAt, + heartbeatAt: drainAt, + state: "draining", + }); + return createJobRealtimeSideEffects({ + occurredAt: durableWorker.heartbeatAt, + realtime: { id: durableWorker.id, kind: "queue" }, + }); + }, + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "updated", + worker: { + drainingAt: drainAt, + heartbeatAt: drainAt, + state: "draining", + }, + }); + await repository.heartbeatWorker({ at: stopAt, workerId: workerOneId }); + expect( + await repository.stopWorker({ + at: new Date(8000), + sideEffectsForWorker: (durableWorker) => { + callbackWorkers.push({ + heartbeatAt: durableWorker.heartbeatAt, + state: "stopped", + }); + expect(durableWorker).toMatchObject({ + drainingAt: drainAt, + heartbeatAt: stopAt, + state: "stopped", + stoppedAt: stopAt, + }); + return createJobRealtimeSideEffects({ + occurredAt: durableWorker.heartbeatAt, + realtime: { id: durableWorker.id, kind: "queue" }, + }); + }, + workerId: workerOneId, + }) + ).toMatchObject({ + kind: "updated", + worker: { + drainingAt: drainAt, + heartbeatAt: stopAt, + state: "stopped", + stoppedAt: stopAt, + }, + }); + + expect(callbackWorkers).toEqual([ + { heartbeatAt: drainAt, state: "draining" }, + { heartbeatAt: stopAt, state: "stopped" }, + ]); + expect( + database.orm + .select({ occurredAt: realtimeEvents.occurredAt }) + .from(realtimeEvents) + .where(eq(realtimeEvents.entityId, workerOneId)) + .orderBy(asc(realtimeEvents.occurredAt)) + .all() + .map(({ occurredAt }) => occurredAt) + ).toEqual([drainAt, stopAt]); + } finally { + database.sqlite.close(true); + } + }); + + test("rolls worker lifecycle mutations back when their callback fails", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const readWorker = () => + database.orm + .select() + .from(workerInstances) + .where(eq(workerInstances.id, workerOneId)) + .get(); + try { + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + let drainCallbackCalls = 0; + const rejectedDrain = repository.beginWorkerDrain({ + at: new Date(5000), + sideEffectsForWorker: (durableWorker) => { + drainCallbackCalls += 1; + expect(durableWorker).toMatchObject({ + drainingAt: new Date(5000), + heartbeatAt: new Date(5000), + state: "draining", + }); + throw new Error("reject drain side effects"); + }, + workerId: workerOneId, + }); + expect(rejectedDrain).rejects.toThrow("reject drain side effects"); + await rejectedDrain.catch(() => {}); + expect(drainCallbackCalls).toBe(1); + expect(readWorker()).toMatchObject({ + drainingAt: null, + heartbeatAt: new Date(2000), + state: "online", + stoppedAt: null, + }); + + await repository.beginWorkerDrain({ + at: new Date(5000), + sideEffectsForWorker: () => noSideEffects, + workerId: workerOneId, + }); + let stopCallbackCalls = 0; + const rejectedStop = repository.stopWorker({ + at: new Date(7000), + sideEffectsForWorker: (durableWorker) => { + stopCallbackCalls += 1; + expect(durableWorker).toMatchObject({ + heartbeatAt: new Date(7000), + state: "stopped", + stoppedAt: new Date(7000), + }); + throw new Error("reject stop side effects"); + }, + workerId: workerOneId, + }); + expect(rejectedStop).rejects.toThrow("reject stop side effects"); + await rejectedStop.catch(() => {}); + expect(stopCallbackCalls).toBe(1); + expect(readWorker()).toMatchObject({ + drainingAt: new Date(5000), + heartbeatAt: new Date(5000), + state: "draining", + stoppedAt: null, + }); + } finally { + database.sqlite.close(true); + } + }); + + test("does not invoke worker lifecycle callbacks without a state mutation", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const callbackStates: Array<"draining" | "stopped"> = []; + const callback = (durableWorker: WorkerInstanceRecord) => { + if (durableWorker.state === "online") { + throw new Error("Lifecycle callback received an unmodified worker"); + } + callbackStates.push(durableWorker.state); + return noSideEffects; + }; + try { + expect( + await repository.beginWorkerDrain({ + at: new Date(3000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toEqual({ kind: "not-found" }); + expect( + await repository.stopWorker({ + at: new Date(3000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toEqual({ kind: "not-found" }); + expect(callbackStates).toEqual([]); + + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + expect( + await repository.stopWorker({ + at: new Date(3000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "state-changed", worker: { state: "online" } }); + expect(callbackStates).toEqual([]); + + expect( + await repository.beginWorkerDrain({ + at: new Date(3000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "updated", worker: { state: "draining" } }); + expect(callbackStates).toEqual(["draining"]); + expect( + await repository.beginWorkerDrain({ + at: new Date(4000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "state-changed", worker: { state: "draining" } }); + expect(callbackStates).toEqual(["draining"]); + + expect( + await repository.stopWorker({ + at: new Date(5000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "updated", worker: { state: "stopped" } }); + expect(callbackStates).toEqual(["draining", "stopped"]); + expect( + await repository.stopWorker({ + at: new Date(6000), + sideEffectsForWorker: callback, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "state-changed", worker: { state: "stopped" } }); + expect(callbackStates).toEqual(["draining", "stopped"]); + } finally { + database.sqlite.close(true); + } + }); + + test("persists claim pause, recovers expired retry-safe work, and drains the worker", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + const run = queuedRun(80, { + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }); + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(run), + run, + }); + await repository.registerWorker({ + ...noSideEffects, + worker: worker(workerOneId), + }); + expect( + await repository.setClaimingPaused({ + ...noSideEffects, + actor: { id: userId, kind: "user" }, + at: new Date(2500), + expectedVersion: 1, + paused: true, + }) + ).toMatchObject({ control: { version: 2 }, kind: "updated" }); + expect( + await repository.claimNextRun({ + sideEffectsForClaim: () => noSideEffects, + at: new Date(3000), + leaseExpiresAt: new Date(5000), + leaseToken: uuid(81), + minimumHeartbeatAt: new Date(1000), + workerId: workerOneId, + }) + ).toEqual({ kind: "paused" }); + await repository.setClaimingPaused({ + ...noSideEffects, + actor: { id: userId, kind: "user" }, + at: new Date(2600), + expectedVersion: 2, + paused: false, + }); + expect( + await repository.claimNextRun({ + sideEffectsForClaim: () => noSideEffects, + at: new Date(3000), + leaseExpiresAt: new Date(5000), + leaseToken: uuid(81), + minimumHeartbeatAt: new Date(1000), + workerId: workerOneId, + }) + ).toMatchObject({ kind: "claimed", run: { attemptCount: 1 } }); + + const rejectedRecovery = repository.recoverExpiredClaims({ + at: new Date(6000), + retryAt: () => new Date(7000), + sideEffectsForRun: () => { + throw new Error("reject recovery side effects"); + }, + }); + expect(rejectedRecovery).rejects.toThrow("reject recovery side effects"); + await rejectedRecovery.catch(() => {}); + expect(repository.findRun(run.id)).toMatchObject({ + eventCount: 2, + leaseToken: uuid(81), + state: "running", + }); + expect( + database.orm + .select({ value: count() }) + .from(resourceLeases) + .where(eq(resourceLeases.jobRunId, run.id)) + .get()?.value + ).toBe(1); + + const recovered = await repository.recoverExpiredClaims({ + at: new Date(6000), + retryAt: () => new Date(7000), + sideEffectsForRun: () => noSideEffects, + }); + expect(recovered).toMatchObject([ + { availableAt: new Date(7000), eventCount: 4, state: "queued" }, + ]); + expect( + repository + .listRunEvents({ limit: 10, runId: run.id }) + .map(({ kind }) => kind) + ).toEqual(["retry-scheduled", "lease-expired", "claimed", "queued"]); + expect( + repository.readQueueState({ minimumHeartbeatAt: new Date(1000) }) + ).toMatchObject({ + control: { claimingPaused: false, version: 3 }, + stateCounts: { queued: 1, running: 0 }, + workers: [{ activeRunCount: 0, worker: { id: workerOneId } }], + }); + await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(7100), + id: run.id, + sideEffectsForRun: () => noSideEffects, + terminalCode: "job/cancelled", + terminalMessage: "Cancelled before the regression fixture.", + }); + const regressedRun = queuedRun(82, { + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }); + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent: queuedEvent(regressedRun), + run: regressedRun, + }); + await repository.claimNextRun({ + at: new Date(8000), + leaseExpiresAt: new Date(10_000), + leaseToken: uuid(82), + minimumHeartbeatAt: new Date(1000), + sideEffectsForClaim: () => noSideEffects, + workerId: workerOneId, + }); + await repository.cancelRun({ + actor: { id: userId, kind: "user" }, + at: new Date(20_000), + id: regressedRun.id, + sideEffectsForRun: () => noSideEffects, + terminalCode: "job/cancel-requested", + terminalMessage: "Cancel the clock-regression fixture.", + }); + let recoverySideEffectAt: Date | undefined; + expect( + await repository.recoverExpiredClaims({ + at: new Date(11_000), + retryAt: () => new Date(12_000), + sideEffectsForRun: (recoveredRun) => { + recoverySideEffectAt = recoveredRun.updatedAt; + return noSideEffects; + }, + }) + ).toMatchObject([{ id: regressedRun.id, state: "cancelled" }]); + expect(recoverySideEffectAt).toEqual(new Date(20_000)); + expect( + await repository.beginWorkerDrain({ + at: new Date(7000), + sideEffectsForWorker: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "updated", worker: { state: "draining" } }); + expect( + await repository.stopWorker({ + at: new Date(8000), + sideEffectsForWorker: () => noSideEffects, + workerId: workerOneId, + }) + ).toMatchObject({ kind: "updated", worker: { state: "stopped" } }); + } finally { + database.sqlite.close(true); + } + }); + + test("rolls a durable transition back when a required side effect is invalid", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + try { + const run = queuedRun(40, { + requestedById: "job-scheduler", + requestedByKind: "system", + scheduledJobId: null, + scheduledJobVersion: null, + triggerType: "system", + }); + const rejectedEnqueue = repository.enqueueManualRun({ + auditEvents: [], + queuedEvent: queuedEvent(run), + realtimeEvents: [ + { + entityId: run.id, + entityType: "job-run", + expiresAt: new Date(1000), + occurredAt: new Date(1000), + operation: "created", + payloadJson: JSON.stringify({ id: run.id }), + topic: "jobs.runs", + }, + ], + run, + }); + expect(rejectedEnqueue).rejects.toThrow(); + await rejectedEnqueue.catch(() => {}); + expect(repository.findRun(run.id)).toBeUndefined(); + expect( + database.orm + .select({ value: count() }) + .from(jobRunEvents) + .where(eq(jobRunEvents.jobRunId, run.id)) + .get()?.value + ).toBe(0); + expect( + database.orm + .select({ value: count() }) + .from(jobRuns) + .where(eq(jobRuns.id, run.id)) + .get()?.value + ).toBe(0); + expect( + database.orm.select({ value: count() }).from(scheduledJobs).get()?.value + ).toBe(0); + } finally { + database.sqlite.close(true); + } + }); +}); diff --git a/greenfield/src/server/domains/jobs/repository.ts b/greenfield/src/server/domains/jobs/repository.ts new file mode 100644 index 000000000..1899a63be --- /dev/null +++ b/greenfield/src/server/domains/jobs/repository.ts @@ -0,0 +1,2536 @@ +import { getTime, toDate } from "date-fns"; +import { + and, + asc, + count, + desc, + eq, + gte, + gt, + inArray, + isNotNull, + isNull, + lt, + lte, + or, + sql, + type SQL, +} from "drizzle-orm"; +import type { SQLiteBunDatabase } from "drizzle-orm/bun-sqlite"; +import * as v from "valibot"; + +import { + jobRunEventMaximum, + jobRunEventMessageMaximumBytes, + jobRunPayloadEventMaximum, + jobRunPayloadEventMaximumBytes, + jobResourceClasses, + jobResourceKeysSchema, + jobRunStates, + jobWorkerSummaryMaximum, + type JobResourceClass, + type JobRunState, + type ScheduleConfiguration, +} from "../../../contracts/jobModel.ts"; +import { + jobRunEventPageMaximum, + jobRunPageMaximum, + type ListJobRunsInput, +} from "../../../contracts/jobs.ts"; +import { + schedulePageMaximum, + type ListScheduleRunsInput, + type ListSchedulesInput, +} from "../../../contracts/schedules.ts"; +import { utf8ByteLength } from "../../../shared/encoding.ts"; +import { parseJsonText } from "../../../shared/json.ts"; +import type { ImmediateDatabaseWriteAdmission } from "../../database/immediateWriteAdmission.ts"; +import { auditEvents } from "../../database/schema/auditEvents.ts"; +import { jobDisableIntents } from "../../database/schema/jobDisableIntents.ts"; +import { jobRunEvents } from "../../database/schema/jobRunEvents.ts"; +import { jobRuns } from "../../database/schema/jobRuns.ts"; +import { jobWorkerControl } from "../../database/schema/jobWorkerControl.ts"; +import { realtimeEvents } from "../../database/schema/realtime.ts"; +import { resourceLeases } from "../../database/schema/resourceLeases.ts"; +import { scheduledJobs } from "../../database/schema/scheduledJobs.ts"; +import { workerInstances } from "../../database/schema/workerInstances.ts"; +import { auditEventInsertSchema } from "../../database/validation/auditEvents.ts"; +import { + jobDisableIntentCloseSchema, + jobDisableIntentInsertSchema, + jobDisableIntentSelectSchema, +} from "../../database/validation/jobDisableIntents.ts"; +import { + jobRunEventInsertSchema, + jobRunEventSelectSchema, +} from "../../database/validation/jobRunEvents.ts"; +import { + jobRunInsertSchema, + jobRunSelectSchema, +} from "../../database/validation/jobRuns.ts"; +import { + jobWorkerControlSelectSchema, + jobWorkerControlUpdateSchema, +} from "../../database/validation/jobWorkerControl.ts"; +import { realtimeEventInsertSchema } from "../../database/validation/realtimeEvents.ts"; +import { + resourceLeaseInsertSchema, + resourceLeaseSelectSchema, +} from "../../database/validation/resourceLeases.ts"; +import { + scheduledJobInsertSchema, + scheduledJobSelectSchema, +} from "../../database/validation/scheduledJobs.ts"; +import { + workerInstanceInsertSchema, + workerInstanceSelectSchema, +} from "../../database/validation/workerInstances.ts"; +import type { SecurityAuditEvent } from "../security/audit.ts"; +import { + type JobDisableIntentRecord, + type JobRunEventRecord, + type JobRunRecord, + type JobWorkerControlRecord, + type ScheduledJobRecord, + type WorkerInstanceRecord, +} from "./records.ts"; + +type TransactionCallback = Parameters[0]; +type JobTransaction = Parameters[0]; +type JobPersistenceDatabase = JobTransaction | SQLiteBunDatabase; + +export type JobDisableIntentInsert = v.InferOutput; +export type JobDisableIntentClose = v.InferOutput; +export type JobRunInsert = v.InferOutput; +export type JobRunEventInsert = v.InferOutput; +export type JobRealtimeEventInsert = v.InferOutput; +export type ScheduledJobInsert = v.InferOutput; +export type WorkerInstanceInsert = v.InferOutput; + +export interface JobMutationSideEffects { + readonly auditEvents: readonly SecurityAuditEvent[]; + readonly realtimeEvents: readonly JobRealtimeEventInsert[]; +} + +export interface ScheduleRecordWithRelations { + readonly activeDisableIntent?: JobDisableIntentRecord; + readonly activeRun?: JobRunRecord; + readonly latestRun?: JobRunRecord; + readonly schedule: ScheduledJobRecord; +} + +export interface JobQueueWorkerRecord { + readonly activeRunCount: number; + readonly worker: WorkerInstanceRecord; +} + +export interface JobQueueState { + readonly activeResourceClasses: readonly JobResourceClass[]; + readonly control: JobWorkerControlRecord; + readonly oldestQueuedAt?: Date; + readonly stateCounts: Readonly>; + readonly workers: readonly JobQueueWorkerRecord[]; +} + +export interface ListJobRunEventsInput { + readonly beforeSequence?: number; + readonly limit: number; + readonly runId: string; +} + +export interface JobRunDetailRecord { + readonly events: readonly JobRunEventRecord[]; + readonly run: JobRunRecord; +} + +export interface ListDueSchedulesInput { + readonly at: Date; + readonly cursor?: { + readonly id: string; + readonly nextRunAt: Date; + }; + readonly limit?: number; +} + +export interface ReadQueueStateInput { + readonly minimumHeartbeatAt: Date; +} + +export interface ListJobRunsWithQueueStateInput extends ListJobRunsInput { + readonly minimumHeartbeatAt: Date; +} + +export interface ReconcileSchedulesInput { + readonly at: Date; + readonly retiredRunCancellation?: { + readonly actor: JobActor; + readonly sideEffectsForRun: (run: JobRunRecord) => JobMutationSideEffects; + readonly terminalCode: string; + readonly terminalMessage: string; + }; + readonly schedules: readonly ScheduledJobInsert[]; + readonly sideEffectsForSchedule: ( + schedule: ScheduledJobRecord + ) => JobMutationSideEffects; +} + +export interface EnqueueManualRunInput extends JobMutationSideEffects { + readonly queuedEvent: JobRunEventInsert; + readonly run: JobRunInsert; +} + +export type EnqueueManualRunResult = + | { readonly kind: "active"; readonly run: JobRunRecord } + | { readonly kind: "action-unavailable" } + | { readonly kind: "idempotency-mismatch"; readonly run: JobRunRecord } + | { readonly kind: "inserted"; readonly run: JobRunRecord } + | { readonly kind: "replayed"; readonly run: JobRunRecord }; + +export interface ScheduleUpdateChanges { + readonly enabled?: boolean; + readonly nextRunAt?: Date | null; + readonly schedule?: ScheduleConfiguration; +} + +export interface ScheduleQueuedCancellation { + readonly at: Date; + readonly terminalCode: string; + readonly terminalMessage: string; +} + +export interface UpdateScheduleRepositoryInput extends JobMutationSideEffects { + readonly at: Date; + readonly closeActiveIntent?: JobDisableIntentClose; + readonly expectedActiveDisableIntentId: string | null; + readonly expectedVersion: number; + readonly id: string; + readonly insertDisableIntent?: JobDisableIntentInsert; + readonly patch: ScheduleUpdateChanges; + readonly queuedCancellation?: ScheduleQueuedCancellation; + readonly queuedCancellationSideEffects?: ( + run: JobRunRecord + ) => JobMutationSideEffects; +} + +export type UpdateScheduleRepositoryResult = + | { readonly kind: "cancellation-not-supported"; readonly run: JobRunRecord } + | { readonly kind: "not-found" } + | { readonly kind: "updated"; readonly schedule: ScheduledJobRecord } + | { readonly kind: "version-changed"; readonly schedule: ScheduledJobRecord }; + +export interface DueScheduleEnqueueInput extends JobMutationSideEffects { + readonly at: Date; + readonly nextRunAt: Date; + readonly observedNextRunAt: Date; + readonly run: JobRunInsert; + readonly scheduleId: string; +} + +export type DueScheduleEnqueueResult = + | { readonly kind: "active"; readonly run: JobRunRecord } + | { readonly kind: "inserted"; readonly run: JobRunRecord } + | { readonly kind: "not-due" } + | { readonly kind: "not-found" } + | { readonly kind: "state-changed"; readonly schedule: ScheduledJobRecord }; + +export interface JobActor { + readonly id: string; + readonly kind: "automation" | "system" | "user"; +} + +export interface JobOperatorActor { + readonly id: string; + readonly kind: "automation" | "user"; +} + +export interface CancelRunRepositoryInput { + readonly actor: JobActor; + readonly at: Date; + readonly id: string; + readonly sideEffectsForRun: (run: JobRunRecord) => JobMutationSideEffects; + readonly terminalCode: string; + readonly terminalMessage: string; +} + +export type CancelRunRepositoryResult = + | { readonly kind: "cancelled"; readonly run: JobRunRecord } + | { readonly kind: "not-found" } + | { readonly kind: "requested"; readonly run: JobRunRecord } + | { readonly kind: "terminal"; readonly run: JobRunRecord } + | { readonly kind: "unsupported"; readonly run: JobRunRecord }; + +export interface SetClaimingPausedRepositoryInput extends JobMutationSideEffects { + readonly actor: JobOperatorActor; + readonly at: Date; + readonly expectedVersion: number; + readonly paused: boolean; +} + +export type SetClaimingPausedRepositoryResult = + | { readonly control: JobWorkerControlRecord; readonly kind: "updated" } + | { readonly control: JobWorkerControlRecord; readonly kind: "version-changed" }; + +export interface RegisterWorkerInput extends JobMutationSideEffects { + readonly worker: WorkerInstanceInsert; +} + +export interface WorkerLifecycleInput { + readonly at: Date; + readonly workerId: string; +} + +export interface WorkerLifecycleMutationInput extends WorkerLifecycleInput { + readonly sideEffectsForWorker: ( + worker: WorkerInstanceRecord + ) => JobMutationSideEffects; +} + +export type WorkerLifecycleResult = + | { readonly kind: "active-runs"; readonly worker: WorkerInstanceRecord } + | { readonly kind: "not-found" } + | { readonly kind: "state-changed"; readonly worker: WorkerInstanceRecord } + | { readonly kind: "updated"; readonly worker: WorkerInstanceRecord }; + +export interface RecoverExpiredClaimsInput { + readonly at: Date; + readonly limit?: number; + readonly retryAt: (run: JobRunRecord) => Date; + readonly sideEffectsForRun: (run: JobRunRecord) => JobMutationSideEffects; +} + +export interface ExpireDisableIntentsInput { + readonly at: Date; + readonly canReenableSchedule: (schedule: ScheduledJobRecord) => boolean; + readonly limit?: number; + readonly nextRunAt: (schedule: ScheduledJobRecord, after: Date) => Date | undefined; + readonly sideEffectsForSchedule: ( + schedule: ScheduledJobRecord, + intent: JobDisableIntentRecord + ) => JobMutationSideEffects; + readonly systemActorId: string; +} + +export type ExpireDisableIntentResult = + | { + readonly intent: JobDisableIntentRecord; + readonly kind: "left-disabled"; + readonly schedule: ScheduledJobRecord; + } + | { + readonly intent: JobDisableIntentRecord; + readonly kind: "next-occurrence-unavailable"; + readonly schedule: ScheduledJobRecord; + } + | { + readonly intent: JobDisableIntentRecord; + readonly kind: "re-enabled"; + readonly schedule: ScheduledJobRecord; + }; + +export interface JobClaimCursor { + readonly availableAt: Date; + readonly availableThrough: Date; + readonly id: string; + readonly priority: number; + readonly queuedAt: Date; +} + +export interface ClaimNextRunInput { + readonly at: Date; + readonly cursor?: JobClaimCursor; + readonly leaseExpiresAt: Date; + readonly leaseToken: string; + readonly minimumHeartbeatAt: Date; + readonly sideEffectsForClaim: (run: JobRunRecord) => JobMutationSideEffects; + readonly workerId: string; +} + +export type JobClaimResult = + | { readonly kind: "claimed"; readonly run: JobRunRecord } + | { readonly kind: "empty" } + | { + readonly cursor: JobClaimCursor; + readonly kind: "page-exhausted"; + } + | { readonly kind: "paused" } + | { readonly kind: "worker-unavailable" }; + +export interface ClaimFenceInput { + readonly at: Date; + readonly leaseToken: string; + readonly runId: string; + readonly workerId: string; +} + +export interface RenewClaimInput extends ClaimFenceInput { + readonly leaseExpiresAt: Date; +} + +export type JobClaimMutationResult = + | { readonly kind: "lost-claim" } + | { readonly kind: "renewed"; readonly run: JobRunRecord }; + +export interface JobClaimCancellation { + readonly cancelRequested: boolean; + readonly valid: boolean; +} + +export interface AppendClaimEventInput extends ClaimFenceInput { + readonly kind: "progress" | "stderr" | "stdout"; + readonly message?: string; + readonly progressJson?: string; + readonly sideEffectsForRun: (run: JobRunRecord) => JobMutationSideEffects; +} + +export type JobAppendEventResult = + | { readonly event: JobRunEventRecord; readonly kind: "appended" } + | { readonly event?: JobRunEventRecord; readonly kind: "truncated" } + | { readonly kind: "dropped" } + | { readonly kind: "lost-claim" }; + +export type JobClaimOutcome = + | { + readonly kind: "cancelled"; + readonly terminalCode: string; + readonly terminalMessage: string; + } + | { + readonly kind: "failed"; + readonly retryAt?: Date; + readonly terminalCode: string; + readonly terminalMessage: string; + } + | { readonly kind: "succeeded"; readonly resultJson: string } + | { + readonly kind: "timed-out"; + readonly terminalCode: string; + readonly terminalMessage: string; + }; + +export interface SettleClaimInput extends ClaimFenceInput { + readonly outcome: JobClaimOutcome; + readonly sideEffectsForRun: (run: JobRunRecord) => JobMutationSideEffects; +} + +export type JobSettlementResult = + | { readonly kind: "lost-claim" } + | { readonly kind: "retry-scheduled"; readonly run: JobRunRecord } + | { readonly kind: "settled"; readonly run: JobRunRecord }; + +export interface JobRepositoryReader { + findActiveDisableIntent(scheduleId: string): JobDisableIntentRecord | undefined; + findActiveRunForSchedule(scheduleId: string): JobRunRecord | undefined; + findLatestRunForSchedule(scheduleId: string): JobRunRecord | undefined; + findRun(id: string): JobRunRecord | undefined; + findRunDetail(input: ListJobRunEventsInput): JobRunDetailRecord | undefined; + findSchedule(id: string): ScheduleRecordWithRelations | undefined; + listDueSchedules(input: ListDueSchedulesInput): ScheduledJobRecord[]; + listRunEvents(input: ListJobRunEventsInput): JobRunEventRecord[]; + listRuns(input: ListJobRunsInput): JobRunRecord[]; + listRunsWithQueueState(input: ListJobRunsWithQueueStateInput): JobRunPageSnapshot; + listScheduleRuns(input: ListScheduleRunsInput): JobRunRecord[]; + listSchedules(input: ListSchedulesInput): ScheduleRecordWithRelations[]; + readClaimCancellation(input: ClaimFenceInput): JobClaimCancellation; + readQueueState(input: ReadQueueStateInput): JobQueueState; + readWorkerControl(): JobWorkerControlRecord; +} + +/** One run page and queue summary read from the same SQLite snapshot. */ +export interface JobRunPageSnapshot { + readonly queue: JobQueueState; + readonly runs: readonly JobRunRecord[]; +} + +export interface JobRepository extends JobRepositoryReader { + appendClaimEvent(input: AppendClaimEventInput): Promise; + beginWorkerDrain(input: WorkerLifecycleMutationInput): Promise; + cancelRun(input: CancelRunRepositoryInput): Promise; + claimNextRun(input: ClaimNextRunInput): Promise; + enqueueManualRun(input: EnqueueManualRunInput): Promise; + enqueueNextDueSchedule( + input: DueScheduleEnqueueInput + ): Promise; + expireDisableIntents( + input: ExpireDisableIntentsInput + ): Promise; + heartbeatWorker( + input: WorkerLifecycleInput + ): Promise; + reconcileSchedules(input: ReconcileSchedulesInput): Promise; + recoverExpiredClaims( + input: RecoverExpiredClaimsInput + ): Promise; + registerWorker(input: RegisterWorkerInput): Promise; + renewClaim(input: RenewClaimInput): Promise; + setClaimingPaused( + input: SetClaimingPausedRepositoryInput + ): Promise; + settleClaim(input: SettleClaimInput): Promise; + stopWorker(input: WorkerLifecycleMutationInput): Promise; + updateSchedule( + input: UpdateScheduleRepositoryInput + ): Promise; +} + +const claimCandidateMaximum = 32; +const recoveryBatchMaximum = 32; +const terminalRunStates = new Set([ + "cancelled", + "failed", + "succeeded", + "timed-out", +]); + +function requiredRow(row: T | undefined, operation: string): T { + if (row === undefined) { + throw new Error(`Jobs repository ${operation} returned no row`); + } + return row; +} + +function requiredValue(value: T | null | undefined, operation: string): T { + if (value === null || value === undefined) { + throw new Error(`Jobs repository ${operation} returned no value`); + } + return value; +} + +type InternalRunEventInput = Pick< + JobRunEventInsert, + "attempt" | "kind" | "occurredAt" | "workerInstanceId" +> & { + readonly message?: string | null; + readonly progressJson?: string | null; +}; + +function parseRun(row: unknown): JobRunRecord { + return v.parse(jobRunSelectSchema, row); +} + +function parseEvent(row: unknown): JobRunEventRecord { + return v.parse(jobRunEventSelectSchema, row); +} + +function parseSchedule(row: unknown): ScheduledJobRecord { + return v.parse(scheduledJobSelectSchema, row); +} + +function runMatchesScheduleExecutionSnapshot( + run: JobRunInsert, + schedule: ScheduledJobRecord +): boolean { + return ( + run.scheduledJobId === schedule.id && + run.scheduledJobVersion === schedule.version && + run.actionKey === schedule.actionKey && + run.payloadJson === schedule.actionPayloadJson && + run.displayName === schedule.name && + run.resourceClass === schedule.resourceClass && + run.resourceKeysJson === schedule.resourceKeysJson && + run.priority === schedule.priority && + run.timeoutMs === schedule.timeoutMs && + run.attemptLimit === schedule.attemptLimit && + run.retrySafe === schedule.retrySafe && + run.cancellationPolicy === schedule.cancellationPolicy + ); +} + +function parseDisableIntent(row: unknown): JobDisableIntentRecord { + return v.parse(jobDisableIntentSelectSchema, row); +} + +function parseWorker(row: unknown): WorkerInstanceRecord { + return v.parse(workerInstanceSelectSchema, row); +} + +function assertLimit(limit: number, maximum: number, operation: string): void { + if (!Number.isSafeInteger(limit) || limit < 1 || limit > maximum) { + throw new RangeError(`Jobs repository ${operation} limit is invalid`); + } +} + +function maximumDate(...dates: readonly Date[]): Date { + return toDate(Math.max(...dates.map((date) => getTime(date)))); +} + +function boundedStructuralMessage(message: string): string { + let byteLength = 0; + let bounded = ""; + for (const codePoint of message) { + const codePointBytes = utf8ByteLength(codePoint); + if (byteLength + codePointBytes > jobRunEventMessageMaximumBytes) break; + bounded += codePoint; + byteLength += codePointBytes; + } + return bounded; +} + +function shiftDurationToStart( + requestedStart: Date, + requestedEnd: Date, + effectiveStart: Date +): Date { + const duration = getTime(requestedEnd) - getTime(requestedStart); + if (duration <= 0) { + throw new RangeError("Job lease duration must be positive"); + } + const shifted = toDate(getTime(effectiveStart) + duration); + if (Number.isNaN(getTime(shifted))) { + throw new RangeError("Job lease expiry is not representable"); + } + return shifted; +} + +function resourceKeys(record: Pick): string[] { + return v.parse(jobResourceKeysSchema, parseJsonText(record.resourceKeysJson)); +} + +function runCursorBoundary( + input: ListJobRunsInput | ListScheduleRunsInput +): SQL | undefined { + if (input.cursor === undefined) return undefined; + const queuedAt = toDate(input.cursor.queuedAtMs); + return or( + lt(jobRuns.queuedAt, queuedAt), + and(eq(jobRuns.queuedAt, queuedAt), lt(jobRuns.id, input.cursor.id)) + ); +} + +function runFilterConditions(input: ListJobRunsInput): SQL[] { + const filters = input.filters; + if (filters === undefined) return []; + return [ + ...(filters.resourceClasses === undefined + ? [] + : [inArray(jobRuns.resourceClass, [...filters.resourceClasses])]), + ...(filters.scheduleId === undefined + ? [] + : [eq(jobRuns.scheduledJobId, filters.scheduleId)]), + ...(filters.states === undefined + ? [] + : [inArray(jobRuns.state, [...filters.states])]), + ...(filters.triggerTypes === undefined + ? [] + : [inArray(jobRuns.triggerType, [...filters.triggerTypes])]), + ]; +} + +function terminalStateForOutcome( + outcome: JobClaimOutcome +): Exclude { + switch (outcome.kind) { + case "succeeded": { + return "succeeded"; + } + case "timed-out": { + return "timed-out"; + } + case "cancelled": { + return "cancelled"; + } + case "failed": { + return "failed"; + } + } +} + +function effectiveSettlementOutcome( + run: JobRunRecord, + requested: JobClaimOutcome +): JobClaimOutcome { + if ( + run.cancelRequestedAt === null || + requested.kind === "cancelled" || + requested.kind === "succeeded" + ) { + return requested; + } + return { + kind: "cancelled", + terminalCode: "cancel-requested", + terminalMessage: "The job action was cancelled.", + }; +} + +class DrizzleJobReader implements JobRepositoryReader { + protected readonly database: JobPersistenceDatabase; + + public constructor(database: JobPersistenceDatabase) { + this.database = database; + } + + public findActiveDisableIntent( + scheduleId: string + ): JobDisableIntentRecord | undefined { + const row = this.database + .select() + .from(jobDisableIntents) + .where( + and( + eq(jobDisableIntents.scheduledJobId, scheduleId), + isNull(jobDisableIntents.endedAt) + ) + ) + .get(); + return row === undefined ? undefined : parseDisableIntent(row); + } + + public findActiveRunForSchedule(scheduleId: string): JobRunRecord | undefined { + const row = this.database + .select() + .from(jobRuns) + .where( + and( + eq(jobRuns.scheduledJobId, scheduleId), + inArray(jobRuns.state, ["queued", "running"]) + ) + ) + .get(); + return row === undefined ? undefined : parseRun(row); + } + + public findLatestRunForSchedule(scheduleId: string): JobRunRecord | undefined { + const row = this.database + .select() + .from(jobRuns) + .where(eq(jobRuns.scheduledJobId, scheduleId)) + .orderBy(desc(jobRuns.queuedAt), desc(jobRuns.id)) + .get(); + return row === undefined ? undefined : parseRun(row); + } + + public findRun(id: string): JobRunRecord | undefined { + const row = this.database.select().from(jobRuns).where(eq(jobRuns.id, id)).get(); + return row === undefined ? undefined : parseRun(row); + } + + public findRunDetail(input: ListJobRunEventsInput): JobRunDetailRecord | undefined { + const run = this.findRun(input.runId); + return run === undefined ? undefined : { events: this.listRunEvents(input), run }; + } + + public findSchedule(id: string): ScheduleRecordWithRelations | undefined { + const row = this.database + .select() + .from(scheduledJobs) + .where(eq(scheduledJobs.id, id)) + .get(); + if (row === undefined) return undefined; + return this.#attachScheduleRelations([parseSchedule(row)])[0]; + } + + public listDueSchedules(input: ListDueSchedulesInput): ScheduledJobRecord[] { + const limit = input.limit ?? claimCandidateMaximum; + assertLimit(limit, claimCandidateMaximum, "due schedule"); + return this.database + .select() + .from(scheduledJobs) + .where( + and( + eq(scheduledJobs.enabled, true), + isNotNull(scheduledJobs.nextRunAt), + lte(scheduledJobs.nextRunAt, input.at), + input.cursor === undefined + ? undefined + : sql`(${scheduledJobs.nextRunAt}, ${scheduledJobs.id}) > + (${getTime(input.cursor.nextRunAt)}, ${input.cursor.id})` + ) + ) + .orderBy(asc(scheduledJobs.nextRunAt), asc(scheduledJobs.id)) + .limit(limit) + .all() + .map((row) => parseSchedule(row)); + } + + public listRunEvents(input: ListJobRunEventsInput): JobRunEventRecord[] { + assertLimit(input.limit, jobRunEventPageMaximum, "run event page"); + return this.database + .select() + .from(jobRunEvents) + .where( + and( + eq(jobRunEvents.jobRunId, input.runId), + input.beforeSequence === undefined + ? undefined + : lt(jobRunEvents.sequence, input.beforeSequence) + ) + ) + .orderBy(desc(jobRunEvents.sequence)) + .limit(input.limit + 1) + .all() + .map((row) => parseEvent(row)); + } + + public listRuns(input: ListJobRunsInput): JobRunRecord[] { + assertLimit(input.limit, jobRunPageMaximum, "run page"); + return this.database + .select() + .from(jobRuns) + .where(and(runCursorBoundary(input), ...runFilterConditions(input))) + .orderBy(desc(jobRuns.queuedAt), desc(jobRuns.id)) + .limit(input.limit + 1) + .all() + .map((row) => parseRun(row)); + } + + public listRunsWithQueueState( + input: ListJobRunsWithQueueStateInput + ): JobRunPageSnapshot { + return Object.freeze({ + queue: this.readQueueState(input), + runs: this.listRuns(input), + }); + } + + public listScheduleRuns(input: ListScheduleRunsInput): JobRunRecord[] { + assertLimit(input.limit, jobRunPageMaximum, "schedule run page"); + return this.database + .select() + .from(jobRuns) + .where(and(eq(jobRuns.scheduledJobId, input.id), runCursorBoundary(input))) + .orderBy(desc(jobRuns.queuedAt), desc(jobRuns.id)) + .limit(input.limit + 1) + .all() + .map((row) => parseRun(row)); + } + + public listSchedules(input: ListSchedulesInput): ScheduleRecordWithRelations[] { + assertLimit(input.limit, schedulePageMaximum, "schedule page"); + const records = this.database + .select() + .from(scheduledJobs) + .where( + and( + input.cursor === undefined + ? undefined + : gt(scheduledJobs.id, input.cursor.id), + input.enabled === "all" + ? undefined + : eq(scheduledJobs.enabled, input.enabled === "enabled") + ) + ) + .orderBy(asc(scheduledJobs.id)) + .limit(input.limit + 1) + .all() + .map((row) => parseSchedule(row)); + return this.#attachScheduleRelations(records); + } + + public readClaimCancellation(input: ClaimFenceInput): JobClaimCancellation { + const row = this.database + .select({ cancelRequestedAt: jobRuns.cancelRequestedAt }) + .from(jobRuns) + .where( + and( + eq(jobRuns.id, input.runId), + eq(jobRuns.state, "running"), + eq(jobRuns.leaseOwnerId, input.workerId), + eq(jobRuns.leaseToken, input.leaseToken), + gt(jobRuns.leaseExpiresAt, input.at) + ) + ) + .get(); + return row === undefined + ? { cancelRequested: false, valid: false } + : { cancelRequested: row.cancelRequestedAt !== null, valid: true }; + } + + public readQueueState(input: ReadQueueStateInput): JobQueueState { + const countRows = this.database + .select({ state: jobRuns.state, value: count() }) + .from(jobRuns) + .groupBy(jobRuns.state) + .all(); + const stateCounts = Object.fromEntries( + jobRunStates.map((state) => [state, 0]) + ) as Record; + for (const row of countRows) stateCounts[row.state] = row.value; + + const oldestQueued = this.database + .select({ queuedAt: jobRuns.queuedAt }) + .from(jobRuns) + .where(eq(jobRuns.state, "queued")) + .orderBy(asc(jobRuns.queuedAt), asc(jobRuns.id)) + .get(); + const activeClassRows = this.database + .selectDistinct({ resourceClass: jobRuns.resourceClass }) + .from(jobRuns) + .where(eq(jobRuns.state, "running")) + .all(); + const activeClassSet = new Set( + activeClassRows.map(({ resourceClass }) => resourceClass) + ); + const activeResourceClasses = jobResourceClasses.filter((resourceClass) => + activeClassSet.has(resourceClass) + ); + + const workerRows = this.database + .select() + .from(workerInstances) + .where( + and( + inArray(workerInstances.state, ["draining", "online"]), + gte(workerInstances.heartbeatAt, input.minimumHeartbeatAt) + ) + ) + .orderBy(asc(workerInstances.id)) + .limit(jobWorkerSummaryMaximum) + .all() + .map((row) => parseWorker(row)); + const workerIds = workerRows.map(({ id }) => id); + const activeCounts = + workerIds.length === 0 + ? [] + : this.database + .select({ ownerId: jobRuns.leaseOwnerId, value: count() }) + .from(jobRuns) + .where( + and( + eq(jobRuns.state, "running"), + inArray(jobRuns.leaseOwnerId, workerIds) + ) + ) + .groupBy(jobRuns.leaseOwnerId) + .all(); + const activeCountByWorker = new Map( + activeCounts.flatMap((row) => + row.ownerId === null ? [] : [[row.ownerId, row.value] as const] + ) + ); + return { + activeResourceClasses, + control: this.readWorkerControl(), + ...(oldestQueued === undefined + ? {} + : { oldestQueuedAt: oldestQueued.queuedAt }), + stateCounts, + workers: workerRows.map((worker) => ({ + activeRunCount: activeCountByWorker.get(worker.id) ?? 0, + worker, + })), + }; + } + + public readWorkerControl(): JobWorkerControlRecord { + const row = this.database + .select() + .from(jobWorkerControl) + .where(eq(jobWorkerControl.id, 1)) + .get(); + return v.parse( + jobWorkerControlSelectSchema, + requiredRow(row, "worker control read") + ); + } + + #attachScheduleRelations( + records: readonly ScheduledJobRecord[] + ): ScheduleRecordWithRelations[] { + if (records.length === 0) return []; + const ids = records.map(({ id }) => id); + const activeIntents = this.database + .select() + .from(jobDisableIntents) + .where( + and( + inArray(jobDisableIntents.scheduledJobId, ids), + isNull(jobDisableIntents.endedAt) + ) + ) + .all() + .map((row) => parseDisableIntent(row)); + const activeRuns = this.database + .select() + .from(jobRuns) + .where( + and( + inArray(jobRuns.scheduledJobId, ids), + inArray(jobRuns.state, ["queued", "running"]) + ) + ) + .all() + .map((row) => parseRun(row)); + const latestIds = this.database.all<{ id: string }>(sql` + SELECT ranked.id + FROM ( + SELECT id, + row_number() OVER ( + PARTITION BY scheduled_job_id + ORDER BY queued_at DESC, id DESC + ) AS position + FROM job_runs + WHERE scheduled_job_id IN (${sql.join( + ids.map((id) => sql`${id}`), + sql`, ` + )}) + ) AS ranked + WHERE ranked.position = 1 + `); + const latestRuns = + latestIds.length === 0 + ? [] + : this.database + .select() + .from(jobRuns) + .where( + inArray( + jobRuns.id, + latestIds.map(({ id }) => id) + ) + ) + .all() + .map((row) => parseRun(row)); + const intentBySchedule = new Map( + activeIntents.flatMap((intent) => + intent.scheduledJobId === null + ? [] + : [[intent.scheduledJobId, intent] as const] + ) + ); + const activeRunBySchedule = new Map( + activeRuns.flatMap((run) => + run.scheduledJobId === null ? [] : [[run.scheduledJobId, run] as const] + ) + ); + const latestRunBySchedule = new Map( + latestRuns.flatMap((run) => + run.scheduledJobId === null ? [] : [[run.scheduledJobId, run] as const] + ) + ); + return records.map((schedule) => ({ + ...(intentBySchedule.has(schedule.id) + ? { activeDisableIntent: intentBySchedule.get(schedule.id) } + : {}), + ...(activeRunBySchedule.has(schedule.id) + ? { activeRun: activeRunBySchedule.get(schedule.id) } + : {}), + ...(latestRunBySchedule.has(schedule.id) + ? { latestRun: latestRunBySchedule.get(schedule.id) } + : {}), + schedule, + })); + } +} + +class DrizzleJobWriter extends DrizzleJobReader { + readonly #transaction: JobTransaction; + + public constructor(transaction: JobTransaction) { + super(transaction); + this.#transaction = transaction; + } + + public reconcileSchedules(input: ReconcileSchedulesInput): ScheduledJobRecord[] { + const records: ScheduledJobRecord[] = []; + const registeredScheduleIds = new Set( + input.schedules.map((schedule) => schedule.id) + ); + for (const candidate of input.schedules) { + const validated = v.parse(scheduledJobInsertSchema, candidate); + const existing = this.#findScheduleRecord(validated.id); + if (existing === undefined) { + const inserted = this.#transaction + .insert(scheduledJobs) + .values(validated) + .returning() + .get(); + const registered = parseSchedule( + requiredRow(inserted, "schedule insert") + ); + records.push(registered); + this.#insertSideEffects(input.sideEffectsForSchedule(registered)); + continue; + } + const metadataChanged = + existing.actionKey !== validated.actionKey || + existing.actionPayloadJson !== validated.actionPayloadJson || + existing.attemptLimit !== validated.attemptLimit || + existing.cancellationPolicy !== validated.cancellationPolicy || + existing.description !== validated.description || + existing.name !== validated.name || + existing.priority !== validated.priority || + existing.resourceClass !== validated.resourceClass || + existing.resourceKeysJson !== validated.resourceKeysJson || + existing.retrySafe !== validated.retrySafe || + existing.timeoutMs !== validated.timeoutMs; + if (!metadataChanged) { + records.push(existing); + continue; + } + const row = this.#transaction + .update(scheduledJobs) + .set({ + actionKey: validated.actionKey, + actionPayloadJson: validated.actionPayloadJson, + attemptLimit: validated.attemptLimit, + cancellationPolicy: validated.cancellationPolicy, + description: validated.description, + name: validated.name, + priority: validated.priority, + resourceClass: validated.resourceClass, + resourceKeysJson: validated.resourceKeysJson, + retrySafe: validated.retrySafe, + timeoutMs: validated.timeoutMs, + updatedAt: maximumDate(existing.updatedAt, validated.updatedAt), + version: existing.version + 1, + }) + .where( + and( + eq(scheduledJobs.id, existing.id), + eq(scheduledJobs.version, existing.version) + ) + ) + .returning() + .get(); + const reconciled = parseSchedule(requiredRow(row, "schedule reconciliation")); + records.push(reconciled); + this.#insertSideEffects(input.sideEffectsForSchedule(reconciled)); + } + const retiredSchedules = this.#transaction + .select() + .from(scheduledJobs) + .where(eq(scheduledJobs.enabled, true)) + .all() + .map((row) => parseSchedule(row)) + .filter((schedule) => !registeredScheduleIds.has(schedule.id)); + for (const schedule of retiredSchedules) { + const queuedScheduleRun = this.#findQueuedScheduleRun(schedule.id); + // Registry retirement disables future scheduling. A queued `never` run keeps + // its immutable execution snapshot and completes through the normal worker + // action-availability path; retirement must not reinterpret it as cancellable. + const cancellableQueuedScheduleRun = + queuedScheduleRun?.cancellationPolicy === "never" + ? undefined + : queuedScheduleRun; + const retiredRunCancellation = input.retiredRunCancellation; + if ( + cancellableQueuedScheduleRun !== undefined && + retiredRunCancellation === undefined + ) { + throw new Error( + "Removed schedule retirement requires queued-run cancellation metadata" + ); + } + const retired = this.#transaction + .update(scheduledJobs) + .set({ + enabled: false, + updatedAt: maximumDate(schedule.updatedAt, input.at), + version: schedule.version + 1, + }) + .where( + and( + eq(scheduledJobs.id, schedule.id), + eq(scheduledJobs.enabled, true), + eq(scheduledJobs.version, schedule.version) + ) + ) + .returning() + .get(); + const retiredSchedule = parseSchedule( + requiredRow(retired, "removed schedule retirement") + ); + if ( + cancellableQueuedScheduleRun !== undefined && + retiredRunCancellation !== undefined + ) { + const cancelled = this.#cancelQueuedRun( + cancellableQueuedScheduleRun, + retiredRunCancellation.actor, + { + at: retiredSchedule.updatedAt, + terminalCode: retiredRunCancellation.terminalCode, + terminalMessage: retiredRunCancellation.terminalMessage, + } + ); + this.#insertSideEffects( + retiredRunCancellation.sideEffectsForRun(cancelled) + ); + } + this.#insertSideEffects(input.sideEffectsForSchedule(retiredSchedule)); + } + return records; + } + + public enqueueManualRun(input: EnqueueManualRunInput): EnqueueManualRunResult { + const run = v.parse(jobRunInsertSchema, input.run); + if (input.queuedEvent.jobRunId !== run.id) { + throw new Error("Queued event does not belong to the inserted manual run"); + } + const existing = this.#findRunByIdempotency( + run.requestedByKind, + run.requestedById, + run.idempotencyKey + ); + if (existing !== undefined) { + return existing.enqueueSha256 === run.enqueueSha256 + ? { kind: "replayed", run: existing } + : { kind: "idempotency-mismatch", run: existing }; + } + if (run.scheduledJobId !== null) { + const schedule = this.#findScheduleRecord(run.scheduledJobId); + if ( + schedule === undefined || + !runMatchesScheduleExecutionSnapshot(run, schedule) + ) { + return { kind: "action-unavailable" }; + } + const active = this.findActiveRunForSchedule(run.scheduledJobId); + if (active !== undefined) return { kind: "active", run: active }; + } + const inserted = this.#transaction.insert(jobRuns).values(run).returning().get(); + const record = parseRun(requiredRow(inserted, "manual run insert")); + this.#insertSuppliedEvent(input.queuedEvent); + this.#insertSideEffects(input); + return { + kind: "inserted", + run: requiredRow(this.findRun(record.id), "manual run refresh"), + }; + } + + public updateSchedule( + input: UpdateScheduleRepositoryInput + ): UpdateScheduleRepositoryResult { + const current = this.#findScheduleRecord(input.id); + if (current === undefined) return { kind: "not-found" }; + if (current.version !== input.expectedVersion) { + return { kind: "version-changed", schedule: current }; + } + const activeIntent = this.findActiveDisableIntent(input.id); + if ((activeIntent?.id ?? null) !== input.expectedActiveDisableIntentId) { + return { kind: "version-changed", schedule: current }; + } + const queuedScheduleRunRecord = + input.patch.enabled === false + ? this.#findQueuedScheduleRun(input.id) + : undefined; + if (queuedScheduleRunRecord?.cancellationPolicy === "never") { + return { + kind: "cancellation-not-supported", + run: queuedScheduleRunRecord, + }; + } + const transitionAt = maximumDate(current.updatedAt, input.at); + const scheduleChanges = + input.patch.schedule === undefined + ? {} + : this.#scheduleColumns(input.patch.schedule); + const enabled = input.patch.enabled ?? current.enabled; + let nextRunAt = current.nextRunAt; + // A pure disable keeps the existing cursor dormant. When the cadence also + // changes, retain the service's recalculated cursor for the new phase. + if ( + input.patch.nextRunAt !== undefined && + (input.patch.enabled !== false || input.patch.schedule !== undefined) + ) { + nextRunAt = input.patch.nextRunAt; + } + if (enabled && nextRunAt === null) { + throw new TypeError("Enabled schedule requires one next occurrence"); + } + + if (input.closeActiveIntent !== undefined) { + if (activeIntent === undefined) { + throw new Error("Schedule disable-intent closure has no active intent"); + } + const closed = this.#transaction + .update(jobDisableIntents) + .set(v.parse(jobDisableIntentCloseSchema, input.closeActiveIntent)) + .where( + and( + eq(jobDisableIntents.id, activeIntent.id), + isNull(jobDisableIntents.endedAt) + ) + ) + .returning() + .get(); + requiredRow(closed, "disable intent closure"); + } + if (input.insertDisableIntent !== undefined) { + this.#transaction + .insert(jobDisableIntents) + .values(v.parse(jobDisableIntentInsertSchema, input.insertDisableIntent)) + .run(); + } + + const row = this.#transaction + .update(scheduledJobs) + .set({ + ...scheduleChanges, + enabled, + nextRunAt, + updatedAt: transitionAt, + version: current.version + 1, + }) + .where( + and( + eq(scheduledJobs.id, input.id), + eq(scheduledJobs.version, input.expectedVersion) + ) + ) + .returning() + .get(); + if (row === undefined) { + throw new Error( + "Schedule update lost its guarded write after intent changes" + ); + } + + if (input.patch.enabled === false && queuedScheduleRunRecord !== undefined) { + if ( + input.queuedCancellation === undefined || + input.queuedCancellationSideEffects === undefined || + input.insertDisableIntent === undefined + ) { + throw new Error( + "Schedule disable requires queued-run cancellation metadata" + ); + } + const actor: JobActor = { + id: input.insertDisableIntent.createdById, + kind: input.insertDisableIntent.createdByKind, + }; + const cancelled = this.#cancelQueuedRun( + queuedScheduleRunRecord, + actor, + input.queuedCancellation + ); + this.#insertSideEffects(input.queuedCancellationSideEffects(cancelled)); + } + this.#insertSideEffects(input); + return { kind: "updated", schedule: parseSchedule(row) }; + } + + public enqueueNextDueSchedule( + input: DueScheduleEnqueueInput + ): DueScheduleEnqueueResult { + const validatedRun = v.parse(jobRunInsertSchema, input.run); + const replay = this.#findRunByIdempotency( + validatedRun.requestedByKind, + validatedRun.requestedById, + validatedRun.idempotencyKey + ); + if (replay !== undefined && replay.enqueueSha256 === validatedRun.enqueueSha256) { + return { kind: "inserted", run: replay }; + } + const schedule = this.#findScheduleRecord(input.scheduleId); + if (schedule === undefined) return { kind: "not-found" }; + if (!runMatchesScheduleExecutionSnapshot(validatedRun, schedule)) { + return { kind: "state-changed", schedule }; + } + if ( + !schedule.enabled || + schedule.nextRunAt === null || + getTime(schedule.nextRunAt) !== getTime(input.observedNextRunAt) + ) { + return { kind: "state-changed", schedule }; + } + if (getTime(schedule.nextRunAt) > getTime(input.at)) { + return { kind: "not-due" }; + } + const active = this.findActiveRunForSchedule(schedule.id); + if (active !== undefined) return { kind: "active", run: active }; + if ( + validatedRun.triggerType !== "schedule" || + validatedRun.scheduledJobId !== schedule.id || + validatedRun.scheduledForAt === null || + getTime(validatedRun.scheduledForAt) !== getTime(schedule.nextRunAt) || + getTime(input.nextRunAt) <= getTime(input.at) + ) { + throw new TypeError("Due schedule enqueue snapshot is inconsistent"); + } + const inserted = this.#transaction + .insert(jobRuns) + .values(validatedRun) + .returning() + .get(); + const run = parseRun(requiredRow(inserted, "due run insert")); + this.#appendEvent(run.id, { + attempt: 0, + kind: "queued", + occurredAt: run.queuedAt, + workerInstanceId: null, + }); + const advanced = this.#transaction + .update(scheduledJobs) + .set({ nextRunAt: input.nextRunAt }) + .where( + and( + eq(scheduledJobs.id, schedule.id), + eq(scheduledJobs.enabled, true), + eq(scheduledJobs.version, schedule.version), + eq(scheduledJobs.nextRunAt, input.observedNextRunAt) + ) + ) + .returning() + .get(); + requiredRow(advanced, "due schedule cursor advance"); + this.#insertSideEffects(input); + return { + kind: "inserted", + run: requiredRow(this.findRun(run.id), "due run refresh"), + }; + } + + public cancelRun(input: CancelRunRepositoryInput): CancelRunRepositoryResult { + const run = this.findRun(input.id); + if (run === undefined) return { kind: "not-found" }; + if (terminalRunStates.has(run.state)) return { kind: "terminal", run }; + const supportsCancellation = + run.cancellationPolicy === "cooperative" || + (run.cancellationPolicy === "queued-only" && run.state === "queued"); + if (!supportsCancellation) return { kind: "unsupported", run }; + if (run.cancelRequestedAt !== null) { + return run.state === "queued" + ? { kind: "cancelled", run } + : { kind: "requested", run }; + } + if (run.state === "queued") { + const cancelled = this.#cancelQueuedRun(run, input.actor, input); + this.#insertSideEffects(input.sideEffectsForRun(cancelled)); + return { kind: "cancelled", run: cancelled }; + } + const at = maximumDate(run.updatedAt, input.at); + const row = this.#transaction + .update(jobRuns) + .set({ + cancelRequestedAt: at, + cancelRequestedById: input.actor.id, + cancelRequestedByKind: input.actor.kind, + stateVersion: run.stateVersion + 1, + updatedAt: at, + }) + .where( + and( + eq(jobRuns.id, run.id), + eq(jobRuns.state, "running"), + isNull(jobRuns.cancelRequestedAt), + eq(jobRuns.stateVersion, run.stateVersion) + ) + ) + .returning() + .get(); + if (row === undefined) { + const observed = requiredRow(this.findRun(run.id), "cancel conflict read"); + return terminalRunStates.has(observed.state) + ? { kind: "terminal", run: observed } + : { kind: "requested", run: observed }; + } + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "cancel-requested", + occurredAt: at, + workerInstanceId: run.leaseOwnerId, + }); + const requested = requiredRow(this.findRun(run.id), "cancel request refresh"); + this.#insertSideEffects(input.sideEffectsForRun(requested)); + return { + kind: "requested", + run: requested, + }; + } + + public setClaimingPaused( + input: SetClaimingPausedRepositoryInput + ): SetClaimingPausedRepositoryResult { + const current = v.parse( + jobWorkerControlSelectSchema, + requiredRow( + this.#transaction + .select() + .from(jobWorkerControl) + .where(eq(jobWorkerControl.id, 1)) + .get(), + "worker control read" + ) + ); + if (current.version !== input.expectedVersion) { + return { control: current, kind: "version-changed" }; + } + const update = v.parse(jobWorkerControlUpdateSchema, { + claimingPaused: input.paused, + updatedAt: maximumDate(current.updatedAt, input.at), + updatedById: input.actor.id, + updatedByKind: input.actor.kind, + version: current.version + 1, + }); + const row = this.#transaction + .update(jobWorkerControl) + .set(update) + .where( + and( + eq(jobWorkerControl.id, 1), + eq(jobWorkerControl.version, input.expectedVersion) + ) + ) + .returning() + .get(); + if (row === undefined) { + const observed = v.parse( + jobWorkerControlSelectSchema, + requiredRow( + this.#transaction + .select() + .from(jobWorkerControl) + .where(eq(jobWorkerControl.id, 1)) + .get(), + "worker control conflict read" + ) + ); + return { control: observed, kind: "version-changed" }; + } + const control = v.parse(jobWorkerControlSelectSchema, row); + this.#insertSideEffects(input); + return { control, kind: "updated" }; + } + + public registerWorker(input: RegisterWorkerInput): WorkerInstanceRecord { + const row = this.#transaction + .insert(workerInstances) + .values(v.parse(workerInstanceInsertSchema, input.worker)) + .returning() + .get(); + const worker = parseWorker(requiredRow(row, "worker registration")); + this.#insertSideEffects(input); + return worker; + } + + public heartbeatWorker( + input: WorkerLifecycleInput + ): WorkerInstanceRecord | undefined { + const worker = this.#findWorker(input.workerId); + if (worker === undefined || worker.state === "stopped") return worker; + const at = maximumDate(worker.heartbeatAt, input.at); + if (getTime(at) === getTime(worker.heartbeatAt)) return worker; + const row = this.#transaction + .update(workerInstances) + .set({ heartbeatAt: at }) + .where( + and( + eq(workerInstances.id, worker.id), + eq(workerInstances.state, worker.state), + lte(workerInstances.heartbeatAt, at) + ) + ) + .returning() + .get(); + return row === undefined ? this.#findWorker(worker.id) : parseWorker(row); + } + + public beginWorkerDrain(input: WorkerLifecycleMutationInput): WorkerLifecycleResult { + const worker = this.#findWorker(input.workerId); + if (worker === undefined) return { kind: "not-found" }; + if (worker.state !== "online") { + return { kind: "state-changed", worker }; + } + const at = maximumDate(worker.heartbeatAt, worker.startedAt, input.at); + const row = this.#transaction + .update(workerInstances) + .set({ drainingAt: at, heartbeatAt: at, state: "draining" }) + .where( + and( + eq(workerInstances.id, worker.id), + eq(workerInstances.state, "online") + ) + ) + .returning() + .get(); + if (row === undefined) { + return { + kind: "state-changed", + worker: requiredRow(this.#findWorker(worker.id), "worker drain read"), + }; + } + const updated = parseWorker(row); + this.#insertSideEffects(input.sideEffectsForWorker(updated)); + return { kind: "updated", worker: updated }; + } + + public stopWorker(input: WorkerLifecycleMutationInput): WorkerLifecycleResult { + const worker = this.#findWorker(input.workerId); + if (worker === undefined) return { kind: "not-found" }; + if (worker.state !== "draining") { + return { kind: "state-changed", worker }; + } + const activeRun = this.#transaction + .select({ id: jobRuns.id }) + .from(jobRuns) + .where(and(eq(jobRuns.state, "running"), eq(jobRuns.leaseOwnerId, worker.id))) + .limit(1) + .get(); + if (activeRun !== undefined) return { kind: "active-runs", worker }; + const at = maximumDate( + worker.heartbeatAt, + requiredValue(worker.drainingAt, "worker draining timestamp"), + input.at + ); + const row = this.#transaction + .update(workerInstances) + .set({ heartbeatAt: at, state: "stopped", stoppedAt: at }) + .where( + and( + eq(workerInstances.id, worker.id), + eq(workerInstances.state, "draining") + ) + ) + .returning() + .get(); + if (row === undefined) { + return { + kind: "state-changed", + worker: requiredRow(this.#findWorker(worker.id), "worker stop read"), + }; + } + const updated = parseWorker(row); + this.#insertSideEffects(input.sideEffectsForWorker(updated)); + return { kind: "updated", worker: updated }; + } + + public expireDisableIntents( + input: ExpireDisableIntentsInput + ): readonly ExpireDisableIntentResult[] { + const limit = input.limit ?? recoveryBatchMaximum; + assertLimit(limit, recoveryBatchMaximum, "disable-intent expiry"); + const intents = this.#transaction + .select() + .from(jobDisableIntents) + .where( + and( + eq(jobDisableIntents.targetKind, "dashboard-schedule"), + isNull(jobDisableIntents.endedAt), + isNotNull(jobDisableIntents.expiresAt), + lte(jobDisableIntents.expiresAt, input.at) + ) + ) + .orderBy(asc(jobDisableIntents.expiresAt), asc(jobDisableIntents.id)) + .limit(limit) + .all() + .map((row) => parseDisableIntent(row)); + + return intents.map((intent) => { + const scheduleId = requiredValue( + intent.scheduledJobId, + "expired intent schedule id" + ); + const schedule = requiredRow( + this.#findScheduleRecord(scheduleId), + "expired intent schedule" + ); + if (schedule.enabled) { + throw new Error("Enabled schedule retains an active disable intent"); + } + if (!input.canReenableSchedule(schedule)) { + const transitionAt = maximumDate( + schedule.updatedAt, + requiredValue(intent.expiresAt, "disable intent expiry"), + input.at + ); + const closedIntent = this.#closeExpiredIntent({ + at: input.at, + context: "retired schedule intent closure", + intent, + systemActorId: input.systemActorId, + transitionAt, + }); + this.#insertSideEffects( + input.sideEffectsForSchedule(schedule, closedIntent) + ); + return { + intent: closedIntent, + kind: "left-disabled" as const, + schedule, + }; + } + const nextRunAt = input.nextRunAt(schedule, input.at); + if (nextRunAt === undefined) { + return { + intent, + kind: "next-occurrence-unavailable" as const, + schedule, + }; + } + if (getTime(nextRunAt) <= getTime(input.at)) { + throw new RangeError( + "Expired disable intent must resume strictly after transaction time" + ); + } + const transitionAt = maximumDate( + schedule.updatedAt, + requiredValue(intent.expiresAt, "disable intent expiry"), + input.at + ); + const closedIntent = this.#closeExpiredIntent({ + at: input.at, + context: "expired intent closure", + intent, + systemActorId: input.systemActorId, + transitionAt, + }); + const scheduleRow = this.#transaction + .update(scheduledJobs) + .set({ + enabled: true, + nextRunAt, + updatedAt: transitionAt, + version: schedule.version + 1, + }) + .where( + and( + eq(scheduledJobs.id, schedule.id), + eq(scheduledJobs.enabled, false), + eq(scheduledJobs.version, schedule.version) + ) + ) + .returning() + .get(); + const resumed = parseSchedule( + requiredRow(scheduleRow, "expired schedule resume") + ); + this.#insertSideEffects(input.sideEffectsForSchedule(resumed, closedIntent)); + return { + intent: closedIntent, + kind: "re-enabled" as const, + schedule: resumed, + }; + }); + } + + public recoverExpiredClaims( + input: RecoverExpiredClaimsInput + ): readonly JobRunRecord[] { + const limit = input.limit ?? recoveryBatchMaximum; + assertLimit(limit, recoveryBatchMaximum, "expired claim recovery"); + const expired = this.#transaction + .select() + .from(jobRuns) + .where( + and(eq(jobRuns.state, "running"), lte(jobRuns.leaseExpiresAt, input.at)) + ) + .orderBy(asc(jobRuns.leaseExpiresAt), asc(jobRuns.id)) + .limit(limit) + .all() + .map((row) => parseRun(row)); + return expired.map((run) => { + const at = maximumDate(run.updatedAt, input.at); + const shouldCancel = run.cancelRequestedAt !== null; + const shouldRetry = + !shouldCancel && run.retrySafe && run.attemptCount < run.attemptLimit; + this.#releaseResources( + run, + requiredValue(run.leaseOwnerId, "expired claim owner"), + requiredValue(run.leaseToken, "expired claim token") + ); + if (shouldRetry) { + const retryAt = maximumDate(at, input.retryAt(run)); + this.#transitionClaim(run, { + availableAt: retryAt, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + state: "queued", + stateVersion: run.stateVersion + 1, + updatedAt: at, + }); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "lease-expired", + occurredAt: at, + workerInstanceId: run.leaseOwnerId, + }); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "retry-scheduled", + occurredAt: at, + workerInstanceId: run.leaseOwnerId, + }); + } else { + const state = shouldCancel ? "cancelled" : "failed"; + this.#transitionClaim(run, { + finishedAt: at, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + state, + stateVersion: run.stateVersion + 1, + terminalCode: shouldCancel + ? "job/cancel-requested" + : "worker/lease-expired", + terminalMessage: shouldCancel + ? "The run was cancelled after its worker lease expired." + : "The worker lease expired and this action is not retryable.", + updatedAt: at, + }); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "lease-expired", + occurredAt: at, + workerInstanceId: run.leaseOwnerId, + }); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: state, + occurredAt: at, + workerInstanceId: run.leaseOwnerId, + }); + } + const refreshed = requiredRow(this.findRun(run.id), "expired claim refresh"); + this.#insertSideEffects(input.sideEffectsForRun(refreshed)); + return refreshed; + }); + } + + public claimNextRun(input: ClaimNextRunInput): JobClaimResult { + if (getTime(input.leaseExpiresAt) <= getTime(input.at)) { + throw new RangeError("Claim lease expiry must be after claim time"); + } + const control = v.parse( + jobWorkerControlSelectSchema, + requiredRow( + this.#transaction + .select() + .from(jobWorkerControl) + .where(eq(jobWorkerControl.id, 1)) + .get(), + "worker control claim read" + ) + ); + if (control.claimingPaused) return { kind: "paused" }; + const worker = this.#findWorker(input.workerId); + if ( + worker === undefined || + worker.state !== "online" || + getTime(worker.heartbeatAt) < getTime(input.minimumHeartbeatAt) + ) { + return { kind: "worker-unavailable" }; + } + const activeCount = requiredRow( + this.#transaction + .select({ value: count() }) + .from(jobRuns) + .where( + and(eq(jobRuns.state, "running"), eq(jobRuns.leaseOwnerId, worker.id)) + ) + .get(), + "worker active count" + ).value; + if (activeCount >= worker.capacity) return { kind: "worker-unavailable" }; + + const availableThrough = input.cursor?.availableThrough ?? input.at; + const candidates: JobRunRecord[] = []; + const appendCandidateRange = (range?: SQL): void => { + const remaining = claimCandidateMaximum - candidates.length; + if (remaining === 0) return; + candidates.push( + ...this.#transaction + .select() + .from(jobRuns) + .where( + and( + eq(jobRuns.state, "queued"), + lte(jobRuns.availableAt, availableThrough), + range + ) + ) + .orderBy( + asc(jobRuns.availableAt), + desc(jobRuns.priority), + asc(jobRuns.queuedAt), + asc(jobRuns.id) + ) + .limit(remaining) + .all() + .map((row) => parseRun(row)) + ); + }; + if (input.cursor === undefined) { + appendCandidateRange(); + } else { + const cursor = input.cursor; + appendCandidateRange( + and( + eq(jobRuns.availableAt, cursor.availableAt), + eq(jobRuns.priority, cursor.priority), + eq(jobRuns.queuedAt, cursor.queuedAt), + gt(jobRuns.id, cursor.id) + ) + ); + appendCandidateRange( + and( + eq(jobRuns.availableAt, cursor.availableAt), + eq(jobRuns.priority, cursor.priority), + gt(jobRuns.queuedAt, cursor.queuedAt) + ) + ); + appendCandidateRange( + and( + eq(jobRuns.availableAt, cursor.availableAt), + lt(jobRuns.priority, cursor.priority) + ) + ); + appendCandidateRange(gt(jobRuns.availableAt, cursor.availableAt)); + } + for (const candidate of candidates) { + const keys = resourceKeys(candidate); + const resourceConflict = + keys.length > 0 && + this.#transaction + .select({ key: resourceLeases.resourceKey }) + .from(resourceLeases) + .where(inArray(resourceLeases.resourceKey, keys)) + .limit(1) + .get() !== undefined; + if (resourceConflict) continue; + + const at = maximumDate(candidate.updatedAt, availableThrough, input.at); + const leaseExpiresAt = shiftDurationToStart( + input.at, + input.leaseExpiresAt, + at + ); + const row = this.#transaction + .update(jobRuns) + .set({ + attemptCount: candidate.attemptCount + 1, + firstStartedAt: candidate.firstStartedAt ?? at, + heartbeatAt: at, + lastAttemptStartedAt: at, + leaseExpiresAt, + leaseOwnerId: worker.id, + leaseToken: input.leaseToken, + state: "running", + stateVersion: candidate.stateVersion + 1, + updatedAt: at, + }) + .where( + and( + eq(jobRuns.id, candidate.id), + eq(jobRuns.state, "queued"), + eq(jobRuns.stateVersion, candidate.stateVersion), + lte(jobRuns.availableAt, availableThrough) + ) + ) + .returning() + .get(); + if (row === undefined) continue; + const claimed = parseRun(row); + for (const resourceKey of keys) { + this.#transaction + .insert(resourceLeases) + .values( + v.parse(resourceLeaseInsertSchema, { + acquiredAt: at, + expiresAt: leaseExpiresAt, + jobRunId: claimed.id, + leaseToken: input.leaseToken, + renewedAt: at, + resourceKey, + workerInstanceId: worker.id, + }) + ) + .run(); + } + this.#appendEvent(claimed.id, { + attempt: claimed.attemptCount, + kind: "claimed", + occurredAt: at, + workerInstanceId: worker.id, + }); + const refreshed = requiredRow( + this.findRun(claimed.id), + "claimed run refresh" + ); + this.#insertSideEffects(input.sideEffectsForClaim(refreshed)); + return { + kind: "claimed", + run: refreshed, + }; + } + const lastCandidate = candidates.at(-1); + if (candidates.length === claimCandidateMaximum && lastCandidate !== undefined) { + return { + cursor: { + availableAt: lastCandidate.availableAt, + availableThrough, + id: lastCandidate.id, + priority: lastCandidate.priority, + queuedAt: lastCandidate.queuedAt, + }, + kind: "page-exhausted", + }; + } + return { kind: "empty" }; + } + + public renewClaim(input: RenewClaimInput): JobClaimMutationResult { + const run = this.#findFencedRun(input); + if (run === undefined) return { kind: "lost-claim" }; + if (run.leaseExpiresAt === null) throw new Error("Active claim has no lease"); + const at = maximumDate( + run.updatedAt, + requiredValue(run.heartbeatAt, "claim heartbeat"), + input.at + ); + const leaseExpiresAt = shiftDurationToStart(input.at, input.leaseExpiresAt, at); + if (getTime(leaseExpiresAt) <= getTime(run.leaseExpiresAt)) { + throw new RangeError("Renewed lease expiry must advance the active lease"); + } + const row = this.#transaction + .update(jobRuns) + .set({ + heartbeatAt: at, + leaseExpiresAt, + updatedAt: at, + }) + .where(this.#claimFence(input)) + .returning() + .get(); + if (row === undefined) return { kind: "lost-claim" }; + const expectedKeys = resourceKeys(parseRun(row)); + const renewed = + expectedKeys.length === 0 + ? [] + : this.#transaction + .update(resourceLeases) + .set({ expiresAt: leaseExpiresAt, renewedAt: at }) + .where( + and( + eq(resourceLeases.jobRunId, input.runId), + eq(resourceLeases.workerInstanceId, input.workerId), + eq(resourceLeases.leaseToken, input.leaseToken), + inArray(resourceLeases.resourceKey, expectedKeys) + ) + ) + .returning() + .all() + .map((lease) => v.parse(resourceLeaseSelectSchema, lease)); + if (renewed.length !== expectedKeys.length) { + throw new Error("Claim resource lease set is incomplete during renewal"); + } + return { + kind: "renewed", + run: requiredRow(this.findRun(input.runId), "renewed claim refresh"), + }; + } + + public appendClaimEvent(input: AppendClaimEventInput): JobAppendEventResult { + const run = this.#findFencedRun(input); + if (run === undefined) return { kind: "lost-claim" }; + const eventBytes = + utf8ByteLength(input.message ?? "") + + utf8ByteLength(input.progressJson ?? ""); + const exhausted = + run.payloadEventCount >= jobRunPayloadEventMaximum || + run.eventCount >= jobRunEventMaximum - 1 || + run.eventBytes + eventBytes > jobRunPayloadEventMaximumBytes; + if (exhausted) { + const alreadyTruncated = + this.#transaction + .select({ sequence: jobRunEvents.sequence }) + .from(jobRunEvents) + .where( + and( + eq(jobRunEvents.jobRunId, run.id), + eq(jobRunEvents.kind, "output-truncated") + ) + ) + .limit(1) + .get() !== undefined; + if (alreadyTruncated || run.eventCount >= jobRunEventMaximum - 1) { + return { kind: "dropped" }; + } + const at = this.#touchRunForEvent(run, input.at); + const event = this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "output-truncated", + occurredAt: at, + workerInstanceId: input.workerId, + }); + const refreshed = requiredRow( + this.findRun(run.id), + "truncated event run refresh" + ); + this.#insertSideEffects(input.sideEffectsForRun(refreshed)); + return { event, kind: "truncated" }; + } + const at = this.#touchRunForEvent(run, input.at); + const event = this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: input.kind, + message: input.message ?? null, + occurredAt: at, + progressJson: input.progressJson ?? null, + workerInstanceId: input.workerId, + }); + const refreshed = requiredRow(this.findRun(run.id), "appended event run refresh"); + this.#insertSideEffects(input.sideEffectsForRun(refreshed)); + return { event, kind: "appended" }; + } + + public settleClaim(input: SettleClaimInput): JobSettlementResult { + const run = this.#findFencedRun(input); + if (run === undefined) return { kind: "lost-claim" }; + const at = maximumDate(run.updatedAt, input.at); + const outcome = effectiveSettlementOutcome(run, input.outcome); + const canRetry = + outcome.kind === "failed" && + outcome.retryAt !== undefined && + run.retrySafe && + run.attemptCount < run.attemptLimit; + if (canRetry) { + const retryAt = maximumDate( + at, + requiredRow(outcome.retryAt, "retry timestamp") + ); + this.#touchRunForEvent(run, at); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "failed", + message: boundedStructuralMessage(outcome.terminalMessage), + occurredAt: at, + workerInstanceId: input.workerId, + }); + this.#releaseResources(run, input.workerId, input.leaseToken); + const row = this.#transaction + .update(jobRuns) + .set({ + availableAt: retryAt, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + state: "queued", + stateVersion: run.stateVersion + 1, + updatedAt: at, + }) + .where(this.#claimFence(input)) + .returning() + .get(); + requiredRow(row, "retry transition"); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "retry-scheduled", + occurredAt: at, + workerInstanceId: input.workerId, + }); + const refreshed = requiredRow(this.findRun(run.id), "retry refresh"); + this.#insertSideEffects(input.sideEffectsForRun(refreshed)); + return { + kind: "retry-scheduled", + run: refreshed, + }; + } + + this.#releaseResources(run, input.workerId, input.leaseToken); + const state = terminalStateForOutcome(outcome); + const row = this.#transaction + .update(jobRuns) + .set({ + finishedAt: at, + heartbeatAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + resultJson: outcome.kind === "succeeded" ? outcome.resultJson : null, + state, + stateVersion: run.stateVersion + 1, + terminalCode: outcome.kind === "succeeded" ? null : outcome.terminalCode, + terminalMessage: + outcome.kind === "succeeded" ? null : outcome.terminalMessage, + updatedAt: at, + }) + .where(this.#claimFence(input)) + .returning() + .get(); + if (row === undefined) return { kind: "lost-claim" }; + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: state, + ...(outcome.kind === "succeeded" + ? {} + : { + message: boundedStructuralMessage(outcome.terminalMessage), + }), + occurredAt: at, + workerInstanceId: input.workerId, + }); + const refreshed = requiredRow(this.findRun(run.id), "settled claim refresh"); + this.#insertSideEffects(input.sideEffectsForRun(refreshed)); + return { + kind: "settled", + run: refreshed, + }; + } + + #appendEvent(runId: string, input: InternalRunEventInput): JobRunEventRecord { + const run = requiredRow(this.findRun(runId), "event parent read"); + const row = this.#transaction + .insert(jobRunEvents) + .values( + v.parse(jobRunEventInsertSchema, { + ...input, + jobRunId: runId, + message: input.message ?? null, + progressJson: input.progressJson ?? null, + sequence: run.eventCount + 1, + }) + ) + .returning() + .get(); + return parseEvent(requiredRow(row, "run event insert")); + } + + #cancelQueuedRun( + run: JobRunRecord, + actor: JobActor, + input: ScheduleQueuedCancellation + ): JobRunRecord { + const at = maximumDate(run.updatedAt, input.at); + const row = this.#transaction + .update(jobRuns) + .set({ + cancelRequestedAt: at, + cancelRequestedById: actor.id, + cancelRequestedByKind: actor.kind, + finishedAt: at, + state: "cancelled", + stateVersion: run.stateVersion + 1, + terminalCode: input.terminalCode, + terminalMessage: input.terminalMessage, + updatedAt: at, + }) + .where( + and( + eq(jobRuns.id, run.id), + eq(jobRuns.state, "queued"), + eq(jobRuns.stateVersion, run.stateVersion) + ) + ) + .returning() + .get(); + requiredRow(row, "queued run cancellation"); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "cancel-requested", + occurredAt: at, + workerInstanceId: null, + }); + this.#appendEvent(run.id, { + attempt: run.attemptCount, + kind: "cancelled", + message: boundedStructuralMessage(input.terminalMessage), + occurredAt: at, + workerInstanceId: null, + }); + return requiredRow(this.findRun(run.id), "cancelled run refresh"); + } + + #claimFence(input: ClaimFenceInput): SQL { + return and( + eq(jobRuns.id, input.runId), + eq(jobRuns.state, "running"), + eq(jobRuns.leaseOwnerId, input.workerId), + eq(jobRuns.leaseToken, input.leaseToken), + gt(jobRuns.leaseExpiresAt, input.at) + ) as SQL; + } + + #findFencedRun(input: ClaimFenceInput): JobRunRecord | undefined { + const row = this.#transaction + .select() + .from(jobRuns) + .where(this.#claimFence(input)) + .get(); + return row === undefined ? undefined : parseRun(row); + } + + #findRunByIdempotency( + requestedByKind: JobRunRecord["requestedByKind"], + requestedById: string, + idempotencyKey: string + ): JobRunRecord | undefined { + const row = this.#transaction + .select() + .from(jobRuns) + .where( + and( + eq(jobRuns.requestedByKind, requestedByKind), + eq(jobRuns.requestedById, requestedById), + eq(jobRuns.idempotencyKey, idempotencyKey) + ) + ) + .get(); + return row === undefined ? undefined : parseRun(row); + } + + #findQueuedScheduleRun(scheduleId: string): JobRunRecord | undefined { + const row = this.#transaction + .select() + .from(jobRuns) + .where( + and( + eq(jobRuns.scheduledJobId, scheduleId), + eq(jobRuns.triggerType, "schedule"), + eq(jobRuns.state, "queued") + ) + ) + .get(); + return row === undefined ? undefined : parseRun(row); + } + + #closeExpiredIntent(input: { + readonly at: Date; + readonly context: string; + readonly intent: JobDisableIntentRecord; + readonly systemActorId: string; + readonly transitionAt: Date; + }): JobDisableIntentRecord { + const row = this.#transaction + .update(jobDisableIntents) + .set( + v.parse(jobDisableIntentCloseSchema, { + endedAt: input.transitionAt, + endedById: input.systemActorId, + endedByKind: "system", + endedReason: "expired", + }) + ) + .where( + and( + eq(jobDisableIntents.id, input.intent.id), + isNull(jobDisableIntents.endedAt), + lte(jobDisableIntents.expiresAt, input.at) + ) + ) + .returning() + .get(); + return parseDisableIntent(requiredRow(row, input.context)); + } + + #findScheduleRecord(id: string): ScheduledJobRecord | undefined { + const row = this.#transaction + .select() + .from(scheduledJobs) + .where(eq(scheduledJobs.id, id)) + .get(); + return row === undefined ? undefined : parseSchedule(row); + } + + #findWorker(id: string): WorkerInstanceRecord | undefined { + const row = this.#transaction + .select() + .from(workerInstances) + .where(eq(workerInstances.id, id)) + .get(); + return row === undefined ? undefined : parseWorker(row); + } + + #insertSideEffects(input: JobMutationSideEffects): void { + for (const event of input.auditEvents) { + this.#transaction + .insert(auditEvents) + .values(v.parse(auditEventInsertSchema, event)) + .run(); + } + for (const event of input.realtimeEvents) { + this.#transaction + .insert(realtimeEvents) + .values(v.parse(realtimeEventInsertSchema, event)) + .run(); + } + } + + #insertSuppliedEvent(input: JobRunEventInsert): JobRunEventRecord { + const row = this.#transaction + .insert(jobRunEvents) + .values(v.parse(jobRunEventInsertSchema, input)) + .returning() + .get(); + return parseEvent(requiredRow(row, "supplied run event insert")); + } + + #releaseResources(run: JobRunRecord, workerId: string, leaseToken: string): void { + const expectedKeys = resourceKeys(run); + const released = this.#transaction + .delete(resourceLeases) + .where( + and( + eq(resourceLeases.jobRunId, run.id), + eq(resourceLeases.workerInstanceId, workerId), + eq(resourceLeases.leaseToken, leaseToken) + ) + ) + .returning({ resourceKey: resourceLeases.resourceKey }) + .all(); + if ( + released.length !== expectedKeys.length || + released.some(({ resourceKey }) => !expectedKeys.includes(resourceKey)) + ) { + throw new Error("Claim resource lease set is incomplete during release"); + } + } + + #scheduleColumns(schedule: ScheduleConfiguration) { + if (schedule.kind === "interval") { + return { + cronExpression: null, + intervalMs: schedule.intervalMs, + scheduleKind: "interval" as const, + timeOfDay: null, + timeZone: null, + }; + } + if (schedule.kind === "daily") { + return { + cronExpression: null, + intervalMs: null, + scheduleKind: "daily" as const, + timeOfDay: schedule.timeOfDay, + timeZone: schedule.timeZone, + }; + } + return { + cronExpression: schedule.expression, + intervalMs: null, + scheduleKind: "cron" as const, + timeOfDay: null, + timeZone: schedule.timeZone, + }; + } + + #touchRunForEvent(run: JobRunRecord, requestedAt: Date): Date { + const at = maximumDate(run.updatedAt, requestedAt); + if (getTime(at) === getTime(run.updatedAt)) return at; + const row = this.#transaction + .update(jobRuns) + .set({ updatedAt: at }) + .where( + and( + eq(jobRuns.id, run.id), + eq(jobRuns.state, "running"), + eq(jobRuns.stateVersion, run.stateVersion) + ) + ) + .returning({ id: jobRuns.id }) + .get(); + requiredRow(row, "run event timestamp update"); + return at; + } + + #transitionClaim( + run: JobRunRecord, + changes: Partial + ): JobRunRecord { + const leaseOwnerId = requiredValue(run.leaseOwnerId, "claim owner"); + const leaseToken = requiredValue(run.leaseToken, "claim token"); + const row = this.#transaction + .update(jobRuns) + .set(changes) + .where( + and( + eq(jobRuns.id, run.id), + eq(jobRuns.state, "running"), + eq(jobRuns.stateVersion, run.stateVersion), + eq(jobRuns.leaseOwnerId, leaseOwnerId), + eq(jobRuns.leaseToken, leaseToken) + ) + ) + .returning() + .get(); + return parseRun(requiredRow(row, "claim transition")); + } +} + +/** + * Creates the SQLite-backed durable jobs repository. + * @param database Process-owned synchronous Drizzle SQLite database. + * @param writeAdmission Process-owned bounded immediate-write admission. + * @returns Validated reads plus admitted atomic schedule/queue/worker transitions. + */ +export function createJobRepository( + database: SQLiteBunDatabase, + writeAdmission: ImmediateDatabaseWriteAdmission +): JobRepository { + // Drizzle exposes the synchronous transaction overload through a conditional + // return type that cannot preserve our generic callback without this narrowing. + const runTransaction = database.transaction.bind(database) as unknown as ( + callback: (transaction: JobTransaction) => T, + config: { behavior: "deferred" | "immediate" } + ) => T; + const read = (callback: (reader: DrizzleJobReader) => T): T => + runTransaction((transaction) => callback(new DrizzleJobReader(transaction)), { + behavior: "deferred", + }); + const write = (callback: (writer: DrizzleJobWriter) => T): Promise => + writeAdmission.run((markTransactionStarted) => + runTransaction( + (transaction) => { + markTransactionStarted(); + return callback(new DrizzleJobWriter(transaction)); + }, + { behavior: "immediate" } + ) + ); + + return Object.freeze({ + appendClaimEvent: (input: AppendClaimEventInput) => + write((writer) => writer.appendClaimEvent(input)), + beginWorkerDrain: (input: WorkerLifecycleMutationInput) => + write((writer) => writer.beginWorkerDrain(input)), + cancelRun: (input: CancelRunRepositoryInput) => + write((writer) => writer.cancelRun(input)), + claimNextRun: (input: ClaimNextRunInput) => + write((writer) => writer.claimNextRun(input)), + enqueueManualRun: (input: EnqueueManualRunInput) => + write((writer) => writer.enqueueManualRun(input)), + enqueueNextDueSchedule: (input: DueScheduleEnqueueInput) => + write((writer) => writer.enqueueNextDueSchedule(input)), + expireDisableIntents: (input: ExpireDisableIntentsInput) => + write((writer) => writer.expireDisableIntents(input)), + findActiveDisableIntent: (scheduleId: string) => + read((reader) => reader.findActiveDisableIntent(scheduleId)), + findActiveRunForSchedule: (scheduleId: string) => + read((reader) => reader.findActiveRunForSchedule(scheduleId)), + findLatestRunForSchedule: (scheduleId: string) => + read((reader) => reader.findLatestRunForSchedule(scheduleId)), + findRun: (id: string) => read((reader) => reader.findRun(id)), + findRunDetail: (input: ListJobRunEventsInput) => + read((reader) => reader.findRunDetail(input)), + findSchedule: (id: string) => read((reader) => reader.findSchedule(id)), + heartbeatWorker: (input: WorkerLifecycleInput) => + write((writer) => writer.heartbeatWorker(input)), + listDueSchedules: (input: ListDueSchedulesInput) => + read((reader) => reader.listDueSchedules(input)), + listRunEvents: (input: ListJobRunEventsInput) => + read((reader) => reader.listRunEvents(input)), + listRuns: (input: ListJobRunsInput) => read((reader) => reader.listRuns(input)), + listRunsWithQueueState: (input: ListJobRunsWithQueueStateInput) => + read((reader) => reader.listRunsWithQueueState(input)), + listScheduleRuns: (input: ListScheduleRunsInput) => + read((reader) => reader.listScheduleRuns(input)), + listSchedules: (input: ListSchedulesInput) => + read((reader) => reader.listSchedules(input)), + readClaimCancellation: (input: ClaimFenceInput) => + read((reader) => reader.readClaimCancellation(input)), + readQueueState: (input: ReadQueueStateInput) => + read((reader) => reader.readQueueState(input)), + readWorkerControl: () => read((reader) => reader.readWorkerControl()), + reconcileSchedules: (input: ReconcileSchedulesInput) => + write((writer) => writer.reconcileSchedules(input)), + recoverExpiredClaims: (input: RecoverExpiredClaimsInput) => + write((writer) => writer.recoverExpiredClaims(input)), + registerWorker: (input: RegisterWorkerInput) => + write((writer) => writer.registerWorker(input)), + renewClaim: (input: RenewClaimInput) => + write((writer) => writer.renewClaim(input)), + setClaimingPaused: (input: SetClaimingPausedRepositoryInput) => + write((writer) => writer.setClaimingPaused(input)), + settleClaim: (input: SettleClaimInput) => + write((writer) => writer.settleClaim(input)), + stopWorker: (input: WorkerLifecycleMutationInput) => + write((writer) => writer.stopWorker(input)), + updateSchedule: (input: UpdateScheduleRepositoryInput) => + write((writer) => writer.updateSchedule(input)), + }); +} diff --git a/greenfield/src/server/domains/jobs/routes.ts b/greenfield/src/server/domains/jobs/routes.ts new file mode 100644 index 000000000..78b09d906 --- /dev/null +++ b/greenfield/src/server/domains/jobs/routes.ts @@ -0,0 +1,116 @@ +import { TRPCError } from "@trpc/server"; +import { Effect } from "effect"; + +import { + jobRunSummarySchema, + jobWorkerControlSchema, + scheduleSummarySchema, +} from "../../../contracts/jobModel.ts"; +import { + cancelJobRunInputSchema, + getJobRunInputSchema, + jobRunDetailSchema, + listJobRunsInputSchema, + listJobRunsResultSchema, + setJobClaimingPausedInputSchema, +} from "../../../contracts/jobs.ts"; +import { + getScheduleInputSchema, + listScheduleRunsInputSchema, + listScheduleRunsResultSchema, + listSchedulesInputSchema, + listSchedulesResultSchema, + runScheduleInputSchema, + updateScheduleInputSchema, +} from "../../../contracts/schedules.ts"; +import { capabilityProcedure, principalKindProcedure } from "../../trpc/trpc.ts"; +import { JobConflictError, JobNotFoundError, JobValidationError } from "./errors.ts"; + +async function runJobEffect(effect: Effect.Effect): Promise { + try { + return await Effect.runPromise(effect); + } catch (error) { + if (error instanceof JobNotFoundError) { + throw new TRPCError({ + cause: error, + code: "NOT_FOUND", + message: "Job resource was not found", + }); + } + if (error instanceof JobConflictError) { + throw new TRPCError({ + cause: error, + code: "CONFLICT", + message: "Job state changed concurrently", + }); + } + if (error instanceof JobValidationError) { + throw new TRPCError({ + cause: error, + code: "BAD_REQUEST", + message: "Schedule update is no longer valid", + }); + } + throw error; + } +} + +const readProcedure = capabilityProcedure("jobs:read"); +const sessionWriteProcedure = principalKindProcedure( + "jobs:write", + "session", + "A user session is required" +); +const runProcedure = capabilityProcedure("jobs:write"); + +/** Capability-scoped durable run and worker-control routes. */ +export const jobRoutes = { + cancelRun: sessionWriteProcedure + .input(cancelJobRunInputSchema) + .output(jobRunSummarySchema) + .mutation(({ ctx, input }) => + runJobEffect(ctx.jobService.cancelRun(ctx.principal, input)) + ), + getRun: readProcedure + .input(getJobRunInputSchema) + .output(jobRunDetailSchema) + .query(({ ctx, input }) => runJobEffect(ctx.jobService.getRun(input))), + listRuns: readProcedure + .input(listJobRunsInputSchema) + .output(listJobRunsResultSchema) + .query(({ ctx, input }) => runJobEffect(ctx.jobService.listRuns(input))), + setClaimingPaused: sessionWriteProcedure + .input(setJobClaimingPausedInputSchema) + .output(jobWorkerControlSchema) + .mutation(({ ctx, input }) => + runJobEffect(ctx.jobService.setClaimingPaused(ctx.principal, input)) + ), +}; + +/** Capability-scoped Dashboard-local schedule routes. */ +export const scheduleRoutes = { + get: readProcedure + .input(getScheduleInputSchema) + .output(scheduleSummarySchema) + .query(({ ctx, input }) => runJobEffect(ctx.jobService.getSchedule(input))), + list: readProcedure + .input(listSchedulesInputSchema) + .output(listSchedulesResultSchema) + .query(({ ctx, input }) => runJobEffect(ctx.jobService.listSchedules(input))), + listRuns: readProcedure + .input(listScheduleRunsInputSchema) + .output(listScheduleRunsResultSchema) + .query(({ ctx, input }) => runJobEffect(ctx.jobService.listScheduleRuns(input))), + run: runProcedure + .input(runScheduleInputSchema) + .output(jobRunSummarySchema) + .mutation(({ ctx, input }) => + runJobEffect(ctx.jobService.runSchedule(ctx.principal, input)) + ), + update: sessionWriteProcedure + .input(updateScheduleInputSchema) + .output(scheduleSummarySchema) + .mutation(({ ctx, input }) => + runJobEffect(ctx.jobService.updateSchedule(ctx.principal, input)) + ), +}; diff --git a/greenfield/src/server/domains/jobs/scheduleTime.test.ts b/greenfield/src/server/domains/jobs/scheduleTime.test.ts new file mode 100644 index 000000000..da7ec94f3 --- /dev/null +++ b/greenfield/src/server/domains/jobs/scheduleTime.test.ts @@ -0,0 +1,58 @@ +import { describe, expect, test } from "bun:test"; + +import { nextScheduleOccurrence } from "./scheduleTime.ts"; + +describe("durable schedule occurrence calculation", () => { + test("advances interval cadence from the original occurrence without drift", () => { + expect( + nextScheduleOccurrence( + { intervalMs: 60_000, kind: "interval" }, + 250_000, + 100_000 + ) + ).toBe(280_000); + }); + + test("keeps a retained interval occurrence that is already strictly future", () => { + expect( + nextScheduleOccurrence({ intervalMs: 60_000, kind: "interval" }, 1000, 60_000) + ).toBe(60_000); + }); + + test("uses Effect Cron's deterministic spring-gap normalization", () => { + const afterMs = Date.UTC(2026, 2, 29, 0, 0); + expect( + nextScheduleOccurrence( + { + kind: "daily", + timeOfDay: "02:30", + timeZone: "Europe/Oslo", + }, + afterMs + ) + ).toBe(Date.UTC(2026, 2, 29, 1, 30)); + }); + + test("chooses one fall-overlap occurrence and never duplicates cadence", () => { + const schedule = { + kind: "daily" as const, + timeOfDay: "02:30", + timeZone: "Europe/Oslo", + }; + const first = nextScheduleOccurrence(schedule, Date.UTC(2026, 9, 25, 0, 0)); + expect(first).toBe(Date.UTC(2026, 9, 25, 0, 30)); + expect(nextScheduleOccurrence(schedule, first ?? 0)).toBe( + Date.UTC(2026, 9, 26, 1, 30) + ); + }); + + test("returns no occurrence when the next timestamp overflows its contract", () => { + expect( + nextScheduleOccurrence( + { intervalMs: 60_000, kind: "interval" }, + 8_640_000_000_000_000, + 8_640_000_000_000_000 + ) + ).toBeUndefined(); + }); +}); diff --git a/greenfield/src/server/domains/jobs/scheduleTime.ts b/greenfield/src/server/domains/jobs/scheduleTime.ts new file mode 100644 index 000000000..e8d135030 --- /dev/null +++ b/greenfield/src/server/domains/jobs/scheduleTime.ts @@ -0,0 +1,60 @@ +import { Cron, Result } from "effect"; +import * as v from "valibot"; + +import { + type ScheduleConfiguration, + jobTimestampSchema, + scheduleConfigurationSchema, +} from "../../../contracts/jobModel.ts"; + +function parseSchedule(schedule: ScheduleConfiguration): ScheduleConfiguration { + return v.parse(scheduleConfigurationSchema, schedule); +} + +function nextCronOccurrence( + expression: string, + timeZone: string, + afterMs: number +): number | undefined { + const parsed = Cron.parse(expression, timeZone); + if (Result.isFailure(parsed)) return undefined; + try { + const next = Cron.next(parsed.success, new Date(afterMs)).getTime(); + return next > afterMs && v.safeParse(jobTimestampSchema, next).success + ? next + : undefined; + } catch { + return undefined; + } +} + +/** + * Computes the first occurrence strictly after a durable timestamp. + * @param schedule Canonical schedule variant. + * @param afterMs Exclusive lower bound. + * @param intervalAnchorMs Original interval occurrence used to avoid cadence drift. + * @returns The next valid occurrence, when one can be represented. + */ +export function nextScheduleOccurrence( + schedule: ScheduleConfiguration, + afterMs: number, + intervalAnchorMs = afterMs +): number | undefined { + const canonical = parseSchedule(schedule); + v.parse(jobTimestampSchema, afterMs); + v.parse(jobTimestampSchema, intervalAnchorMs); + if (canonical.kind === "interval") { + if (intervalAnchorMs > afterMs) return intervalAnchorMs; + const elapsed = Math.max(0, afterMs - intervalAnchorMs); + const steps = Math.floor(elapsed / canonical.intervalMs) + 1; + const next = intervalAnchorMs + steps * canonical.intervalMs; + return v.safeParse(jobTimestampSchema, next).success ? next : undefined; + } + if (canonical.kind === "daily") { + const [hourText, minuteText] = canonical.timeOfDay.split(":"); + const hour = Number(hourText); + const minute = Number(minuteText); + return nextCronOccurrence(`${minute} ${hour} * * *`, canonical.timeZone, afterMs); + } + return nextCronOccurrence(canonical.expression, canonical.timeZone, afterMs); +} diff --git a/greenfield/src/server/domains/jobs/service.test.ts b/greenfield/src/server/domains/jobs/service.test.ts new file mode 100644 index 000000000..123460fd7 --- /dev/null +++ b/greenfield/src/server/domains/jobs/service.test.ts @@ -0,0 +1,1342 @@ +import { describe, expect, test } from "bun:test"; + +import { and, asc, eq, gt, max } from "drizzle-orm"; +import { Effect } from "effect"; + +import { jobWorkerFreshnessMs } from "../../../contracts/jobModel.ts"; +import type { AuthenticatedPrincipal } from "../../../contracts/security.ts"; +import { auditEvents } from "../../database/schema/auditEvents.ts"; +import { realtimeEvents } from "../../database/schema/realtime.ts"; +import { testImmediateDatabaseWriteAdmission } from "../../test/support/databaseWriteAdmission.ts"; +import { + authenticationTestNow, + authenticationTestUserId, + openAuthenticationTestDatabase, +} from "../security/testSupport/authentication.ts"; +import { JobConflictError, JobValidationError } from "./errors.ts"; +import type { + JobRunEventRecord, + ScheduledJobRecord, + WorkerInstanceRecord, +} from "./records.ts"; +import { + createJobRepository, + type JobMutationSideEffects, + type JobRepository, + type JobRunInsert, +} from "./repository.ts"; +import { createJobService, reconcileJobSchedules } from "./service.ts"; + +function createIdGenerator(): () => string { + let index = 1; + return () => `019fdf20-0000-7000-8000-${String(index++).padStart(12, "0")}`; +} + +const serviceNowMs = () => authenticationTestNow.getTime(); +const noSideEffects: JobMutationSideEffects = Object.freeze({ + auditEvents: Object.freeze([]), + realtimeEvents: Object.freeze([]), +}); + +function scheduledRun(schedule: ScheduledJobRecord, at: Date, id: string): JobRunInsert { + if (schedule.nextRunAt === null) { + throw new Error("Expected an enabled schedule cursor"); + } + return { + actionKey: schedule.actionKey, + attemptLimit: schedule.attemptLimit, + availableAt: at, + cancellationPolicy: schedule.cancellationPolicy, + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: schedule.name, + enqueueSha256: "2".repeat(64), + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id, + idempotencyKey: "2".repeat(32), + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadJson: schedule.actionPayloadJson, + priority: schedule.priority, + queuedAt: at, + requestedById: "system.scheduler", + requestedByKind: "system", + resourceClass: schedule.resourceClass, + resourceKeysJson: schedule.resourceKeysJson, + resultJson: null, + retrySafe: schedule.retrySafe, + scheduledForAt: schedule.nextRunAt, + scheduledJobId: schedule.id, + scheduledJobVersion: schedule.version, + state: "queued", + terminalCode: null, + terminalMessage: null, + timeoutMs: schedule.timeoutMs, + triggerType: "schedule", + updatedAt: at, + }; +} + +describe("durable jobs service", () => { + test("accepts a full-form cadence edit while an enabled schedule stays enabled", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + expect( + Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { + disableIntent: { reason: "Already disabled" }, + enabled: false, + schedule: { intervalMs: 120_000, kind: "interval" }, + }, + }) + ) + ).rejects.toBeInstanceOf(JobValidationError); + await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { disableIntent: null, enabled: true }, + }) + ); + expect( + Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 2, + id: "system.worker-smoke", + patch: { + disableIntent: null, + enabled: true, + schedule: { + intervalMs: 86_400_000, + kind: "interval", + }, + }, + }) + ) + ).rejects.toBeInstanceOf(JobValidationError); + + const updated = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 2, + id: "system.worker-smoke", + patch: { + disableIntent: null, + enabled: true, + schedule: { intervalMs: 120_000, kind: "interval" }, + }, + }) + ); + + expect(updated).toMatchObject({ + enabled: true, + nextRunAtMs: authenticationTestNow.getTime() + 120_000, + schedule: { intervalMs: 120_000, kind: "interval" }, + version: 3, + }); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("atomically invalidates run and schedule projections for a manual enqueue", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + const previousEventId = + fixture.database.orm + .select({ value: max(realtimeEvents.id) }) + .from(realtimeEvents) + .get()?.value ?? 0; + + const run = await Effect.runPromise( + service.runSchedule(principal, { + id: "system.worker-smoke", + idempotencyKey: "e".repeat(32), + }) + ); + const addedEvents = fixture.database.orm + .select({ + entityId: realtimeEvents.entityId, + id: realtimeEvents.id, + topic: realtimeEvents.topic, + }) + .from(realtimeEvents) + .where(gt(realtimeEvents.id, previousEventId)) + .orderBy(asc(realtimeEvents.id)) + .all() + .map(({ entityId, topic }) => ({ entityId, topic })); + + expect(addedEvents).toEqual([ + { entityId: run.id, topic: "jobs.runs" }, + { entityId: "system.worker-smoke", topic: "schedules.records" }, + ]); + expect(repository.findRun(run.id)).toBeDefined(); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("builds direct cancellation side effects from the durable run snapshot", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + const workerId = "019fdf20-0000-7000-8000-000000000900"; + const leaseToken = "019fdf20-0000-7000-8000-000000000901"; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const setupService = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + const queued = await Effect.runPromise( + setupService.runSchedule(principal, { + id: "system.worker-smoke", + idempotencyKey: "3".repeat(32), + }) + ); + const queuedRecord = repository.findRun(queued.id); + if (queuedRecord === undefined) throw new Error("Missing queued run"); + const transitionAt = new Date(authenticationTestNow.getTime() + 60_000); + await repository.registerWorker({ + ...noSideEffects, + worker: { + capacity: 1, + drainingAt: null, + heartbeatAt: transitionAt, + id: workerId, + pid: 1234, + releaseId: "a".repeat(40), + startedAt: transitionAt, + state: "online", + stoppedAt: null, + }, + }); + expect( + await repository.claimNextRun({ + at: transitionAt, + leaseExpiresAt: new Date(transitionAt.getTime() + 30_000), + leaseToken, + minimumHeartbeatAt: transitionAt, + sideEffectsForClaim: () => noSideEffects, + workerId, + }) + ).toMatchObject({ + kind: "claimed", + run: { id: queued.id, updatedAt: transitionAt }, + }); + const previousEventId = + fixture.database.orm + .select({ value: max(realtimeEvents.id) }) + .from(realtimeEvents) + .get()?.value ?? 0; + const staleReadRepository: JobRepository = { + ...repository, + findRun: (id) => + id === queued.id ? queuedRecord : repository.findRun(id), + }; + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository: staleReadRepository, + }); + + const result = await Effect.runPromise( + service.cancelRun(principal, { id: queued.id }) + ); + const replay = await Effect.runPromise( + service.cancelRun(principal, { id: queued.id }) + ); + const addedEvents = fixture.database.orm + .select({ + entityId: realtimeEvents.entityId, + occurredAt: realtimeEvents.occurredAt, + topic: realtimeEvents.topic, + }) + .from(realtimeEvents) + .where(gt(realtimeEvents.id, previousEventId)) + .orderBy(asc(realtimeEvents.id)) + .all(); + const cancellationAudits = fixture.database.orm + .select({ + action: auditEvents.action, + occurredAt: auditEvents.occurredAt, + outcome: auditEvents.outcome, + }) + .from(auditEvents) + .where( + and( + eq(auditEvents.action, "jobs.run.cancel"), + eq(auditEvents.targetId, queued.id) + ) + ) + .all(); + + expect(result).toMatchObject({ + cancelRequestedAtMs: transitionAt.getTime(), + id: queued.id, + state: "running", + updatedAtMs: transitionAt.getTime(), + }); + expect(replay).toEqual(result); + expect(addedEvents).toEqual([ + { + entityId: queued.id, + occurredAt: transitionAt, + topic: "jobs.runs", + }, + { + entityId: "system.worker-smoke", + occurredAt: transitionAt, + topic: "schedules.records", + }, + ]); + expect(cancellationAudits).toEqual([ + { + action: "jobs.run.cancel", + occurredAt: transitionAt, + outcome: "accepted", + }, + ]); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("timestamps disable cancellation side effects from the cancelled run", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { disableIntent: null, enabled: true }, + }) + ); + const relation = repository.findSchedule("system.worker-smoke"); + if (relation === undefined) throw new Error("Missing enabled schedule"); + const schedule = relation.schedule; + const runAt = schedule.nextRunAt; + if (runAt === null) throw new Error("Missing enabled schedule cursor"); + if (schedule.intervalMs === null) { + throw new Error("Expected the smoke interval schedule"); + } + const run = scheduledRun( + schedule, + runAt, + "019fdf20-0000-7000-8000-000000000902" + ); + expect( + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: runAt, + nextRunAt: new Date(runAt.getTime() + schedule.intervalMs), + observedNextRunAt: runAt, + run, + scheduleId: schedule.id, + }) + ).toMatchObject({ kind: "inserted", run: { id: run.id } }); + const previousEventId = + fixture.database.orm + .select({ value: max(realtimeEvents.id) }) + .from(realtimeEvents) + .get()?.value ?? 0; + + await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 2, + id: schedule.id, + patch: { + disableIntent: { reason: "Clock-regression maintenance" }, + enabled: false, + }, + }) + ); + const cancelled = repository.findRun(run.id); + const addedEvents = fixture.database.orm + .select({ + entityId: realtimeEvents.entityId, + occurredAt: realtimeEvents.occurredAt, + topic: realtimeEvents.topic, + }) + .from(realtimeEvents) + .where(gt(realtimeEvents.id, previousEventId)) + .orderBy(asc(realtimeEvents.id)) + .all(); + const cancellationAudits = fixture.database.orm + .select({ + occurredAt: auditEvents.occurredAt, + outcome: auditEvents.outcome, + }) + .from(auditEvents) + .where( + and( + eq(auditEvents.action, "jobs.run.cancel"), + eq(auditEvents.targetId, run.id) + ) + ) + .all(); + + expect(cancelled).toMatchObject({ + state: "cancelled", + updatedAt: runAt, + }); + expect(addedEvents).toEqual([ + { entityId: run.id, occurredAt: runAt, topic: "jobs.runs" }, + { + entityId: schedule.id, + occurredAt: runAt, + topic: "schedules.records", + }, + { + entityId: schedule.id, + occurredAt: authenticationTestNow, + topic: "schedules.records", + }, + ]); + expect(cancellationAudits).toEqual([ + { occurredAt: runAt, outcome: "cancelled" }, + ]); + expect(repository.findSchedule(schedule.id)?.schedule).toMatchObject({ + enabled: false, + updatedAt: authenticationTestNow, + version: 3, + }); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("preserves an active disable intent across a schedule-only update", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ + generateId, + nowMs: serviceNowMs, + repository, + }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + expect( + Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { + schedule: { + intervalMs: 86_400_000, + kind: "interval", + }, + }, + }) + ) + ).rejects.toBeInstanceOf(JobValidationError); + expect( + Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { + disableIntent: { reason: "Already disabled" }, + enabled: false, + }, + }) + ) + ).rejects.toBeInstanceOf(JobValidationError); + const unchanged = repository.findSchedule("system.worker-smoke"); + expect(unchanged?.activeDisableIntent).toBeUndefined(); + expect(unchanged?.schedule).toMatchObject({ enabled: false, version: 1 }); + + const enabled = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { disableIntent: null, enabled: true }, + }) + ); + expect(enabled.nextRunAtMs).toBe( + authenticationTestNow.getTime() + 86_400_000 + ); + const disabled = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 2, + id: "system.worker-smoke", + patch: { + disableIntent: { reason: "Operator maintenance" }, + enabled: false, + }, + }) + ); + const originalIntent = disabled.activeDisableIntent; + expect(disabled).toMatchObject({ + enabled: false, + version: 3, + }); + expect(disabled.nextRunAtMs).toBeUndefined(); + expect( + repository.findSchedule("system.worker-smoke")?.schedule.nextRunAt + ).toEqual(new Date(enabled.nextRunAtMs!)); + expect(originalIntent).toMatchObject({ + reason: "Operator maintenance", + }); + + const updated = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 3, + id: "system.worker-smoke", + patch: { + schedule: { intervalMs: 120_000, kind: "interval" }, + }, + }) + ); + + expect(updated).toMatchObject({ + activeDisableIntent: originalIntent, + enabled: false, + schedule: { intervalMs: 120_000, kind: "interval" }, + version: 4, + }); + expect( + repository.findSchedule("system.worker-smoke")?.activeDisableIntent + ).toMatchObject({ + endedAt: null, + id: originalIntent?.id, + reason: "Operator maintenance", + }); + expect( + repository.findSchedule("system.worker-smoke")?.schedule.nextRunAt + ).toEqual(new Date(authenticationTestNow.getTime() + 120_000)); + + const replacement = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 4, + id: "system.worker-smoke", + patch: { + disableIntent: { + expiresAtMs: authenticationTestNow.getTime() + 600_000, + reason: "Extended operator maintenance", + }, + enabled: false, + }, + }) + ); + expect(replacement).toMatchObject({ + activeDisableIntent: { + reason: "Extended operator maintenance", + }, + enabled: false, + version: 5, + }); + expect(replacement.activeDisableIntent?.id).not.toBe(originalIntent?.id); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("stores the recalculated dormant cursor when cadence changes during disable", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + const enabled = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { disableIntent: null, enabled: true }, + }) + ); + const disabled = await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 2, + id: "system.worker-smoke", + patch: { + disableIntent: { reason: "Change cadence during maintenance" }, + enabled: false, + schedule: { intervalMs: 120_000, kind: "interval" }, + }, + }) + ); + + expect(disabled).toMatchObject({ + enabled: false, + schedule: { intervalMs: 120_000, kind: "interval" }, + version: 3, + }); + expect(disabled.nextRunAtMs).toBeUndefined(); + expect(enabled.nextRunAtMs).toBe( + authenticationTestNow.getTime() + 86_400_000 + ); + expect( + repository.findSchedule("system.worker-smoke")?.schedule.nextRunAt + ).toEqual(new Date(authenticationTestNow.getTime() + 120_000)); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("atomically cancels queued schedule work retired from the registry", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const retiredScheduleId = "system.worker-smoke-retired"; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const registered = repository.findSchedule("system.worker-smoke")?.schedule; + if (registered === undefined) throw new Error("Missing registered schedule"); + const runAt = new Date(authenticationTestNow.getTime() + 60_000); + await repository.reconcileSchedules({ + at: authenticationTestNow, + schedules: [ + registered, + { + ...registered, + enabled: true, + id: retiredScheduleId, + nextRunAt: runAt, + }, + ], + sideEffectsForSchedule: () => noSideEffects, + }); + const retiredSchedule = repository.findSchedule(retiredScheduleId)?.schedule; + if (retiredSchedule === undefined) { + throw new Error("Missing retired schedule fixture"); + } + if (retiredSchedule.intervalMs === null) { + throw new Error("Expected the smoke interval schedule"); + } + const run = scheduledRun( + retiredSchedule, + runAt, + "019fdf20-0000-7000-8000-000000000903" + ); + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: runAt, + nextRunAt: new Date(runAt.getTime() + retiredSchedule.intervalMs), + observedNextRunAt: runAt, + run, + scheduleId: retiredScheduleId, + }); + const previousEventId = + fixture.database.orm + .select({ value: max(realtimeEvents.id) }) + .from(realtimeEvents) + .get()?.value ?? 0; + + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + + expect(repository.findRun(run.id)).toMatchObject({ + eventCount: 3, + state: "cancelled", + terminalCode: "cancelled/schedule-retired", + updatedAt: runAt, + }); + expect(repository.findSchedule(retiredScheduleId)?.schedule).toMatchObject({ + enabled: false, + updatedAt: authenticationTestNow, + version: 2, + }); + expect( + fixture.database.orm + .select({ + entityId: realtimeEvents.entityId, + occurredAt: realtimeEvents.occurredAt, + topic: realtimeEvents.topic, + }) + .from(realtimeEvents) + .where(gt(realtimeEvents.id, previousEventId)) + .orderBy(asc(realtimeEvents.id)) + .all() + ).toEqual([ + { entityId: run.id, occurredAt: runAt, topic: "jobs.runs" }, + { + entityId: retiredScheduleId, + occurredAt: runAt, + topic: "schedules.records", + }, + { + entityId: retiredScheduleId, + occurredAt: authenticationTestNow, + topic: "schedules.records", + }, + ]); + expect( + fixture.database.orm + .select({ + action: auditEvents.action, + actorId: auditEvents.actorId, + actorKind: auditEvents.actorKind, + occurredAt: auditEvents.occurredAt, + outcome: auditEvents.outcome, + }) + .from(auditEvents) + .where( + and( + eq(auditEvents.action, "jobs.run.cancel"), + eq(auditEvents.targetId, run.id) + ) + ) + .all() + ).toEqual([ + { + action: "jobs.run.cancel", + actorId: "jobs-scheduler", + actorKind: "system", + occurredAt: runAt, + outcome: "cancelled", + }, + ]); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("retires a schedule without cancelling its queued never-cancellable run", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const retiredScheduleId = "system.worker-smoke-never-retired"; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const registered = repository.findSchedule("system.worker-smoke")?.schedule; + if (registered === undefined) throw new Error("Missing registered schedule"); + const runAt = new Date(authenticationTestNow.getTime() + 60_000); + await repository.reconcileSchedules({ + at: authenticationTestNow, + schedules: [ + registered, + { + ...registered, + cancellationPolicy: "never", + enabled: true, + id: retiredScheduleId, + nextRunAt: runAt, + }, + ], + sideEffectsForSchedule: () => noSideEffects, + }); + const retiredSchedule = repository.findSchedule(retiredScheduleId)?.schedule; + if (retiredSchedule === undefined) { + throw new Error("Missing never-cancellable retired schedule fixture"); + } + if (retiredSchedule.intervalMs === null) { + throw new Error("Expected the smoke interval schedule"); + } + const run = scheduledRun( + retiredSchedule, + runAt, + "019fdf20-0000-7000-8000-000000000904" + ); + await repository.enqueueNextDueSchedule({ + ...noSideEffects, + at: runAt, + nextRunAt: new Date(runAt.getTime() + retiredSchedule.intervalMs), + observedNextRunAt: runAt, + run, + scheduleId: retiredScheduleId, + }); + const previousEventId = + fixture.database.orm + .select({ value: max(realtimeEvents.id) }) + .from(realtimeEvents) + .get()?.value ?? 0; + + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + + expect(repository.findRun(run.id)).toMatchObject({ + cancelRequestedAt: null, + eventCount: 1, + state: "queued", + terminalCode: null, + updatedAt: runAt, + }); + expect(repository.findSchedule(retiredScheduleId)?.schedule).toMatchObject({ + enabled: false, + updatedAt: authenticationTestNow, + version: 2, + }); + expect( + fixture.database.orm + .select({ + entityId: realtimeEvents.entityId, + occurredAt: realtimeEvents.occurredAt, + topic: realtimeEvents.topic, + }) + .from(realtimeEvents) + .where(gt(realtimeEvents.id, previousEventId)) + .orderBy(asc(realtimeEvents.id)) + .all() + ).toEqual([ + { + entityId: retiredScheduleId, + occurredAt: authenticationTestNow, + topic: "schedules.records", + }, + ]); + expect( + fixture.database.orm + .select({ action: auditEvents.action }) + .from(auditEvents) + .where( + and( + eq(auditEvents.action, "jobs.run.cancel"), + eq(auditEvents.targetId, run.id) + ) + ) + .all() + ).toEqual([]); + + const eventCount = fixture.database.orm + .select() + .from(realtimeEvents) + .all().length; + const auditCount = fixture.database.orm + .select() + .from(auditEvents) + .all().length; + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + expect(fixture.database.orm.select().from(realtimeEvents).all()).toHaveLength( + eventCount + ); + expect(fixture.database.orm.select().from(auditEvents).all()).toHaveLength( + auditCount + ); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("rejects mutations for a schedule outside the exact action registry pair", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const registered = repository.findSchedule("system.worker-smoke")?.schedule; + if (registered === undefined) throw new Error("Missing registered schedule"); + await repository.reconcileSchedules({ + at: authenticationTestNow, + schedules: [{ ...registered, id: "system.worker-smoke-retired" }], + sideEffectsForSchedule: () => ({ + auditEvents: [], + realtimeEvents: [], + }), + }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + + const runError = await Effect.runPromise( + service.runSchedule(principal, { + id: "system.worker-smoke-retired", + idempotencyKey: "f".repeat(32), + }) + ).catch((error: unknown) => error); + expect(runError).toBeInstanceOf(JobConflictError); + expect(runError).toMatchObject({ reason: "action-unavailable" }); + expect( + Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke-retired", + patch: { disableIntent: null, enabled: true }, + }) + ) + ).rejects.toBeInstanceOf(JobConflictError); + expect( + repository.findSchedule("system.worker-smoke-retired")?.schedule + ).toMatchObject({ enabled: false, version: 1 }); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("rejects a manual enqueue when code metadata changes after the service read", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + let attemptedRunId: string | undefined; + const interleavedRepository: JobRepository = { + ...repository, + enqueueManualRun: async (input) => { + attemptedRunId = input.run.id; + const current = repository.findSchedule( + input.run.scheduledJobId ?? "" + )?.schedule; + if (current === undefined) { + throw new Error("Missing schedule for interleaved enqueue"); + } + await repository.reconcileSchedules({ + at: new Date(authenticationTestNow.getTime() + 1), + schedules: [ + { + ...current, + name: "Worker smoke from the next release", + updatedAt: new Date(authenticationTestNow.getTime() + 1), + }, + ], + sideEffectsForSchedule: () => noSideEffects, + }); + return repository.enqueueManualRun(input); + }, + }; + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository: interleavedRepository, + }); + + const error = await Effect.runPromise( + service.runSchedule(principal, { + id: "system.worker-smoke", + idempotencyKey: "d".repeat(32), + }) + ).catch((error: unknown) => error); + + expect(error).toBeInstanceOf(JobConflictError); + expect(error).toMatchObject({ reason: "action-unavailable" }); + expect(attemptedRunId).toBeDefined(); + expect(repository.findRun(attemptedRunId ?? "")).toBeUndefined(); + expect( + repository.findSchedule("system.worker-smoke")?.schedule + ).toMatchObject({ + name: "Worker smoke from the next release", + version: 2, + }); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("maps a never-cancellable queued schedule run to a declared conflict", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ generateId, nowMs: serviceNowMs, repository }); + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + await Effect.runPromise( + service.updateSchedule(principal, { + expectedVersion: 1, + id: "system.worker-smoke", + patch: { disableIntent: null, enabled: true }, + }) + ); + const runSummary = await Effect.runPromise( + service.runSchedule(principal, { + id: "system.worker-smoke", + idempotencyKey: "1".repeat(32), + }) + ); + const run = repository.findRun(runSummary.id); + if (run === undefined) throw new Error("Expected the manual run fixture"); + const conflictRepository: JobRepository = { + ...repository, + updateSchedule: () => + Promise.resolve({ + kind: "cancellation-not-supported", + run, + }), + }; + const conflictService = createJobService({ + generateId, + nowMs: serviceNowMs, + repository: conflictRepository, + }); + + expect( + Effect.runPromise( + conflictService.updateSchedule(principal, { + expectedVersion: 2, + id: "system.worker-smoke", + patch: { + disableIntent: { reason: "Maintenance" }, + enabled: false, + }, + }) + ) + ).rejects.toBeInstanceOf(JobConflictError); + const unchangedSchedule = repository.findSchedule("system.worker-smoke"); + expect(unchangedSchedule?.activeDisableIntent).toBeUndefined(); + expect(unchangedSchedule?.schedule).toMatchObject({ + enabled: true, + version: 2, + }); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("reads run and events from one snapshot across an interleaved worker transition", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const generateId = createIdGenerator(); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + + try { + await reconcileJobSchedules({ + generateId, + nowMs: serviceNowMs, + repository, + }); + const setupService = createJobService({ + generateId, + nowMs: serviceNowMs, + repository, + }); + const queued = await Effect.runPromise( + setupService.runSchedule(principal, { + id: "system.worker-smoke", + idempotencyKey: "A".repeat(32), + }) + ); + const snapshot = repository.findRunDetail({ + limit: 10, + runId: queued.id, + }); + if (snapshot === undefined) throw new Error("Missing queued run snapshot"); + + const interleavedEvent = { + attempt: 1, + jobRunId: queued.id, + kind: "claimed", + message: null, + occurredAt: new Date(snapshot.run.updatedAt.getTime() + 1), + progressJson: null, + sequence: 2, + workerInstanceId: "019fdf20-0000-7000-8000-000000000099", + } satisfies JobRunEventRecord; + const listSnapshot = repository.listRunsWithQueueState({ + limit: 10, + minimumHeartbeatAt: new Date(0), + }); + const expectedMinimumHeartbeatAt = new Date( + serviceNowMs() - jobWorkerFreshnessMs + ); + const workerStartedAt = new Date(expectedMinimumHeartbeatAt.getTime() - 1000); + const workerRecord = ( + id: string, + heartbeatAt: Date + ): WorkerInstanceRecord => ({ + capacity: 1, + drainingAt: null, + heartbeatAt, + id, + pid: 1234, + releaseId: "a".repeat(40), + startedAt: workerStartedAt, + state: "online", + stoppedAt: null, + }); + const staleWorker = workerRecord( + "019fdf20-0000-7000-8000-000000000097", + new Date(expectedMinimumHeartbeatAt.getTime() - 1) + ); + const boundaryWorker = workerRecord( + "019fdf20-0000-7000-8000-000000000098", + expectedMinimumHeartbeatAt + ); + let snapshotReads = 0; + let listSnapshotReads = 0; + let legacyRunReads = 0; + let legacyEventReads = 0; + let legacyListReads = 0; + let legacyQueueReads = 0; + let observedMinimumHeartbeatAt: Date | undefined; + const interleavedRepository: JobRepository = { + ...repository, + findRun: () => { + legacyRunReads += 1; + return snapshot.run; + }, + findRunDetail: () => { + snapshotReads += 1; + return snapshot; + }, + listRunEvents: () => { + legacyEventReads += 1; + return [interleavedEvent, ...snapshot.events]; + }, + listRuns: () => { + legacyListReads += 1; + return []; + }, + listRunsWithQueueState: (input) => { + listSnapshotReads += 1; + observedMinimumHeartbeatAt = input.minimumHeartbeatAt; + return { + ...listSnapshot, + queue: { + ...listSnapshot.queue, + workers: [staleWorker, boundaryWorker] + .filter( + (worker) => + worker.heartbeatAt.getTime() >= + input.minimumHeartbeatAt.getTime() + ) + .map((worker) => ({ activeRunCount: 0, worker })), + }, + }; + }, + readQueueState: () => { + legacyQueueReads += 1; + return listSnapshot.queue; + }, + }; + const service = createJobService({ + generateId, + nowMs: serviceNowMs, + repository: interleavedRepository, + }); + + const detail = await Effect.runPromise( + service.getRun({ eventLimit: 10, id: queued.id }) + ); + const listing = await Effect.runPromise(service.listRuns({ limit: 10 })); + + expect(detail.events.map(({ sequence }) => sequence)).toEqual([1]); + expect(detail.run).toMatchObject({ attemptCount: 0, eventCount: 1 }); + expect(listing.runs.map(({ id }) => id)).toEqual([queued.id]); + expect(observedMinimumHeartbeatAt).toEqual(expectedMinimumHeartbeatAt); + expect(listing.summary.workers.map(({ id }) => id)).toEqual([ + boundaryWorker.id, + ]); + expect(listing.summary.workers[0]?.heartbeatAtMs).toBe( + expectedMinimumHeartbeatAt.getTime() + ); + expect({ + legacyEventReads, + legacyListReads, + legacyQueueReads, + legacyRunReads, + listSnapshotReads, + snapshotReads, + }).toEqual({ + legacyEventReads: 0, + legacyListReads: 0, + legacyQueueReads: 0, + legacyRunReads: 0, + listSnapshotReads: 1, + snapshotReads: 1, + }); + } finally { + fixture.database.sqlite.close(true); + } + }); + + test("reads only worker control before changing the claiming state", async () => { + const fixture = await openAuthenticationTestDatabase(authenticationTestNow); + const repository = createJobRepository( + fixture.database.orm, + testImmediateDatabaseWriteAdmission + ); + const principal: AuthenticatedPrincipal = { + authorizationVersion: 1, + authenticatorId: fixture.session.prefix, + capabilities: ["jobs:read", "jobs:write"], + id: authenticationTestUserId, + kind: "session", + }; + let queueReads = 0; + let workerControlReads = 0; + const narrowRepository: JobRepository = { + ...repository, + readQueueState: () => { + queueReads += 1; + throw new Error("Queue summary should not be read before pausing claims"); + }, + readWorkerControl: () => { + workerControlReads += 1; + return repository.readWorkerControl(); + }, + }; + const service = createJobService({ + generateId: createIdGenerator(), + nowMs: serviceNowMs, + repository: narrowRepository, + }); + + try { + const control = await Effect.runPromise( + service.setClaimingPaused(principal, { + expectedVersion: 1, + paused: true, + }) + ); + + expect(control).toMatchObject({ claimingPaused: true, version: 2 }); + expect({ queueReads, workerControlReads }).toEqual({ + queueReads: 0, + workerControlReads: 1, + }); + } finally { + fixture.database.sqlite.close(true); + } + }); +}); diff --git a/greenfield/src/server/domains/jobs/service.ts b/greenfield/src/server/domains/jobs/service.ts new file mode 100644 index 000000000..b8f8601d1 --- /dev/null +++ b/greenfield/src/server/domains/jobs/service.ts @@ -0,0 +1,886 @@ +import { getTime, max as maximumDate, subMilliseconds, toDate } from "date-fns"; +import { Context, Data, Effect } from "effect"; +import * as v from "valibot"; + +import { + type JobRunSummary, + type JobWorkerControl, + type ScheduleConfiguration, + type ScheduleSummary, + jobWorkerFreshnessMs, + jobTimestampSchema, +} from "../../../contracts/jobModel.ts"; +import { + type JobRunDetail, + type ListJobRunsInput, + type ListJobRunsResult, + type CancelJobRunInput, + type GetJobRunInput, + type SetJobClaimingPausedInput, + jobRunDetailSchema, + listJobRunsResultSchema, +} from "../../../contracts/jobs.ts"; +import { + type GetScheduleInput, + type ListScheduleRunsInput, + type ListScheduleRunsResult, + type ListSchedulesInput, + type ListSchedulesResult, + type RunScheduleInput, + type UpdateScheduleInput, + listScheduleRunsResultSchema, + listSchedulesResultSchema, +} from "../../../contracts/schedules.ts"; +import type { AuthenticatedPrincipal } from "../../../contracts/security.ts"; +import { isDatabaseRuntimeWriteUnavailableError } from "../../database/runtime/databaseErrors.ts"; +import { sha256Hex } from "../../shared/crypto.ts"; +import { + type JobActionRegistration, + findJobActionRegistration, + isRegisteredJobSchedule, + jobActionRegistrations, +} from "./actionRegistry.ts"; +import { + JobConflictError, + JobNotFoundError, + type JobOperationError, + JobValidationError, +} from "./errors.ts"; +import { + type JobRunRecord, + toJobRunEvent, + toJobRunResult, + toJobRunSummary, + toJobWorkerControl, + toJobWorkerSummary, + toScheduleSummary, +} from "./records.ts"; +import { buildRegisteredSchedule } from "./registeredSchedule.ts"; +import { + type JobMutationSideEffects, + type JobRepository, + type ScheduleRecordWithRelations, +} from "./repository.ts"; +import { nextScheduleOccurrence } from "./scheduleTime.ts"; +import { + type JobAuditActor, + createJobMutationSideEffects, + createJobRealtimeSideEffects, +} from "./sideEffects.ts"; + +const systemActor = Object.freeze({ + authenticatorId: null, + id: "jobs-scheduler", + kind: "system", +} satisfies JobAuditActor); + +class JobUnexpectedOperationError extends Data.TaggedError( + "JobUnexpectedOperationError" +)<{ readonly cause: unknown }> {} + +interface JobServiceShape { + readonly cancelRun: ( + principal: AuthenticatedPrincipal, + input: CancelJobRunInput + ) => Effect.Effect; + readonly getRun: ( + input: GetJobRunInput + ) => Effect.Effect; + readonly getSchedule: ( + input: GetScheduleInput + ) => Effect.Effect; + readonly listRuns: (input: ListJobRunsInput) => Effect.Effect; + readonly listScheduleRuns: ( + input: ListScheduleRunsInput + ) => Effect.Effect; + readonly listSchedules: ( + input: ListSchedulesInput + ) => Effect.Effect; + readonly runSchedule: ( + principal: AuthenticatedPrincipal, + input: RunScheduleInput + ) => Effect.Effect; + readonly setClaimingPaused: ( + principal: AuthenticatedPrincipal, + input: SetJobClaimingPausedInput + ) => Effect.Effect; + readonly updateSchedule: ( + principal: AuthenticatedPrincipal, + input: UpdateScheduleInput + ) => Effect.Effect; +} + +/** Effect service for durable job inventory and Dashboard-local schedules. */ +export class JobService extends Context.Service()( + "mira-dashboard/server/domains/jobs/JobService" +) {} + +export interface JobServiceDependencies { + readonly generateId?: () => string; + readonly nowMs?: () => number; + readonly repository: JobRepository; + readonly wakeEventPump?: () => Promise | void; +} + +export type JobScheduleReconciliationDependencies = JobServiceDependencies; + +interface AuthenticatedJobActor { + readonly id: string; + readonly kind: "automation" | "user"; +} + +function principalActor(principal: AuthenticatedPrincipal): AuthenticatedJobActor { + return { + id: principal.id, + kind: principal.kind === "session" ? "user" : "automation", + }; +} + +function principalAuditActor(principal: AuthenticatedPrincipal): JobAuditActor { + return { + authenticatorId: principal.authenticatorId, + id: principal.id, + kind: principal.kind === "session" ? "user" : "automation", + }; +} + +function readEffect( + operation: () => T, + isExpected: (error: unknown) => error is E +): Effect.Effect { + return Effect.try({ + catch: (error) => + isExpected(error) ? error : new JobUnexpectedOperationError({ cause: error }), + try: operation, + }).pipe( + Effect.catchIf( + (error): error is JobUnexpectedOperationError => + error instanceof JobUnexpectedOperationError, + (error) => Effect.die(error.cause) + ) + ); +} + +function mutationEffect( + operation: () => Promise +): Effect.Effect { + return Effect.tryPromise({ + catch: (error) => + error instanceof JobConflictError || + error instanceof JobNotFoundError || + error instanceof JobValidationError || + isDatabaseRuntimeWriteUnavailableError(error) + ? error + : new JobUnexpectedOperationError({ cause: error }), + try: operation, + }).pipe( + Effect.catchTag("JobUnexpectedOperationError", (error) => Effect.die(error.cause)) + ); +} + +function pageResult( + records: readonly TRecord[], + limit: number, + map: (record: TRecord) => TValue +): { readonly hasNextPage: boolean; readonly page: TValue[] } { + return { + hasNextPage: records.length > limit, + page: records.slice(0, limit).map((record) => map(record)), + }; +} + +function readSchedule( + repository: JobRepository, + id: string +): ScheduleRecordWithRelations { + const relation = repository.findSchedule(id); + if (relation === undefined) { + throw new JobNotFoundError({ id, resource: "schedule" }); + } + return relation; +} + +function listRuns( + repository: JobRepository, + input: ListJobRunsInput, + minimumWorkerHeartbeatAt: Date +): ListJobRunsResult { + const snapshot = repository.listRunsWithQueueState({ + ...input, + minimumHeartbeatAt: minimumWorkerHeartbeatAt, + }); + const { hasNextPage, page } = pageResult(snapshot.runs, input.limit, toJobRunSummary); + const queue = snapshot.queue; + const last = page.at(-1); + return v.parse(listJobRunsResultSchema, { + ...(hasNextPage && last !== undefined + ? { nextCursor: { id: last.id, queuedAtMs: last.queuedAtMs } } + : {}), + runs: page, + summary: { + activeResourceClasses: [...queue.activeResourceClasses], + control: toJobWorkerControl(queue.control), + ...(queue.oldestQueuedAt === undefined + ? {} + : { oldestQueuedAtMs: getTime(queue.oldestQueuedAt) }), + stateCounts: queue.stateCounts, + workers: queue.workers.map(({ activeRunCount, worker }) => + toJobWorkerSummary(worker, activeRunCount) + ), + }, + }); +} + +function getRun(repository: JobRepository, input: GetJobRunInput): JobRunDetail { + const detail = repository.findRunDetail({ + ...(input.eventCursor === undefined + ? {} + : { beforeSequence: input.eventCursor.sequence }), + limit: input.eventLimit, + runId: input.id, + }); + if (detail === undefined) { + throw new JobNotFoundError({ id: input.id, resource: "job-run" }); + } + const { events: records, run } = detail; + const { hasNextPage, page } = pageResult(records, input.eventLimit, toJobRunEvent); + const last = page.at(-1); + return v.parse(jobRunDetailSchema, { + events: page, + ...(hasNextPage && last !== undefined + ? { nextEventCursor: { sequence: last.sequence } } + : {}), + ...(run.resultJson === null ? {} : { result: toJobRunResult(run) }), + run: toJobRunSummary(run), + }); +} + +function listSchedules( + repository: JobRepository, + input: ListSchedulesInput +): ListSchedulesResult { + const { hasNextPage, page } = pageResult( + repository.listSchedules(input), + input.limit, + ({ activeDisableIntent, activeRun, latestRun, schedule }) => + toScheduleSummary(schedule, { + ...(activeDisableIntent === undefined ? {} : { activeDisableIntent }), + ...(activeRun === undefined ? {} : { activeRun }), + ...(latestRun === undefined ? {} : { latestRun }), + }) + ); + const last = page.at(-1); + return v.parse(listSchedulesResultSchema, { + ...(hasNextPage && last !== undefined ? { nextCursor: { id: last.id } } : {}), + schedules: page, + }); +} + +function listScheduleRuns( + repository: JobRepository, + input: ListScheduleRunsInput +): ListScheduleRunsResult { + readSchedule(repository, input.id); + const { hasNextPage, page } = pageResult( + repository.listScheduleRuns(input), + input.limit, + toJobRunSummary + ); + const last = page.at(-1); + return v.parse(listScheduleRunsResultSchema, { + ...(hasNextPage && last !== undefined + ? { nextCursor: { id: last.id, queuedAtMs: last.queuedAtMs } } + : {}), + runs: page, + }); +} + +function operationTime(nowMs: () => number, durableDates: readonly Date[]): Date { + const now = toDate(v.parse(jobTimestampSchema, nowMs())); + return maximumDate([now, ...durableDates]); +} + +function minimumWorkerHeartbeatAt(nowMs: () => number): Date { + return subMilliseconds(operationTime(nowMs, []), jobWorkerFreshnessMs); +} + +function schedulesAreEqual( + left: ScheduleConfiguration, + right: ScheduleConfiguration +): boolean { + if (left.kind !== right.kind) return false; + if (left.kind === "interval" && right.kind === "interval") { + return left.intervalMs === right.intervalMs; + } + if (left.kind === "daily" && right.kind === "daily") { + return left.timeOfDay === right.timeOfDay && left.timeZone === right.timeZone; + } + if (left.kind === "cron" && right.kind === "cron") { + return left.expression === right.expression && left.timeZone === right.timeZone; + } + return false; +} + +function mutationSideEffects( + generateId: () => string, + input: Omit[0], "auditId"> +): JobMutationSideEffects { + return createJobMutationSideEffects({ ...input, auditId: generateId() }); +} + +type DurableRunMutationDescription = Omit< + Parameters[0], + "auditId" | "occurredAt" | "realtime" | "targetId" | "targetType" +>; + +function durableRunMutationSideEffects( + generateId: () => string, + run: JobRunRecord, + description: DurableRunMutationDescription +): JobMutationSideEffects { + const runSideEffects = mutationSideEffects(generateId, { + ...description, + occurredAt: run.updatedAt, + realtime: { + id: run.id, + kind: "run", + operation: "updated", + }, + targetId: run.id, + targetType: "job-run", + }); + if (run.scheduledJobId === null) return runSideEffects; + const scheduleSideEffects = createJobRealtimeSideEffects({ + occurredAt: run.updatedAt, + realtime: { + id: run.scheduledJobId, + kind: "schedule", + operation: "updated", + }, + }); + return Object.freeze({ + auditEvents: runSideEffects.auditEvents, + realtimeEvents: Object.freeze([ + ...runSideEffects.realtimeEvents, + ...scheduleSideEffects.realtimeEvents, + ]), + }); +} + +function scheduleInsertShape(registration: JobActionRegistration, at: Date) { + const schedule = buildRegisteredSchedule(registration, at); + if (schedule === undefined) { + throw new JobValidationError({ + id: registration.scheduleId, + reason: "next-occurrence-unavailable", + resource: "schedule", + }); + } + return schedule; +} + +/** + * Creates the domain service and reconciles the reviewed schedule directory. + * @param dependencies Process-owned repository, clock, IDs, and realtime wakeup. + * @returns Durable jobs service used by both jobs and schedules routers. + */ +export function createJobService( + dependencies: JobServiceDependencies +): JobService["Service"] { + const generateId = dependencies.generateId ?? (() => Bun.randomUUIDv7()); + const nowMs = dependencies.nowMs ?? Date.now; + + async function wake(): Promise { + if (dependencies.wakeEventPump === undefined) return; + try { + await dependencies.wakeEventPump(); + } catch { + // Durable outbox state is authoritative; a later pump cycle will observe it. + } + } + + const service: JobService["Service"] = { + cancelRun: (principal, input) => + mutationEffect(async () => { + const current = dependencies.repository.findRun(input.id); + if (current === undefined) { + throw new JobNotFoundError({ + id: input.id, + resource: "job-run", + }); + } + const at = operationTime(nowMs, [current.updatedAt]); + const result = await dependencies.repository.cancelRun({ + actor: principalActor(principal), + at, + id: input.id, + sideEffectsForRun: (run) => + durableRunMutationSideEffects(generateId, run, { + action: "jobs.run.cancel", + actor: principalAuditActor(principal), + outcome: "accepted", + }), + terminalCode: "cancelled/operator-request", + terminalMessage: "Cancelled by an operator", + }); + if (result.kind === "not-found") { + throw new JobNotFoundError({ id: input.id, resource: "job-run" }); + } + if (result.kind === "unsupported") { + throw new JobConflictError({ + id: input.id, + reason: "cancellation-not-supported", + resource: "job-run", + }); + } + if (result.kind === "cancelled" || result.kind === "requested") { + await wake(); + } + return toJobRunSummary(result.run); + }), + getRun: (input) => + readEffect( + () => getRun(dependencies.repository, input), + (error): error is JobNotFoundError => error instanceof JobNotFoundError + ), + getSchedule: (input) => + readEffect( + () => { + const relation = readSchedule(dependencies.repository, input.id); + return toScheduleSummary(relation.schedule, relation); + }, + (error): error is JobNotFoundError => error instanceof JobNotFoundError + ), + listRuns: (input) => + readEffect( + () => + listRuns( + dependencies.repository, + input, + minimumWorkerHeartbeatAt(nowMs) + ), + (_error): _error is never => false + ), + listScheduleRuns: (input) => + readEffect( + () => listScheduleRuns(dependencies.repository, input), + (error): error is JobNotFoundError => error instanceof JobNotFoundError + ), + listSchedules: (input) => + readEffect( + () => listSchedules(dependencies.repository, input), + (_error): _error is never => false + ), + runSchedule: (principal, input) => + mutationEffect(async () => { + const { schedule } = readSchedule(dependencies.repository, input.id); + const registration = findJobActionRegistration(schedule.actionKey); + if (!isRegisteredJobSchedule(schedule.id, schedule.actionKey)) { + throw new JobConflictError({ + id: input.id, + reason: "action-unavailable", + resource: "schedule", + }); + } + if (registration?.manualExposure !== "jobs-write") { + throw new JobConflictError({ + id: input.id, + reason: "action-not-manually-exposed", + resource: "schedule", + }); + } + const at = operationTime(nowMs, [schedule.updatedAt]); + const runId = generateId(); + const actor = principalActor(principal); + const runSideEffects = mutationSideEffects(generateId, { + action: "jobs.run.enqueue", + actor: principalAuditActor(principal), + occurredAt: at, + outcome: "accepted", + realtime: { + id: runId, + kind: "run", + operation: "created", + }, + targetId: runId, + targetType: "job-run", + }); + const scheduleSideEffects = createJobRealtimeSideEffects({ + occurredAt: at, + realtime: { + id: schedule.id, + kind: "schedule", + operation: "updated", + }, + }); + const result = await dependencies.repository.enqueueManualRun({ + auditEvents: runSideEffects.auditEvents, + queuedEvent: { + attempt: 0, + jobRunId: runId, + kind: "queued", + message: null, + occurredAt: at, + progressJson: null, + sequence: 1, + workerInstanceId: null, + }, + run: { + actionKey: schedule.actionKey, + attemptLimit: schedule.attemptLimit, + availableAt: at, + cancellationPolicy: schedule.cancellationPolicy, + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: schedule.name, + enqueueSha256: sha256Hex( + JSON.stringify({ + procedure: "schedules.run", + scheduleId: schedule.id, + triggerType: "manual", + version: 1, + }) + ), + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id: runId, + idempotencyKey: input.idempotencyKey, + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadJson: schedule.actionPayloadJson, + priority: schedule.priority, + queuedAt: at, + requestedById: actor.id, + requestedByKind: actor.kind, + resourceClass: schedule.resourceClass, + resourceKeysJson: schedule.resourceKeysJson, + resultJson: null, + retrySafe: schedule.retrySafe, + scheduledForAt: null, + scheduledJobId: schedule.id, + scheduledJobVersion: schedule.version, + state: "queued", + terminalCode: null, + terminalMessage: null, + timeoutMs: schedule.timeoutMs, + triggerType: "manual", + updatedAt: at, + }, + realtimeEvents: Object.freeze([ + ...runSideEffects.realtimeEvents, + ...scheduleSideEffects.realtimeEvents, + ]), + }); + if (result.kind === "idempotency-mismatch") { + throw new JobConflictError({ + id: input.id, + reason: "idempotency-mismatch", + resource: "schedule", + }); + } + if (result.kind === "action-unavailable") { + throw new JobConflictError({ + id: input.id, + reason: "action-unavailable", + resource: "schedule", + }); + } + if (result.kind === "active") { + throw new JobConflictError({ + id: input.id, + reason: "run-already-active", + resource: "schedule", + }); + } + if (result.kind === "inserted") await wake(); + return toJobRunSummary(result.run); + }), + setClaimingPaused: (principal, input) => + mutationEffect(async () => { + const current = dependencies.repository.readWorkerControl(); + const at = operationTime(nowMs, [current.updatedAt]); + const result = await dependencies.repository.setClaimingPaused({ + actor: principalActor(principal), + at, + expectedVersion: input.expectedVersion, + paused: input.paused, + ...mutationSideEffects(generateId, { + action: "jobs.claim.pause", + actor: principalAuditActor(principal), + occurredAt: at, + outcome: "accepted", + realtime: { id: "worker-control", kind: "queue" }, + targetId: "worker-control", + targetType: "job-worker", + }), + }); + if (result.kind === "version-changed") { + throw new JobConflictError({ + id: "worker-control", + reason: "version-changed", + resource: "worker-control", + }); + } + await wake(); + return toJobWorkerControl(result.control); + }), + updateSchedule: (principal, input) => + mutationEffect(async () => { + const current = readSchedule(dependencies.repository, input.id); + if ( + !isRegisteredJobSchedule( + current.schedule.id, + current.schedule.actionKey + ) + ) { + throw new JobConflictError({ + id: input.id, + reason: "action-unavailable", + resource: "schedule", + }); + } + const at = operationTime(nowMs, [current.schedule.updatedAt]); + const currentConfiguration = toScheduleSummary( + current.schedule, + current + ).schedule; + const changesSchedule = + input.patch.schedule !== undefined && + !schedulesAreEqual(input.patch.schedule, currentConfiguration); + const editsEnabledScheduleCadence = + current.schedule.enabled && + input.patch.enabled === true && + changesSchedule; + const replacesActiveDisableIntent = + current.activeDisableIntent !== undefined && + current.schedule.enabled === false && + input.patch.enabled === false && + input.patch.disableIntent !== undefined && + input.patch.disableIntent !== null; + if ( + input.patch.enabled !== undefined && + input.patch.enabled === current.schedule.enabled && + !replacesActiveDisableIntent && + !editsEnabledScheduleCadence + ) { + throw new JobValidationError({ + id: input.id, + reason: "enabled-state-unchanged", + resource: "schedule", + }); + } + if ( + input.patch.enabled === undefined && + input.patch.schedule !== undefined && + !changesSchedule + ) { + throw new JobValidationError({ + id: input.id, + reason: "schedule-unchanged", + resource: "schedule", + }); + } + if ( + input.patch.disableIntent?.expiresAtMs !== undefined && + input.patch.disableIntent.expiresAtMs <= getTime(at) + ) { + throw new JobValidationError({ + id: input.id, + reason: "disable-intent-expired", + resource: "schedule", + }); + } + const targetEnabled = input.patch.enabled ?? current.schedule.enabled; + const targetConfiguration = + input.patch.schedule === undefined ? undefined : input.patch.schedule; + const shouldCalculateNextRun = + targetEnabled || targetConfiguration !== undefined; + const cadenceAnchorMs = + targetConfiguration === undefined + ? getTime( + current.schedule.nextRunAt ?? current.schedule.createdAt + ) + : getTime(at); + const existingNextRunAtMs = + current.schedule.nextRunAt === null + ? undefined + : getTime(current.schedule.nextRunAt); + const nextRunAtMs = shouldCalculateNextRun + ? nextScheduleOccurrence( + targetConfiguration ?? currentConfiguration, + getTime(at), + cadenceAnchorMs + ) + : existingNextRunAtMs; + if (shouldCalculateNextRun && nextRunAtMs === undefined) { + throw new JobValidationError({ + id: input.id, + reason: "next-occurrence-unavailable", + resource: "schedule", + }); + } + const actor = principalActor(principal); + const result = await dependencies.repository.updateSchedule({ + at, + ...(current.activeDisableIntent === undefined || + input.patch.enabled === undefined + ? {} + : { + closeActiveIntent: { + endedAt: at, + endedById: actor.id, + endedByKind: actor.kind, + endedReason: + input.patch.enabled === true + ? "re-enabled" + : "replaced", + }, + }), + expectedActiveDisableIntentId: + current.activeDisableIntent?.id ?? null, + expectedVersion: input.expectedVersion, + id: input.id, + ...(input.patch.disableIntent === undefined || + input.patch.disableIntent === null + ? {} + : { + insertDisableIntent: { + createdAt: at, + createdById: actor.id, + createdByKind: actor.kind, + endedAt: null, + endedById: null, + endedByKind: null, + endedReason: null, + expiresAt: + input.patch.disableIntent.expiresAtMs === undefined + ? null + : toDate(input.patch.disableIntent.expiresAtMs), + externalJobId: null, + externalProvider: null, + id: generateId(), + reason: input.patch.disableIntent.reason, + scheduledJobId: input.id, + targetKind: "dashboard-schedule", + }, + }), + patch: { + ...(input.patch.enabled === undefined + ? {} + : { enabled: input.patch.enabled }), + nextRunAt: nextRunAtMs === undefined ? null : toDate(nextRunAtMs), + ...(targetConfiguration === undefined + ? {} + : { schedule: targetConfiguration }), + }, + ...(input.patch.enabled === false + ? { + queuedCancellation: { + at, + terminalCode: "cancelled/schedule-disabled", + terminalMessage: + "Cancelled because the schedule was disabled", + }, + queuedCancellationSideEffects: (run) => + durableRunMutationSideEffects(generateId, run, { + action: "jobs.run.cancel", + actor: principalAuditActor(principal), + outcome: "cancelled", + }), + } + : {}), + ...mutationSideEffects(generateId, { + action: "jobs.schedule.update", + actor: principalAuditActor(principal), + occurredAt: at, + outcome: "accepted", + realtime: { + id: input.id, + kind: "schedule", + operation: "updated", + }, + targetId: input.id, + targetType: "schedule", + }), + }); + if (result.kind === "not-found") { + throw new JobNotFoundError({ id: input.id, resource: "schedule" }); + } + if (result.kind === "cancellation-not-supported") { + throw new JobConflictError({ + id: result.run.id, + reason: "cancellation-not-supported", + resource: "job-run", + }); + } + if (result.kind === "version-changed") { + throw new JobConflictError({ + id: input.id, + reason: "version-changed", + resource: "schedule", + }); + } + await wake(); + const updated = readSchedule(dependencies.repository, input.id); + return toScheduleSummary(updated.schedule, updated); + }), + }; + + return service; +} + +/** + * Reconciles the reviewed action directory before web or worker traffic is accepted. + * @param dependencies Process-owned repository, clock, IDs, and realtime wakeup. + * @returns Promise that resolves only after the durable reconciliation commits. + */ +export async function reconcileJobSchedules( + dependencies: JobScheduleReconciliationDependencies +): Promise { + const generateId = dependencies.generateId ?? (() => Bun.randomUUIDv7()); + const nowMs = dependencies.nowMs ?? Date.now; + const at = toDate(v.parse(jobTimestampSchema, nowMs())); + await dependencies.repository.reconcileSchedules({ + at, + retiredRunCancellation: { + actor: systemActor, + sideEffectsForRun: (run) => + durableRunMutationSideEffects(generateId, run, { + action: "jobs.run.cancel", + actor: systemActor, + outcome: "cancelled", + }), + terminalCode: "cancelled/schedule-retired", + terminalMessage: + "Cancelled because the schedule was retired from the action registry", + }, + schedules: jobActionRegistrations.map((registration) => + scheduleInsertShape(registration, at) + ), + sideEffectsForSchedule: (schedule) => + mutationSideEffects(generateId, { + action: "jobs.schedule.reconcile", + actor: systemActor, + occurredAt: schedule.updatedAt, + outcome: "succeeded", + realtime: { + id: schedule.id, + kind: "schedule", + operation: "updated", + }, + targetId: schedule.id, + targetType: "schedule", + }), + }); + if (dependencies.wakeEventPump !== undefined) { + try { + await dependencies.wakeEventPump(); + } catch { + // Reconciliation and its durable outbox rows already committed. + } + } +} diff --git a/greenfield/src/server/domains/jobs/sideEffects.test.ts b/greenfield/src/server/domains/jobs/sideEffects.test.ts new file mode 100644 index 000000000..b403b1b1e --- /dev/null +++ b/greenfield/src/server/domains/jobs/sideEffects.test.ts @@ -0,0 +1,100 @@ +import { describe, expect, test } from "bun:test"; + +import { + createJobMutationSideEffects, + createJobRealtimeSideEffects, +} from "./sideEffects.ts"; + +const auditId = "018f6f50-6a9e-7b88-8000-000000000001"; +const runId = "018f6f50-6a9e-7b88-8000-000000000002"; + +describe("jobs mutation side effects", () => { + test("builds one redacted audit row and compact run invalidation", () => { + const occurredAt = new Date(1000); + const sideEffects = createJobMutationSideEffects({ + action: "jobs.run.enqueue", + actor: { + authenticatorId: "a".repeat(32), + id: "018f6f50-6a9e-7b88-8000-000000000003", + kind: "user", + }, + auditId, + occurredAt, + outcome: "accepted", + realtime: { id: runId, kind: "run", operation: "created" }, + requestId: "request-1", + targetId: runId, + targetType: "job-run", + }); + + expect(sideEffects.auditEvents).toEqual([ + { + action: "jobs.run.enqueue", + actorId: "018f6f50-6a9e-7b88-8000-000000000003", + actorKind: "user", + authenticatorId: "a".repeat(32), + id: auditId, + metadataJson: "{}", + occurredAt, + outcome: "accepted", + requestId: "request-1", + targetId: runId, + targetType: "job-run", + }, + ]); + expect(sideEffects.realtimeEvents).toEqual([ + { + entityId: runId, + entityType: "job-run", + expiresAt: new Date(604_801_000), + occurredAt, + operation: "created", + payloadJson: JSON.stringify({ id: runId }), + topic: "jobs.runs", + }, + ]); + expect(Object.isFrozen(sideEffects)).toBeTrue(); + }); + + test("builds a queue snapshot without a phantom run identity", () => { + const sideEffects = createJobMutationSideEffects({ + action: "jobs.claim.pause", + actor: { authenticatorId: null, id: "jobs-worker", kind: "system" }, + auditId, + occurredAt: new Date(1000), + outcome: "succeeded", + realtime: { id: "worker-control", kind: "queue" }, + targetId: "worker-control", + targetType: "job-worker", + }); + expect(sideEffects.realtimeEvents[0]).toMatchObject({ + entityId: "worker-control", + entityType: "job-queue", + operation: "snapshot-required", + topic: "jobs.runs", + }); + }); + + test("builds realtime-only run invalidation for durable timeline events", () => { + const occurredAt = new Date(2000); + + const sideEffects = createJobRealtimeSideEffects({ + occurredAt, + realtime: { id: runId, kind: "run", operation: "updated" }, + }); + + expect(sideEffects.auditEvents).toEqual([]); + expect(sideEffects.realtimeEvents).toEqual([ + { + entityId: runId, + entityType: "job-run", + expiresAt: new Date(604_802_000), + occurredAt, + operation: "updated", + payloadJson: JSON.stringify({ id: runId }), + topic: "jobs.runs", + }, + ]); + expect(Object.isFrozen(sideEffects.realtimeEvents)).toBeTrue(); + }); +}); diff --git a/greenfield/src/server/domains/jobs/sideEffects.ts b/greenfield/src/server/domains/jobs/sideEffects.ts new file mode 100644 index 000000000..94a09f6c5 --- /dev/null +++ b/greenfield/src/server/domains/jobs/sideEffects.ts @@ -0,0 +1,154 @@ +import { addMilliseconds } from "date-fns"; +import * as v from "valibot"; + +import { + jobChangePayloadSchema, + jobRealtimeRoutingSchema, + jobRealtimeTopics, +} from "../../../contracts/jobRealtime.ts"; +import type { JsonObject } from "../../../shared/json.ts"; +import { auditEventInsertSchema } from "../../database/validation/auditEvents.ts"; +import { realtimeEventInsertSchema } from "../../database/validation/realtimeEvents.ts"; +import { defaultRealtimeRetentionMilliseconds } from "../realtime/retention.ts"; +import type { JobMutationSideEffects } from "./repository.ts"; + +export type JobAuditEventInsert = v.InferOutput; +export type JobRealtimeEventInsert = v.InferOutput; + +/** Redacted durable actor identity used for jobs-domain audit rows. */ +export interface JobAuditActor { + readonly authenticatorId: string | null; + readonly id: string; + readonly kind: "automation" | "system" | "user"; +} + +export type JobRealtimeTarget = + | { readonly id: string; readonly kind: "queue" } + | { + readonly id: string; + readonly kind: "run"; + readonly operation: "created" | "updated"; + } + | { + readonly id: string; + readonly kind: "schedule"; + readonly operation: "created" | "updated"; + }; + +export interface CreateJobMutationSideEffectsInput { + readonly action: string; + readonly actor: JobAuditActor; + readonly auditId: string; + readonly metadata?: JsonObject; + readonly occurredAt: Date; + readonly outcome: "accepted" | "cancelled" | "failed" | "succeeded"; + readonly realtime?: JobRealtimeTarget; + readonly realtimeRetentionMs?: number; + readonly requestId?: string; + readonly targetId: string; + readonly targetType: "job-run" | "job-worker" | "schedule"; +} + +/** Realtime-only invalidation input for durable mutations that are their own history. */ +export interface CreateJobRealtimeSideEffectsInput { + readonly occurredAt: Date; + readonly realtime: JobRealtimeTarget; + readonly realtimeRetentionMs?: number; +} + +function createJobRealtimeEvent( + target: JobRealtimeTarget, + occurredAt: Date, + retentionMs: number +): JobRealtimeEventInsert { + const routing = (() => { + if (target.kind === "run") { + return { + entityType: "job-run" as const, + operation: target.operation, + topic: jobRealtimeTopics.runs, + }; + } + if (target.kind === "schedule") { + return { + entityType: "schedule" as const, + operation: target.operation, + topic: jobRealtimeTopics.schedules, + }; + } + return { + entityType: "job-queue" as const, + operation: "snapshot-required" as const, + topic: jobRealtimeTopics.runs, + }; + })(); + v.parse(jobRealtimeRoutingSchema, routing); + const payload = v.parse(jobChangePayloadSchema, { id: target.id }); + return v.parse(realtimeEventInsertSchema, { + entityId: target.id, + entityType: routing.entityType, + expiresAt: addMilliseconds(occurredAt, retentionMs), + occurredAt, + operation: routing.operation, + payloadJson: JSON.stringify(payload), + topic: routing.topic, + }); +} + +/** + * Builds a compact realtime invalidation without duplicating a durable domain event in + * the security audit log. + * @param input Committed transition time and affected jobs-domain target. + * @returns Frozen realtime-only side effects for the surrounding transaction. + */ +export function createJobRealtimeSideEffects( + input: CreateJobRealtimeSideEffectsInput +): JobMutationSideEffects { + return Object.freeze({ + auditEvents: Object.freeze([]), + realtimeEvents: Object.freeze([ + createJobRealtimeEvent( + input.realtime, + input.occurredAt, + input.realtimeRetentionMs ?? defaultRealtimeRetentionMilliseconds + ), + ]), + }); +} + +/** + * Builds validated rows that a repository appends in the same transaction as state. + * @param input Redacted audit and compact invalidation description. + * @returns Immutable side-effect rows for one atomic mutation. + */ +export function createJobMutationSideEffects( + input: CreateJobMutationSideEffectsInput +): JobMutationSideEffects { + const auditEvent = v.parse(auditEventInsertSchema, { + action: input.action, + actorId: input.actor.id, + actorKind: input.actor.kind, + authenticatorId: input.actor.authenticatorId, + id: input.auditId, + metadataJson: JSON.stringify(input.metadata ?? {}), + occurredAt: input.occurredAt, + outcome: input.outcome, + requestId: input.requestId ?? null, + targetId: input.targetId, + targetType: input.targetType, + }); + const realtimeEvents = + input.realtime === undefined + ? [] + : createJobRealtimeSideEffects({ + occurredAt: input.occurredAt, + realtime: input.realtime, + ...(input.realtimeRetentionMs === undefined + ? {} + : { realtimeRetentionMs: input.realtimeRetentionMs }), + }).realtimeEvents; + return Object.freeze({ + auditEvents: Object.freeze([auditEvent]), + realtimeEvents: Object.freeze(realtimeEvents), + }); +} diff --git a/greenfield/src/server/domains/jobs/testSupport/service.ts b/greenfield/src/server/domains/jobs/testSupport/service.ts new file mode 100644 index 000000000..a32d148f3 --- /dev/null +++ b/greenfield/src/server/domains/jobs/testSupport/service.ts @@ -0,0 +1,30 @@ +import { Effect } from "effect"; + +import { JobService } from "../service.ts"; + +function unexpectedJobServiceCall(method: string): () => Effect.Effect { + return () => + Effect.die(new Error(`Test job service received an unexpected call: ${method}`)); +} + +/** + * Creates a fail-closed durable jobs service for unrelated router/server tests. + * @param overrides Exact methods exercised by the current test. + * @returns Complete jobs-domain test double. + */ +export function createTestJobService( + overrides: Partial = {} +): JobService["Service"] { + return JobService.of({ + cancelRun: unexpectedJobServiceCall("cancelRun"), + getRun: unexpectedJobServiceCall("getRun"), + getSchedule: unexpectedJobServiceCall("getSchedule"), + listRuns: unexpectedJobServiceCall("listRuns"), + listScheduleRuns: unexpectedJobServiceCall("listScheduleRuns"), + listSchedules: unexpectedJobServiceCall("listSchedules"), + runSchedule: unexpectedJobServiceCall("runSchedule"), + setClaimingPaused: unexpectedJobServiceCall("setClaimingPaused"), + updateSchedule: unexpectedJobServiceCall("updateSchedule"), + ...overrides, + }); +} diff --git a/greenfield/src/server/domains/jobs/workerRuntime.test.ts b/greenfield/src/server/domains/jobs/workerRuntime.test.ts new file mode 100644 index 000000000..de70f2ddf --- /dev/null +++ b/greenfield/src/server/domains/jobs/workerRuntime.test.ts @@ -0,0 +1,229 @@ +import { describe, expect, test } from "bun:test"; + +import type { SQLiteBunDatabase } from "drizzle-orm/bun-sqlite"; + +import type { ImmediateDatabaseWriteAdmission } from "../../database/immediateWriteAdmission.ts"; +import type { JobWorkerCoordinator } from "./coordinator.ts"; +import type { JobRepository } from "./repository.ts"; +import { + createDashboardWorkerRuntime, + createSystemJobWorkerSideEffects, + type DashboardWorkerRuntimeDependencies, + type DashboardWorkerRuntimeOptions, +} from "./workerRuntime.ts"; + +const noSideEffects = Object.freeze({ + auditEvents: Object.freeze([]), + realtimeEvents: Object.freeze([]), +}); + +const runtimeOptions: DashboardWorkerRuntimeOptions = { + database: { + migrationsDirectory: "/srv/mira-dashboard/releases/test/migrations", + releaseId: "a".repeat(40), + startupMode: "validate-only", + stateDirectory: "/srv/mira-dashboard/state", + }, + pid: 123, + releaseId: "a".repeat(40), + sideEffects: { + forQueue: () => noSideEffects, + forRun: () => noSideEffects, + forRunEvent: () => noSideEffects, + forSchedule: () => noSideEffects, + forScheduleEvent: () => noSideEffects, + }, + workerInstanceId: Bun.randomUUIDv7(), +}; + +function deferred() { + let resolveDeferred: ((value: T | PromiseLike) => void) | undefined; + let rejectDeferred: ((reason?: unknown) => void) | undefined; + const promise = new Promise((resolve, reject) => { + resolveDeferred = resolve; + rejectDeferred = reject; + }); + return { + promise, + reject(error: unknown) { + rejectDeferred?.(error); + }, + resolve(value: T) { + resolveDeferred?.(value); + }, + }; +} + +function runtimeFixture(initializationFailure?: Error) { + const events: string[] = []; + const coordinatorCompletion = deferred(); + const forceSignals: Array = []; + const coordinator: JobWorkerCoordinator = Object.freeze({ + completion: coordinatorCompletion.promise, + dispose(forceSignal?: AbortSignal) { + events.push("coordinator-dispose"); + forceSignals.push(forceSignal); + coordinatorCompletion.resolve(); + return Promise.resolve(); + }, + initialize() { + events.push("coordinator-initialize"); + return initializationFailure === undefined + ? Promise.resolve() + : Promise.reject(initializationFailure); + }, + }); + const repository = Object.freeze({}) as JobRepository; + const dependencies = { + createCoordinator(options) { + events.push("coordinator-create"); + expect(options.repository).toBe(repository); + return coordinator; + }, + createDatabaseRuntime() { + return Object.freeze({ + dispose() { + events.push("database-dispose"); + return Promise.resolve(); + }, + initialize() { + events.push("database-initialize"); + return Promise.resolve({ + database: Object.freeze({}) as SQLiteBunDatabase, + writeAdmission: Object.freeze({ + run() { + return Promise.reject( + new Error("Write admission is unused") + ); + }, + }) satisfies ImmediateDatabaseWriteAdmission, + }); + }, + }); + }, + createRepository() { + events.push("repository-create"); + return repository; + }, + } satisfies DashboardWorkerRuntimeDependencies; + return { + coordinatorCompletion, + dependencies, + events, + forceSignals, + }; +} + +describe("Dashboard worker runtime", () => { + test("emits created only for explicit run-creation actions", () => { + const sideEffects = createSystemJobWorkerSideEffects(); + const at = new Date(1000); + const eventRunId = Bun.randomUUIDv7(); + + for (const action of ["jobs.run.enqueue", "jobs.run.enqueue-scheduled"]) { + expect( + sideEffects.forRun({ + action, + at, + outcome: "accepted", + targetId: Bun.randomUUIDv7(), + }).realtimeEvents[0]?.operation + ).toBe("created"); + } + expect( + sideEffects.forRun({ + action: "jobs.run.enqueue-retry-observed", + at, + outcome: "accepted", + targetId: Bun.randomUUIDv7(), + }).realtimeEvents[0]?.operation + ).toBe("updated"); + expect( + sideEffects.forRunEvent({ + action: "jobs.run.event", + at, + outcome: "accepted", + targetId: eventRunId, + }) + ).toEqual({ + auditEvents: [], + realtimeEvents: [ + expect.objectContaining({ + entityId: eventRunId, + operation: "updated", + topic: "jobs.runs", + }), + ], + }); + const scheduleId = "system.worker-smoke"; + expect( + sideEffects.forScheduleEvent({ + action: "jobs.run.succeeded", + at, + outcome: "succeeded", + targetId: scheduleId, + }) + ).toEqual({ + auditEvents: [], + realtimeEvents: [ + expect.objectContaining({ + entityId: scheduleId, + operation: "updated", + topic: "schedules.records", + }), + ], + }); + }); + + test("owns database then coordinator and disposes them in reverse order", async () => { + const fixture = runtimeFixture(); + const runtime = createDashboardWorkerRuntime( + runtimeOptions, + fixture.dependencies + ); + const force = new AbortController(); + + await runtime.initialize(); + await runtime.dispose(force.signal); + + expect(fixture.events).toEqual([ + "database-initialize", + "repository-create", + "coordinator-create", + "coordinator-initialize", + "coordinator-dispose", + "database-dispose", + ]); + expect(fixture.forceSignals).toEqual([force.signal]); + expect(await runtime.completion).toBeUndefined(); + }); + + test("exposes an unexpected coordinator failure through completion", async () => { + const fixture = runtimeFixture(); + const runtime = createDashboardWorkerRuntime( + runtimeOptions, + fixture.dependencies + ); + const failure = new Error("coordinator failed"); + + await runtime.initialize(); + fixture.coordinatorCompletion.reject(failure); + + expect(await runtime.completion.catch((error: unknown) => error)).toBe(failure); + await runtime.dispose(); + }); + + test("preserves initialization failure and still releases the database", async () => { + const failure = new Error("coordinator initialization failed"); + const fixture = runtimeFixture(failure); + const runtime = createDashboardWorkerRuntime( + runtimeOptions, + fixture.dependencies + ); + const completion = runtime.completion.catch((error: unknown) => error); + + expect(await runtime.initialize().catch((error: unknown) => error)).toBe(failure); + expect(await completion).toBe(failure); + expect(fixture.events).toContain("database-dispose"); + }, 1000); +}); diff --git a/greenfield/src/server/domains/jobs/workerRuntime.ts b/greenfield/src/server/domains/jobs/workerRuntime.ts new file mode 100644 index 000000000..057fbefe9 --- /dev/null +++ b/greenfield/src/server/domains/jobs/workerRuntime.ts @@ -0,0 +1,271 @@ +import type { SQLiteBunDatabase } from "drizzle-orm/bun-sqlite"; +import { ManagedRuntime } from "effect"; + +import type { DashboardWorkerRuntime } from "../../../shared/workerRuntime.ts"; +import type { ImmediateDatabaseWriteAdmission } from "../../database/immediateWriteAdmission.ts"; +import { + databaseRuntimeLayer, + DatabaseRuntimeService, + type DatabaseRuntimeLayerOptions, +} from "../../database/runtime/databaseService.ts"; +import { + createJobWorkerCoordinator, + type JobWorkerCoordinator, + type JobWorkerSideEffectFactory, + type JobWorkerSideEffectInput, +} from "./coordinator.ts"; +import { createJobRepository, type JobRepository } from "./repository.ts"; +import { + createJobMutationSideEffects, + createJobRealtimeSideEffects, +} from "./sideEffects.ts"; + +export interface DashboardWorkerRuntimeOptions { + readonly database: DatabaseRuntimeLayerOptions; + readonly pid: number; + readonly releaseId: string; + readonly sideEffects: JobWorkerSideEffectFactory; + readonly workerInstanceId: string; +} + +interface WorkerDatabaseRuntimeContext { + readonly database: SQLiteBunDatabase; + readonly writeAdmission: ImmediateDatabaseWriteAdmission; +} + +interface WorkerDatabaseRuntime { + dispose(): Promise; + initialize(): Promise; +} + +export interface DashboardWorkerRuntimeDependencies { + readonly createCoordinator: typeof createJobWorkerCoordinator; + readonly createDatabaseRuntime: ( + options: DatabaseRuntimeLayerOptions + ) => WorkerDatabaseRuntime; + readonly createRepository: ( + database: SQLiteBunDatabase, + writeAdmission: ImmediateDatabaseWriteAdmission + ) => JobRepository; +} + +function createWorkerDatabaseRuntime( + options: DatabaseRuntimeLayerOptions +): WorkerDatabaseRuntime { + const runtime = ManagedRuntime.make(databaseRuntimeLayer(options)); + let initialization: Promise | undefined; + let disposal: Promise | undefined; + const initialize = async (): Promise => { + const service = await runtime.runPromise(DatabaseRuntimeService); + const writeAdmission: ImmediateDatabaseWriteAdmission = Object.freeze({ + run(operation: (markTransactionStarted: () => void) => T): Promise { + return runtime.runPromise(service.runImmediateWrite(operation)); + }, + }); + return Object.freeze({ database: service.orm, writeAdmission }); + }; + return Object.freeze({ + dispose() { + disposal ??= runtime.dispose(); + return disposal; + }, + initialize() { + if (disposal !== undefined) { + return Promise.reject(new Error("Worker database runtime is disposed")); + } + initialization ??= initialize(); + return initialization; + }, + }); +} + +const defaultDependencies: DashboardWorkerRuntimeDependencies = Object.freeze({ + createCoordinator: createJobWorkerCoordinator, + createDatabaseRuntime: createWorkerDatabaseRuntime, + createRepository: createJobRepository, +}); + +function normalizeWorkerRuntimeFailure(error: unknown): Error { + return error instanceof Error + ? error + : new Error("Dashboard worker runtime failed", { cause: error }); +} + +async function preservePrimaryFailure( + primary: unknown, + cleanup: () => Promise +): Promise { + const failure = normalizeWorkerRuntimeFailure(primary); + try { + await cleanup(); + } catch { + // The initiating defect is the actionable process failure. + } + throw failure; +} + +const runCreatingActions: ReadonlySet = new Set([ + "jobs.run.enqueue", + "jobs.run.enqueue-scheduled", +]); + +/** + * Builds required system audit and realtime rows without granting action authority. + * @param generateId UUIDv7 generator for append-only audit identities. + * @returns Worker-scoped side-effect factory for queue, run, and schedule transitions. + */ +export function createSystemJobWorkerSideEffects( + generateId: () => string = () => Bun.randomUUIDv7() +): JobWorkerSideEffectFactory { + const actor = Object.freeze({ + authenticatorId: null, + id: "system.jobs-worker", + kind: "system" as const, + }); + return Object.freeze({ + forQueue(input: JobWorkerSideEffectInput) { + return createJobMutationSideEffects({ + action: input.action, + actor, + auditId: generateId(), + occurredAt: input.at, + outcome: input.outcome, + realtime: { id: "jobs.queue", kind: "queue" }, + targetId: input.targetId, + targetType: "job-worker", + }); + }, + forRun(input: JobWorkerSideEffectInput) { + return createJobMutationSideEffects({ + action: input.action, + actor, + auditId: generateId(), + occurredAt: input.at, + outcome: input.outcome, + realtime: { + id: input.targetId, + kind: "run", + operation: runCreatingActions.has(input.action) + ? "created" + : "updated", + }, + targetId: input.targetId, + targetType: "job-run", + }); + }, + forRunEvent(input: JobWorkerSideEffectInput) { + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "run", + operation: "updated", + }, + }); + }, + forSchedule(input: JobWorkerSideEffectInput) { + return createJobMutationSideEffects({ + action: input.action, + actor, + auditId: generateId(), + occurredAt: input.at, + outcome: input.outcome, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + targetId: input.targetId, + targetType: "schedule", + }); + }, + forScheduleEvent(input: JobWorkerSideEffectInput) { + return createJobRealtimeSideEffects({ + occurredAt: input.at, + realtime: { + id: input.targetId, + kind: "schedule", + operation: "updated", + }, + }); + }, + }); +} + +/** + * Creates the worker's ordered database and durable-coordinator ownership scope. + * @param options Exact release/database identity and required atomic side effects. + * @param dependencies Injectable construction boundaries for focused lifecycle tests. + * @returns One idempotent runtime whose completion rejects on loop failure. + */ +export function createDashboardWorkerRuntime( + options: DashboardWorkerRuntimeOptions, + dependencies: DashboardWorkerRuntimeDependencies = defaultDependencies +): DashboardWorkerRuntime { + const databaseRuntime = dependencies.createDatabaseRuntime(options.database); + let coordinator: JobWorkerCoordinator | undefined; + let initializePromise: Promise | undefined; + let disposePromise: Promise | undefined; + let resolveCompletion: (() => void) | undefined; + let rejectCompletion: ((error: unknown) => void) | undefined; + const completion = new Promise((resolve, reject) => { + resolveCompletion = resolve; + rejectCompletion = reject; + }); + + const initialize = async (): Promise => { + try { + const database = await databaseRuntime.initialize(); + const repository = dependencies.createRepository( + database.database, + database.writeAdmission + ); + coordinator = dependencies.createCoordinator({ + databaseReleaseId: options.releaseId, + pid: options.pid, + repository, + sideEffects: options.sideEffects, + workerInstanceId: options.workerInstanceId, + }); + void coordinator.completion.then(resolveCompletion, rejectCompletion); + await coordinator.initialize(); + } catch (error) { + rejectCompletion?.(error); + return preservePrimaryFailure(error, () => databaseRuntime.dispose()); + } + }; + + const dispose = async (forceSignal?: AbortSignal): Promise => { + let failure: unknown; + if (initializePromise !== undefined) { + try { + await initializePromise; + await coordinator?.dispose(forceSignal); + } catch (error) { + failure = error; + } + } + try { + await databaseRuntime.dispose(); + } catch (error) { + failure ??= error; + } + if (failure !== undefined) throw normalizeWorkerRuntimeFailure(failure); + resolveCompletion?.(); + }; + + return Object.freeze({ + completion, + dispose(forceSignal?: AbortSignal) { + disposePromise ??= dispose(forceSignal); + return disposePromise; + }, + initialize() { + if (disposePromise !== undefined) { + return Promise.reject(new Error("Dashboard worker runtime is disposed")); + } + initializePromise ??= initialize(); + return initializePromise; + }, + }); +} diff --git a/greenfield/src/server/domains/jobs/workerSystem.test.ts b/greenfield/src/server/domains/jobs/workerSystem.test.ts new file mode 100644 index 000000000..2a0e30dfa --- /dev/null +++ b/greenfield/src/server/domains/jobs/workerSystem.test.ts @@ -0,0 +1,147 @@ +import { describe, expect, test } from "bun:test"; + +import { testImmediateDatabaseWriteAdmission } from "../../test/support/databaseWriteAdmission.ts"; +import { openFreshMigratedDatabase } from "../../test/support/freshDatabase.ts"; +import { createJobWorkerCoordinator } from "./coordinator.ts"; +import { + createJobRepository, + type JobMutationSideEffects, + type JobRunEventInsert, + type JobRunInsert, +} from "./repository.ts"; +import { createSystemJobWorkerSideEffects } from "./workerRuntime.ts"; + +const noSideEffects: JobMutationSideEffects = Object.freeze({ + auditEvents: Object.freeze([]), + realtimeEvents: Object.freeze([]), +}); +const terminalRunStates = new Set(["cancelled", "failed", "succeeded", "timed-out"]); + +async function waitForTerminal( + readState: () => string | undefined +): Promise { + const deadline = Date.now() + 2000; + let state = readState(); + while (state === undefined || !terminalRunStates.has(state)) { + if (Date.now() >= deadline) return state; + await Bun.sleep(2); + state = readState(); + } + return state; +} + +describe("durable job worker system", () => { + test("stops polling when any terminal state is observed", async () => { + for (const terminalState of terminalRunStates) { + let reads = 0; + expect( + await waitForTerminal(() => { + reads += 1; + return terminalState; + }) + ).toBe(terminalState); + expect(reads).toBe(1); + } + }); + + test("claims and settles a repository-enqueued smoke run", async () => { + const database = await openFreshMigratedDatabase(); + const repository = createJobRepository( + database.orm, + testImmediateDatabaseWriteAdmission + ); + const nowMs = Date.now(); + const workerId = Bun.randomUUIDv7(); + const runId = Bun.randomUUIDv7(); + const coordinator = createJobWorkerCoordinator({ + databaseReleaseId: "a".repeat(40), + generateId: () => Bun.randomUUIDv7(), + nowMs: () => nowMs, + pid: 1234, + repository, + sideEffects: createSystemJobWorkerSideEffects(), + timings: { + cancellationPollMs: 2, + claimLeaseMs: 100, + claimRenewalMs: 20, + heartbeatMs: 20, + idlePollMs: 2, + schedulePollMs: 20, + workerFreshnessMs: 50, + }, + workerInstanceId: workerId, + }); + try { + await coordinator.initialize(); + const schedule = repository.findSchedule("system.worker-smoke"); + if (schedule === undefined) + throw new Error("Smoke schedule was not reconciled"); + const at = new Date(nowMs); + const run: JobRunInsert = { + actionKey: "system.worker-smoke", + attemptLimit: 3, + availableAt: at, + cancellationPolicy: "cooperative", + cancelRequestedAt: null, + cancelRequestedById: null, + cancelRequestedByKind: null, + displayName: "Worker smoke", + enqueueSha256: "b".repeat(64), + finishedAt: null, + firstStartedAt: null, + heartbeatAt: null, + id: runId, + idempotencyKey: "c".repeat(64), + lastAttemptStartedAt: null, + leaseExpiresAt: null, + leaseOwnerId: null, + leaseToken: null, + payloadJson: "{}", + priority: 0, + queuedAt: at, + requestedById: Bun.randomUUIDv7(), + requestedByKind: "user", + resourceClass: "light", + resourceKeysJson: '["database"]', + resultJson: null, + retrySafe: true, + scheduledForAt: null, + scheduledJobId: schedule.schedule.id, + scheduledJobVersion: schedule.schedule.version, + state: "queued", + terminalCode: null, + terminalMessage: null, + timeoutMs: 30_000, + triggerType: "manual", + updatedAt: at, + }; + const queuedEvent: JobRunEventInsert = { + attempt: 0, + jobRunId: run.id, + kind: "queued", + message: null, + occurredAt: at, + progressJson: null, + sequence: 1, + workerInstanceId: null, + }; + await repository.enqueueManualRun({ + ...noSideEffects, + queuedEvent, + run, + }); + + expect(await waitForTerminal(() => repository.findRun(run.id)?.state)).toBe( + "succeeded" + ); + expect(repository.findRun(run.id)).toMatchObject({ + attemptCount: 1, + resultJson: expect.stringContaining('"status":"ok"'), + state: "succeeded", + }); + } finally { + await coordinator.dispose().catch(() => {}); + database.sqlite.close(true); + } + }); +}); diff --git a/greenfield/src/server/test/support/requestContext.ts b/greenfield/src/server/test/support/requestContext.ts index 1adfeaa83..bd3cb47dd 100644 --- a/greenfield/src/server/test/support/requestContext.ts +++ b/greenfield/src/server/test/support/requestContext.ts @@ -7,6 +7,8 @@ import type { } from "../../../contracts/security.ts"; import type { AgentService } from "../../domains/agents/service.ts"; import { createTestAgentService } from "../../domains/agents/testSupport/service.ts"; +import type { JobService } from "../../domains/jobs/service.ts"; +import { createTestJobService } from "../../domains/jobs/testSupport/service.ts"; import type { MonitoringCatalogService } from "../../domains/monitoring/catalogService.ts"; import type { MonitoringService } from "../../domains/monitoring/service.ts"; import { @@ -394,6 +396,7 @@ export interface TestServerSecurityServices { readonly automationSecurityLifecycle: AutomationSecurityLifecycleService; readonly mfaAccountLifecycle: MfaAccountLifecycleService; readonly mfaLoginLifecycle: MfaLoginLifecycleService; + readonly jobService: JobService["Service"]; readonly monitoringCatalogService: MonitoringCatalogService["Service"]; readonly monitoringService: MonitoringService["Service"]; readonly securityAuditLifecycle: SecurityAuditLifecycleService; @@ -423,6 +426,7 @@ export function createTestServerSecurityServices( overrides.mfaAccountLifecycle ?? createTestMfaAccountLifecycleService(), mfaLoginLifecycle: overrides.mfaLoginLifecycle ?? createTestMfaLoginLifecycleService(), + jobService: overrides.jobService ?? createTestJobService(), monitoringCatalogService: overrides.monitoringCatalogService ?? createTestMonitoringCatalogService(), monitoringService: overrides.monitoringService ?? createTestMonitoringService(), @@ -511,6 +515,7 @@ export function createTestRequestContext( readonly automationSecurityLifecycle?: AutomationSecurityLifecycleService; readonly mfaAccountLifecycle?: MfaAccountLifecycleService; readonly mfaLoginLifecycle?: MfaLoginLifecycleService; + readonly jobService?: JobService["Service"]; readonly monitoringCatalogService?: MonitoringCatalogService["Service"]; readonly monitoringService?: MonitoringService["Service"]; readonly request?: Request; @@ -538,6 +543,7 @@ export function createTestRequestContext( options.mfaAccountLifecycle ?? createTestMfaAccountLifecycleService(), mfaLoginLifecycle: options.mfaLoginLifecycle ?? createTestMfaLoginLifecycleService(), + jobService: options.jobService ?? createTestJobService(), monitoringCatalogService: options.monitoringCatalogService ?? createTestMonitoringCatalogService(), monitoringService: options.monitoringService ?? createTestMonitoringService(), diff --git a/greenfield/src/server/test/system/serverGatewayCredentialVerification.test.ts b/greenfield/src/server/test/system/serverGatewayCredentialVerification.test.ts index 7446b01fc..807e70133 100644 --- a/greenfield/src/server/test/system/serverGatewayCredentialVerification.test.ts +++ b/greenfield/src/server/test/system/serverGatewayCredentialVerification.test.ts @@ -246,6 +246,21 @@ describe("native Gateway bootstrap verification through the real server", () => test("propagates a real HTTP abort through Effect to the native socket", async () => { const system = await openGatewayVerificationSystem("silent", 5000); const controller = new AbortController(); + const protectedTables = [ + "audit_events", + "auth_rate_limit_buckets", + "auth_sessions", + "users", + ] as const; + const countsBeforeAbort: Record<(typeof protectedTables)[number], number> = { + audit_events: countRows(system.database, "audit_events"), + auth_rate_limit_buckets: countRows( + system.database, + "auth_rate_limit_buckets" + ), + auth_sessions: countRows(system.database, "auth_sessions"), + users: countRows(system.database, "users"), + }; try { const request = postAbortableBootstrap( system.server, @@ -263,13 +278,8 @@ describe("native Gateway bootstrap verification through the real server", () => }); await waitForGatewayConnectionState(system.gateway, { open: 0 }); expect(system.gateway.acceptedConnections).toBe(1); - for (const table of [ - "audit_events", - "auth_rate_limit_buckets", - "auth_sessions", - "users", - ] as const) { - expect(countRows(system.database, table)).toBe(0); + for (const table of protectedTables) { + expect(countRows(system.database, table)).toBe(countsBeforeAbort[table]); } } finally { controller.abort(); diff --git a/greenfield/src/server/trpc/appRouter.ts b/greenfield/src/server/trpc/appRouter.ts index 067a4a06c..10e762bf0 100644 --- a/greenfield/src/server/trpc/appRouter.ts +++ b/greenfield/src/server/trpc/appRouter.ts @@ -1,4 +1,10 @@ import { agentProcedureNames, agentRouter } from "../domains/agents/procedures.ts"; +import { + jobProcedureNames, + jobRouter, + scheduleProcedureNames, + scheduleRouter, +} from "../domains/jobs/procedures.ts"; import { incidentProcedureNames, incidentRouter, @@ -42,9 +48,11 @@ export const appRouter = router({ automationSecurity: automationSecurityRouter, events: eventsRouter, incidents: incidentRouter, + jobs: jobRouter, monitoring: monitoringRouter, notifications: notificationRouter, reports: reportRouter, + schedules: scheduleRouter, securityAudit: securityAuditRouter, system: systemRouter, tasks: taskRouter, @@ -58,9 +66,11 @@ export const appRouterProcedureNames = Object.freeze([ ...namespacedProcedureNames("automationSecurity", automationSecurityProcedureNames), ...namespacedProcedureNames("events", eventsProcedureNames), ...namespacedProcedureNames("incidents", incidentProcedureNames), + ...namespacedProcedureNames("jobs", jobProcedureNames), ...namespacedProcedureNames("monitoring", monitoringProcedureNames), ...namespacedProcedureNames("notifications", notificationProcedureNames), ...namespacedProcedureNames("reports", reportProcedureNames), + ...namespacedProcedureNames("schedules", scheduleProcedureNames), ...namespacedProcedureNames("securityAudit", securityAuditProcedureNames), ...namespacedProcedureNames("system", systemProcedureNames), ...namespacedProcedureNames("tasks", taskProcedureNames), diff --git a/greenfield/src/server/trpc/context.test.ts b/greenfield/src/server/trpc/context.test.ts index 460b3d783..5e8ab248e 100644 --- a/greenfield/src/server/trpc/context.test.ts +++ b/greenfield/src/server/trpc/context.test.ts @@ -1,6 +1,7 @@ import { describe, expect, test } from "bun:test"; import { createTestAgentService } from "../domains/agents/testSupport/service.ts"; +import { createTestJobService } from "../domains/jobs/testSupport/service.ts"; import { createTestMonitoringCatalogService, createTestMonitoringService, @@ -35,6 +36,7 @@ describe("tRPC request context", () => { createTestAutomationSecurityLifecycleService(); const monitoringCatalogService = createTestMonitoringCatalogService(); const monitoringService = createTestMonitoringService(); + const jobService = createTestJobService(); const responseHeaders = new Headers(); const context = await createRequestContext({ @@ -65,6 +67,7 @@ describe("tRPC request context", () => { }, mfaAccountLifecycle: createTestMfaAccountLifecycleService(), mfaLoginLifecycle: createTestMfaLoginLifecycleService(), + jobService, monitoringCatalogService, monitoringService, pendingLoginCredential: credentials.pendingLogin, @@ -94,6 +97,7 @@ describe("tRPC request context", () => { expect(context.automationSecurityLifecycle).toBe(automationSecurityLifecycle); expect(context.monitoringCatalogService).toBe(monitoringCatalogService); expect(context.monitoringService).toBe(monitoringService); + expect(context.jobService).toBe(jobService); expect(context.authenticationClientSourceId).toBe("client-source-1"); expect(context.pendingLoginCredential).toEqual({ kind: "present", @@ -127,6 +131,7 @@ describe("tRPC request context", () => { authenticateCredential: () => ({ authentication: { kind: "anonymous" } }), mfaAccountLifecycle: createTestMfaAccountLifecycleService(), mfaLoginLifecycle: createTestMfaLoginLifecycleService(), + jobService: createTestJobService(), monitoringCatalogService: createTestMonitoringCatalogService(), monitoringService: createTestMonitoringService(), pendingLoginCredential: credentials.pendingLogin, @@ -168,6 +173,7 @@ describe("tRPC request context", () => { }), mfaAccountLifecycle: createTestMfaAccountLifecycleService(), mfaLoginLifecycle: createTestMfaLoginLifecycleService(), + jobService: createTestJobService(), monitoringCatalogService: createTestMonitoringCatalogService(), monitoringService: createTestMonitoringService(), pendingLoginCredential: credentials.pendingLogin, diff --git a/greenfield/src/server/trpc/context.ts b/greenfield/src/server/trpc/context.ts index 03b888299..a2dbcf240 100644 --- a/greenfield/src/server/trpc/context.ts +++ b/greenfield/src/server/trpc/context.ts @@ -1,5 +1,6 @@ import type { RequestAuthentication } from "../../contracts/security.ts"; import type { AgentService } from "../domains/agents/service.ts"; +import type { JobService } from "../domains/jobs/service.ts"; import type { MonitoringCatalogService } from "../domains/monitoring/catalogService.ts"; import type { MonitoringService } from "../domains/monitoring/service.ts"; import type { AuthenticationLifecycleService } from "../domains/security/authenticationLifecycle.ts"; @@ -35,6 +36,7 @@ export interface RequestContextOptions { readonly authenticateCredential: AuthenticateCredential; readonly mfaAccountLifecycle: MfaAccountLifecycleService; readonly mfaLoginLifecycle: MfaLoginLifecycleService; + readonly jobService: JobService["Service"]; readonly monitoringCatalogService: MonitoringCatalogService["Service"]; readonly monitoringService: MonitoringService["Service"]; readonly pendingLoginCredential: PendingLoginCredential; @@ -55,6 +57,7 @@ export interface RequestContext { readonly authenticationLease?: AuthenticationLease; readonly mfaAccountLifecycle: MfaAccountLifecycleService; readonly mfaLoginLifecycle: MfaLoginLifecycleService; + readonly jobService: JobService["Service"]; readonly monitoringCatalogService: MonitoringCatalogService["Service"]; readonly monitoringService: MonitoringService["Service"]; readonly pendingLoginCredential: PendingLoginCredential; @@ -86,6 +89,7 @@ export async function createRequestContext( automationSecurityLifecycle: options.automationSecurityLifecycle, mfaAccountLifecycle: options.mfaAccountLifecycle, mfaLoginLifecycle: options.mfaLoginLifecycle, + jobService: options.jobService, monitoringCatalogService: options.monitoringCatalogService, monitoringService: options.monitoringService, ...(resolution.lease && { authenticationLease: resolution.lease }), diff --git a/greenfield/src/server/trpc/procedureErrorPolicy.ts b/greenfield/src/server/trpc/procedureErrorPolicy.ts index fb097dda8..8216859bf 100644 --- a/greenfield/src/server/trpc/procedureErrorPolicy.ts +++ b/greenfield/src/server/trpc/procedureErrorPolicy.ts @@ -208,6 +208,21 @@ export const procedureExpectedErrorPolicy = freezeProcedureExpectedErrorPolicy({ ], "incidents.get": ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], "incidents.list": ["FORBIDDEN", "UNAUTHORIZED"], + "jobs.cancelRun": [ + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + "jobs.getRun": ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + "jobs.listRuns": ["FORBIDDEN", "UNAUTHORIZED"], + "jobs.setClaimingPaused": [ + "CONFLICT", + "FORBIDDEN", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], "monitoring.submitCompleteSnapshot": [ "BAD_REQUEST", "CONFLICT", @@ -254,6 +269,24 @@ export const procedureExpectedErrorPolicy = freezeProcedureExpectedErrorPolicy({ "SERVICE_UNAVAILABLE", "UNAUTHORIZED", ], + "schedules.get": ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + "schedules.list": ["FORBIDDEN", "UNAUTHORIZED"], + "schedules.listRuns": ["FORBIDDEN", "NOT_FOUND", "UNAUTHORIZED"], + "schedules.run": [ + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], + "schedules.update": [ + "BAD_REQUEST", + "CONFLICT", + "FORBIDDEN", + "NOT_FOUND", + "SERVICE_UNAVAILABLE", + "UNAUTHORIZED", + ], "securityAudit.listEvents": ["FORBIDDEN", "UNAUTHORIZED"], "system.runtimeIdentity": [], "tasks.addUpdate": [ diff --git a/greenfield/src/shared/databaseMigrationManifest.ts b/greenfield/src/shared/databaseMigrationManifest.ts index eec3a194d..7ac82d660 100644 --- a/greenfield/src/shared/databaseMigrationManifest.ts +++ b/greenfield/src/shared/databaseMigrationManifest.ts @@ -13,8 +13,8 @@ export const migrationManifest = Object.freeze; + dispose(forceSignal?: AbortSignal): Promise; + initialize(): Promise; +} diff --git a/greenfield/src/test/integration/delivery/productionReleaseLifecycle.test.ts b/greenfield/src/test/integration/delivery/productionReleaseLifecycle.test.ts index a857c2c45..956f75e11 100644 --- a/greenfield/src/test/integration/delivery/productionReleaseLifecycle.test.ts +++ b/greenfield/src/test/integration/delivery/productionReleaseLifecycle.test.ts @@ -1,9 +1,12 @@ +import { Database } from "bun:sqlite"; import { afterEach, describe, expect, test } from "bun:test"; import { cp, lstat, mkdtemp, readFile, symlink } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; +import { drizzle } from "drizzle-orm/bun-sqlite"; import { Effect } from "effect"; +import * as v from "valibot"; import type { BuildSourceIdentity } from "../../../../scripts/buildSourceIdentity.ts"; import { buildDashboardRelease } from "../../../../scripts/delivery/buildRelease.ts"; @@ -21,6 +24,10 @@ import { pointProductionProcessesAtRelease } from "../../../../scripts/delivery/ import { prepareProtectedProductionStatePath } from "../../../../scripts/delivery/productionStateFilesystem.ts"; import type { ReleaseRuntimeIdentity } from "../../../../scripts/delivery/releaseIdentity.ts"; import { removeProductionDeliveryFixtures } from "../../../../scripts/testSupport/productionDeliveryFixture.ts"; +import { jobRunSummarySchema } from "../../../contracts/jobModel.ts"; +import { jobRunDetailSchema } from "../../../contracts/jobs.ts"; +import { seedAuthenticationTestDatabase } from "../../../server/domains/security/testSupport/authentication.ts"; +import { dashboardSessionCookieName } from "../../../server/rawHttp/authenticationCredentials.ts"; const sourceProjectRoot = path.resolve(import.meta.dir, "../../../.."); const releaseId = "d".repeat(40); @@ -98,19 +105,104 @@ function webEnvironment(projectRoot: string, port: number): Record | undefined -): Promise { - if (!child || child.exitCode !== null) return; + child: Bun.Subprocess<"ignore", "ignore", "ignore"> +): Promise { + if (child.exitCode !== null) { + throw new Error("Production child exited before the shutdown signal"); + } child.kill("SIGTERM"); - const exited = await Promise.race([ - child.exited.then(() => true), - Bun.sleep(5000).then(() => false), + const exitCodeBeforeDeadline = await Promise.race([ + child.exited, + Bun.sleep(5000).then(() => null), ]); - if (!exited && child.exitCode === null) { + let forced = false; + if (exitCodeBeforeDeadline === null && child.exitCode === null) { child.kill("SIGKILL"); - await child.exited; + forced = true; + } + const exitCode = await child.exited; + if (!forced && exitCode !== 0) { + throw new Error( + "Production child did not exit cleanly after the shutdown signal" + ); + } + return Object.freeze({ exitCode, forced }); +} + +interface TrpcEnvelope { + readonly error?: unknown; + readonly result?: { readonly data?: { readonly json?: unknown } }; +} + +async function runBundledWorkerSmoke( + databasePath: string, + port: number +): Promise> { + const sqlite = new Database(databasePath, { + create: false, + readwrite: true, + strict: true, + }); + sqlite.exec("PRAGMA busy_timeout = 5000"); + sqlite.exec("PRAGMA foreign_keys = ON"); + let sessionToken: string; + try { + sessionToken = seedAuthenticationTestDatabase( + drizzle({ client: sqlite }), + new Date() + ).session.token; + } finally { + sqlite.close(true); + } + + const headers = { + cookie: `${dashboardSessionCookieName}=${sessionToken}`, + }; + const enqueueResponse = await fetch(`http://127.0.0.1:${port}/trpc/schedules.run`, { + body: JSON.stringify({ + json: { + id: "system.worker-smoke", + idempotencyKey: "cmVsZWFzZS13b3JrZXItc21va2UtMjAyNi0wOC0wNw", + }, + }), + headers: { ...headers, "content-type": "application/json" }, + method: "POST", + }); + const enqueueBody = (await enqueueResponse.json()) as TrpcEnvelope; + expect(enqueueResponse.status).toBe(200); + expect(enqueueBody.error).toBeUndefined(); + const queued = v.parse(jobRunSummarySchema, enqueueBody.result?.data?.json); + expect(queued).toMatchObject({ + actionKey: "system.worker-smoke", + state: "queued", + triggerType: "manual", + }); + + const deadline = Date.now() + 15_000; + let last: v.InferOutput | undefined; + while (Date.now() < deadline) { + const input = encodeURIComponent(JSON.stringify({ json: { id: queued.id } })); + const response = await fetch( + `http://127.0.0.1:${port}/trpc/jobs.getRun?input=${input}`, + { headers } + ); + const body = (await response.json()) as TrpcEnvelope; + expect(response.status).toBe(200); + expect(body.error).toBeUndefined(); + last = v.parse(jobRunDetailSchema, body.result?.data?.json); + if (last.run.state === "succeeded") return last; + if (["cancelled", "failed", "timed-out"].includes(last.run.state)) break; + await Bun.sleep(50); } + throw new Error( + `Bundled worker smoke did not succeed: ${last?.run.state ?? "missing"}` + ); } class DirectProcessController implements ProductionServiceController { @@ -118,6 +210,9 @@ class DirectProcessController implements ProductionServiceController { readonly #paths: Parameters[1]; readonly #port: number; readonly #projectRoot: string; + readonly #stopResults: Array< + ChildStopResult & { readonly process: "web" | "worker" } + > = []; #web: Bun.Subprocess<"ignore", "ignore", "ignore"> | undefined; #worker: Bun.Subprocess<"ignore", "ignore", "ignore"> | undefined; @@ -137,6 +232,12 @@ class DirectProcessController implements ProductionServiceController { return Promise.resolve(); } + get stopResults(): readonly (ChildStopResult & { + readonly process: "web" | "worker"; + })[] { + return Object.freeze([...this.#stopResults]); + } + async start( release: PublishedProductionRelease, runtime: InstalledProductionRuntime @@ -178,10 +279,35 @@ class DirectProcessController implements ProductionServiceController { async stop(): Promise { const web = this.#web; const worker = this.#worker; + if (!web && !worker) return; this.#web = undefined; this.#worker = undefined; - await stopChild(web); - await stopChild(worker); + if (!web || !worker) { + const child = web ?? worker; + if (child && child.exitCode === null) { + child.kill("SIGKILL"); + await child.exited; + } + throw new Error("Production process pair was incomplete during shutdown"); + } + const failures: unknown[] = []; + try { + this.#stopResults.push( + Object.freeze({ ...(await stopChild(web)), process: "web" }) + ); + } catch (error) { + failures.push(error); + } + try { + this.#stopResults.push( + Object.freeze({ ...(await stopChild(worker)), process: "worker" }) + ); + } catch (error) { + failures.push(error); + } + if (failures.length > 0) { + throw new AggregateError(failures, "Production process shutdown failed"); + } } async verifyReady(): Promise { @@ -207,6 +333,25 @@ class DirectProcessController implements ProductionServiceController { } describe("disposable production release lifecycle", () => { + test("rejects a child that exits before its shutdown signal", async () => { + const child = Bun.spawn([process.execPath, "-e", "process.exit(0)"], { + stderr: "ignore", + stdin: "ignore", + stdout: "ignore", + }); + expect(await child.exited).toBe(0); + let stopError: unknown; + try { + await stopChild(child); + } catch (error) { + stopError = error; + } + expect(stopError).toBeInstanceOf(Error); + expect((stopError as Error).message).toBe( + "Production child exited before the shutdown signal" + ); + }); + test("builds, migrates, activates, serves, logs, and shuts down exact artifacts", async () => { const runtimeIdentity = Object.freeze({ revision: Bun.revision, @@ -247,6 +392,19 @@ describe("disposable production release lifecycle", () => { const browser = await fetch(`http://127.0.0.1:${port}/`); expect(browser.status).toBe(200); expect(await browser.text()).toContain("Mira Dashboard"); + const smoke = await runBundledWorkerSmoke( + path.join(paths.stateDirectory, "mira-dashboard.db"), + port + ); + expect(smoke.result).toMatchObject({ + databaseReleaseId: releaseId, + status: "ok", + }); + await services.stop(); + expect(services.stopResults).toEqual([ + { exitCode: 0, forced: false, process: "web" }, + { exitCode: 0, forced: false, process: "worker" }, + ]); const [webLog, workerLog, databaseStatus] = await Promise.all([ readFile(path.join(paths.stateDirectory, "logs/web.ndjson"), "utf8"), readFile( @@ -259,6 +417,7 @@ describe("disposable production release lifecycle", () => { ]); expect(webLog).toContain('"event":"runtime.started"'); expect(workerLog).toContain('"event":"runtime.started"'); + expect(workerLog).toContain('"event":"runtime.stopped"'); expect(databaseStatus.isFile()).toBeTrue(); expect(databaseStatus.mode & 0o777n).toBe(0o600n); } finally { diff --git a/greenfield/src/test/parity/fixtures/greenfield-contracts.json b/greenfield/src/test/parity/fixtures/greenfield-contracts.json index d5fee3b6e..8afe2d2de 100644 --- a/greenfield/src/test/parity/fixtures/greenfield-contracts.json +++ b/greenfield/src/test/parity/fixtures/greenfield-contracts.json @@ -182,6 +182,22 @@ "kind": "query", "name": "incidents.list" }, + { + "kind": "mutation", + "name": "jobs.cancelRun" + }, + { + "kind": "query", + "name": "jobs.getRun" + }, + { + "kind": "query", + "name": "jobs.listRuns" + }, + { + "kind": "mutation", + "name": "jobs.setClaimingPaused" + }, { "kind": "mutation", "name": "monitoring.submitCompleteSnapshot" @@ -226,6 +242,26 @@ "kind": "mutation", "name": "reports.upsert" }, + { + "kind": "query", + "name": "schedules.get" + }, + { + "kind": "query", + "name": "schedules.list" + }, + { + "kind": "query", + "name": "schedules.listRuns" + }, + { + "kind": "mutation", + "name": "schedules.run" + }, + { + "kind": "mutation", + "name": "schedules.update" + }, { "kind": "query", "name": "securityAudit.listEvents" diff --git a/greenfield/src/test/parity/fixtures/legacy-endpoints.json b/greenfield/src/test/parity/fixtures/legacy-endpoints.json index 65c57f351..4489df345 100644 --- a/greenfield/src/test/parity/fixtures/legacy-endpoints.json +++ b/greenfield/src/test/parity/fixtures/legacy-endpoints.json @@ -589,7 +589,7 @@ "purpose": "Lists recent executions plus queue/worker summary; `?include=claims` adds pause state.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["jobs.listRuns"], "phase": "phase-3" @@ -602,7 +602,7 @@ "purpose": "Reads one execution, including its persisted progress/result output snapshot.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["jobs.getRun"], "phase": "phase-3" @@ -615,7 +615,7 @@ "purpose": "Lists Dashboard scheduled jobs.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["schedules.list"], "phase": "phase-3" @@ -628,7 +628,7 @@ "purpose": "Reads a scheduled job.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["schedules.get"], "phase": "phase-3" @@ -641,7 +641,7 @@ "purpose": "Lists job run history.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["schedules.listRuns"], "phase": "phase-3" @@ -1023,7 +1023,7 @@ "purpose": "Pauses/resumes new worker claims; running work is not cancelled.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["jobs.setClaimingPaused"], "phase": "phase-3" @@ -1036,7 +1036,7 @@ "purpose": "Updates scheduled job settings and intentional-disable metadata.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["schedules.update"], "phase": "phase-3" @@ -1608,7 +1608,7 @@ "purpose": "Cancels queued work or requests cooperative cancellation of a running execution.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["jobs.cancelRun"], "phase": "phase-3" @@ -1621,7 +1621,7 @@ "purpose": "Queues a scheduled job and returns `202`.", "section": "Jobs And Cron", "target": { - "delivery": "planned", + "delivery": "implemented", "kind": "procedure", "names": ["schedules.run"], "phase": "phase-3" diff --git a/greenfield/src/worker/runtime.ts b/greenfield/src/worker/runtime.ts index aa2ea05f1..074564320 100644 --- a/greenfield/src/worker/runtime.ts +++ b/greenfield/src/worker/runtime.ts @@ -1,5 +1 @@ -/** Database-validation lifecycle owned by the worker process. */ -export interface DashboardWorkerRuntime { - dispose(): Promise; - initialize(): Promise; -} +export type { DashboardWorkerRuntime } from "../shared/workerRuntime.ts";