Skip to content

feat(rewrite): add Phase 3 cache foundation #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mira-2026 wants to merge 2 commits into from
+5,226 −204
Open

feat(rewrite): add Phase 3 cache foundation #403

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
58efc2b
feat(cache): add claim-fenced system host projections
mira-2026 Aug 8, 2026
69bc914
fix(cache): reject invalid attempts asynchronously
mira-2026 Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion greenfield/docs/architecture/greenfield-rewrite/data-and-security.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ queryable lifecycle.
| Chat replay | unique `chat_run_events(chat_run_id, sequence)` |
| Deployment history | `deployments(state, updated_at_ms DESC)` |
| Docker history | `docker_update_events(managed_service_id, created_at_ms DESC)` |
| Cache refresh/expiry | `cache_entries(status, expires_at_ms)` |
| Cache refresh/expiry | `cache_entries(last_attempt_status, expires_at_ms, key)` |
| Audit cursor | `audit_events(occurred_at_ms DESC, id DESC)` plus request/target indexes |

Primary keys and unique constraints already create indexes; the schema does not add redundant
Expand Down
47 changes: 38 additions & 9 deletions greenfield/docs/architecture/greenfield-rewrite/progress.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@
This matrix is the living phase status. Update it in the same change that materially advances or
closes a phase; dated entries below provide the evidence, not a second status source.

| Phase | Status | Current evidence and remaining gate |
| ----------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 — Evidence and qualification | Complete | All eight mandatory spikes pass on exact Bun revision `17d6843606d76620cb55d31424d7fb0aed51c367`: build, transport, cross-process SQLite/outbox, Drizzle/Bun SQLite, browser data, chat batching, shutdown, and capped resources. Source-derived parity and the OpenClaw source audit pass as additional evidence. |
| 1 — Foundation | Complete | The self-contained future root builds immutable browser/web/worker artifacts, protects project-local production state, installs exact Bun and systemd artifacts, migrates a database copy, atomically promotes the release/database pair, serves readiness/browser assets, writes project-local logs, and proves crash-safe rollback and shutdown in a disposable lifecycle. |
| 2 — Trust and transport | Complete for the stated server scope | Authentication, MFA, WebAuthn, automation credentials, audit, authenticated renewable SSE, one-shot native Gateway bootstrap verification, and the consolidated [threat model](../../security/greenfield-phase-two-threat-model.md) have executable evidence. Browser UI and production cutover remain later gates. |
| 3 — Core operator domains | Started | Task and agent-directory parity are implemented with durable history, realtime invalidation, and browser workflows. Monitoring ingestion plus report, incident, and notification server parity are implemented; report, incident, and global notification browser state are also complete. Dashboard-local durable schedules/jobs, real worker execution, and their `/jobs` operator UI are implemented. OpenClaw cron, overview, and cache/metrics remain open. |
| 4 — Gateway and chat | Not started | The Phase 2 verifier is one-shot only. Persistent native Gateway lifecycle, current-protocol re-audit, sessions, chat journal/recovery, attachments, and frontend remain open. |
| 5 — Privileged and external domains | Not started | Worker-owned file/media, Docker, database, OpenClaw, GitHub, deployment, backup, and other privileged adapters remain open. |
| 6 — Parity, hardening, and cutover | Not started | Full UI parity, generated `/docs`, load/resource/restore evidence, cutover rehearsal, fresh production database, and legacy removal remain open. |
| Phase | Status | Current evidence and remaining gate |
| ----------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 0 — Evidence and qualification | Complete | All eight mandatory spikes pass on exact Bun revision `17d6843606d76620cb55d31424d7fb0aed51c367`: build, transport, cross-process SQLite/outbox, Drizzle/Bun SQLite, browser data, chat batching, shutdown, and capped resources. Source-derived parity and the OpenClaw source audit pass as additional evidence. |
| 1 — Foundation | Complete | The self-contained future root builds immutable browser/web/worker artifacts, protects project-local production state, installs exact Bun and systemd artifacts, migrates a database copy, atomically promotes the release/database pair, serves readiness/browser assets, writes project-local logs, and proves crash-safe rollback and shutdown in a disposable lifecycle. |
| 2 — Trust and transport | Complete for the stated server scope | Authentication, MFA, WebAuthn, automation credentials, audit, authenticated renewable SSE, one-shot native Gateway bootstrap verification, and the consolidated [threat model](../../security/greenfield-phase-two-threat-model.md) have executable evidence. Browser UI and production cutover remain later gates. |
| 3 — Core operator domains | Started | Task and agent-directory parity are implemented with durable history, realtime invalidation, and browser workflows. Monitoring ingestion plus report, incident, and notification server parity are implemented; report, incident, and global notification browser state are also complete. Dashboard-local durable schedules/jobs, real worker execution, their `/jobs` operator UI, and the first claim-fenced `system.host` cache provider are implemented. OpenClaw cron, overview, cache browser consumption, and metrics remain open. |
| 4 — Gateway and chat | Not started | The Phase 2 verifier is one-shot only. Persistent native Gateway lifecycle, current-protocol re-audit, sessions, chat journal/recovery, attachments, and frontend remain open. |
| 5 — Privileged and external domains | Not started | Worker-owned file/media, Docker, database, OpenClaw, GitHub, deployment, backup, and other privileged adapters remain open. |
| 6 — Parity, hardening, and cutover | Not started | Full UI parity, generated `/docs`, load/resource/restore evidence, cutover rehearsal, fresh production database, and legacy removal remain open. |

### 2026-08-03 — Phase 0 started

Expand Down Expand Up @@ -968,3 +968,32 @@ full-browser parity, production rehearsal, cutover, and legacy deletion remain o
versioned schedule updates, and lost-response-safe manual execution. Legacy `/jobs` frontend
parity remains `planned` because the required OpenClaw cron half belongs to Phase 4; overview and
cache/metrics also remain open.

### 2026-08-08 — Phase 3 cache contract, persistence, and first provider

- Three typed procedures now expose one exact cache entry, a bounded payload-free status snapshot,
and caller-scoped idempotent manual refresh. `cache.getStatus` returns at most 128 canonical rows
with one clamped read clock, the full `totalCount`, and an explicit `truncated` flag. The reviewed
parity inventory marks `cache.getEntry`, `cache.getStatus`, and `cache.refreshEntry` implemented;
the composite heartbeat projection remains planned until its OpenClaw cron dependencies exist.
- Durable `cache_entries` keep last-known-good payload, metadata, source, schema, success, and expiry
separate from the latest attempt result. Freshness is derived at read time, so a failed refresh
can remain fresh until expiry and then become stale; a failure without prior data remains missing.
SQL and Valibot invariants reject partial projection groups and inconsistent attempt state.
- The pure job registry now separates browser-safe action definitions from worker-only executors.
The first provider, `system.host`, collects bounded `node:os` and `statfs("/")` values without a
shell, rejects unsafe integers and control characters, and persists only fixed redacted failure
details. Its code-owned daily schedule is due immediately on first registration while later
reconciliation preserves cadence.
- Cache success or failure and its `cache.entries` realtime invalidation commit atomically only
while the worker still owns the exact running attempt, worker ID, lease token, and unexpired
claim. A retry claim fences the prior attempt; failures preserve last-known-good data. Manual
replay is resolved before mutable provider, action, or schedule lookup, and the public
`manualRunAvailable` flag is derived from the current exact action definition rather than stored
database state.
- `cache:read` and `cache:write` are enforced at the contract, tRPC, automation-capability, and SQL
boundaries. The web and worker processes open independent repositories and write-admission
scopes over the same database file and claim-fencing protocol. API enqueue wakes the web event
pump locally; worker-originated cache outbox rows are discovered through its bounded adaptive
cross-process polling. Cache browser consumption, metrics, overview, persistent OpenClaw cron,
and the remaining Phase 3 exit gates stay open.
5 changes: 4 additions & 1 deletion greenfield/docs/generated/procedures.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,10 @@
| `automationSecurity.replaceCapabilities` | mutation | automation-security | MFA enrollment required; recent MFA when enabled | [input](./schemas/automationSecurity.replaceCapabilities.input.schema.json) | [output](./schemas/automationSecurity.replaceCapabilities.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | `mfa_enrollment_required`, `step_up_required` | Atomically replaces a principal's least-privilege capability set. |
| `automationSecurity.revokeCredential` | mutation | automation-security | MFA enrollment required; recent MFA when enabled | [input](./schemas/automationSecurity.revokeCredential.input.schema.json) | [output](./schemas/automationSecurity.revokeCredential.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | `mfa_enrollment_required`, `step_up_required` | Explicitly revokes one automation credential after client cutover. |
| `automationSecurity.rotateCredential` | mutation | automation-security | MFA enrollment required; recent MFA when enabled | [input](./schemas/automationSecurity.rotateCredential.input.schema.json) | [output](./schemas/automationSecurity.rotateCredential.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `PRECONDITION_FAILED`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | `mfa_enrollment_required`, `step_up_required` | Stages a linked replacement credential without revoking its predecessor. |
| `events.stream` | subscription | events | Authenticated; per-topic: agents:read, jobs:read, notifications:read, reports:read, tasks:read | [input](./schemas/events.stream.input.schema.json) | [output](./schemas/events.stream.output.schema.json) | `BAD_REQUEST`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `TOO_MANY_REQUESTS`, `UNAUTHORIZED` | None | Streams authorized durable changes with tracked resume cursors. |
| `cache.getEntry` | query | cache | Authenticated: cache:read | [input](./schemas/cache.getEntry.input.schema.json) | [output](./schemas/cache.getEntry.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Loads one cache projection with last-known-good data and freshness. |
| `cache.getStatus` | query | cache | Authenticated: cache:read | [input](./schemas/cache.getStatus.input.schema.json) | [output](./schemas/cache.getStatus.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists bounded cache freshness and attempt state with an exact total. |
| `cache.refreshEntry` | mutation | cache | Authenticated: cache:write | [input](./schemas/cache.refreshEntry.input.schema.json) | [output](./schemas/cache.refreshEntry.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Enqueues one caller-scoped idempotent cache refresh. |
| `events.stream` | subscription | events | Authenticated; per-topic: agents:read, cache:read, jobs:read, notifications:read, reports:read, tasks:read | [input](./schemas/events.stream.input.schema.json) | [output](./schemas/events.stream.output.schema.json) | `BAD_REQUEST`, `FORBIDDEN`, `SERVICE_UNAVAILABLE`, `TOO_MANY_REQUESTS`, `UNAUTHORIZED` | None | Streams authorized durable changes with tracked resume cursors. |
| `incidents.get` | query | incidents | Authenticated: reports:read | [input](./schemas/incidents.get.input.schema.json) | [output](./schemas/incidents.get.output.schema.json) | `FORBIDDEN`, `NOT_FOUND`, `UNAUTHORIZED` | None | Loads one exact incident lifecycle record. |
| `incidents.list` | query | incidents | Authenticated: reports:read | [input](./schemas/incidents.list.input.schema.json) | [output](./schemas/incidents.list.output.schema.json) | `FORBIDDEN`, `UNAUTHORIZED` | None | Lists stable incident lifecycle rows for report navigation. |
| `jobs.cancelRun` | mutation | jobs | Authenticated browser session: jobs:write | [input](./schemas/jobs.cancelRun.input.schema.json) | [output](./schemas/jobs.cancelRun.output.schema.json) | `CONFLICT`, `FORBIDDEN`, `NOT_FOUND`, `SERVICE_UNAVAILABLE`, `UNAUTHORIZED` | None | Cancels a queued run or requests cooperative running cancellation. |
Expand Down
1 change: 1 addition & 0 deletions greenfield/docs/generated/realtime-events.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
| Topic | Payload | Snapshot | Retention | Summary |
| --- | --- | --- | --- | --- |
| `agents.status` | [payload](./schemas/agents.status.realtime.payload.schema.json) | `agents.listStatuses` | 7 days | Invalidates one agent status row after a durable metadata change. |
| `cache.entries` | [payload](./schemas/cache.entries.realtime.payload.schema.json) | `cache.getStatus` | 7 days | Invalidates one cache projection after a claim-fenced refresh attempt. |
| `jobs.runs` | [payload](./schemas/jobs.runs.realtime.payload.schema.json) | `jobs.listRuns` | 7 days | Invalidates durable run rows and exact queue state. |
| `monitoring.incidents` | [payload](./schemas/monitoring.incidents.realtime.payload.schema.json) | `incidents.list` | 7 days | Invalidates incident lifecycle rows after a complete monitor snapshot. |
| `monitoring.notifications` | [payload](./schemas/monitoring.notifications.realtime.payload.schema.json) | `notifications.list` | 7 days | Invalidates Dashboard notifications after catalog changes. |
Expand Down
4 changes: 3 additions & 1 deletion greenfield/docs/generated/schemas/automationSecurity.createPrincipal.input.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
"enum": [
"agents:read",
"agents:write",
"cache:read",
"cache:write",
"jobs:read",
"jobs:write",
"monitoring:write",
Expand All @@ -20,7 +22,7 @@
],
"type": "string"
},
"maxItems": 11,
"maxItems": 13,
"uniqueItems": true
},
"id": {
Expand Down
Loading