diff --git a/docs-site/src/content/docs/guides/codex-log-guard-reclaim.md b/docs-site/src/content/docs/guides/codex-log-guard-reclaim.md new file mode 100644 index 000000000..2ceab93f6 --- /dev/null +++ b/docs-site/src/content/docs/guides/codex-log-guard-reclaim.md @@ -0,0 +1,98 @@ +--- +title: Codex Log Guard Reclaim +description: Manually reclaim free pages from Codex diagnostic-log SQLite storage with bounded incremental vacuuming. +--- + +Reclaim is the manual space-recovery stage of Codex Log Guard. It compacts the canonical Codex `logs_2.sqlite` database only when the database and runtime pass the same safety checks used by Log Guard protection. + +Reclaim is **never scheduled automatically** and never runs merely because the Storage page is opened. The dashboard requires an explicit Compact action and a second confirmation before it sends the mutation request. + +## What Reclaim does + +OpenCodex performs a bounded offline maintenance sequence: + +1. resolve the canonical `logs_2.sqlite` through Codex's effective `sqlite_home`; +2. verify the file identity and known Codex log schema; +3. verify process enumeration succeeded and no supported Codex writer process is running; +4. acquire the dedicated cross-process Log Guard lock; +5. repeat the Codex-process check while that lock is held; +6. open the existing database read/write without create semantics and prove an immediate SQLite writer can be acquired; +7. require `PRAGMA auto_vacuum` to already be `INCREMENTAL`; +8. run `PRAGMA quick_check` before maintenance; +9. run a full WAL checkpoint and refuse a busy/incomplete checkpoint; +10. execute bounded `PRAGMA incremental_vacuum(N)` batches, checkpointing after each batch; +11. run `PRAGMA quick_check` again after maintenance; and +12. report before/after database, WAL, page-count, freelist, and reclaimable-byte metrics. + +The default batch target is approximately **8 MiB of SQLite pages**. A single invocation reclaims at most approximately **256 MiB of pages**, with an additional finite iteration cap. If more free pages remain, the result is reported as partial and you can invoke Compact again later. + +The byte limits are converted to page counts using the database's actual SQLite page size. They are bounds on logical SQLite pages processed, not claims about SSD/NAND write volume. + +## Safety guarantees + +Reclaim deliberately does **not**: + +- run full `VACUUM`; +- change `auto_vacuum` mode on an existing Codex database; +- delete, truncate, rename, or otherwise manipulate Codex `-wal` / `-shm` files directly; +- delete diagnostic rows; +- modify Log Guard protection triggers or unrelated user triggers; +- run while Codex is detected as active; +- proceed when process enumeration is uncertain; +- proceed on an unknown future log schema; or +- continue after a failed SQLite integrity check. + +A busy Log Guard lock, busy SQLite writer, or busy initial checkpoint is returned as an explicit refusal rather than being retried in the background. If checkpoint contention appears only after an incremental-vacuum batch has already committed, OpenCodex reports the work already completed as a successful partial result with `stopReason: "busy"` instead of claiming that nothing changed. + +## CLI + +Inspect reclaimable space first: + +```bash +ocx storage codex-logs status +``` + +Run one bounded maintenance pass: + +```bash +ocx storage codex-logs compact +``` + +For machine-readable before/after metrics: + +```bash +ocx storage codex-logs compact --json +``` + +If the result reports that more reclaimable space remains, stop there unless you explicitly want another bounded pass. OpenCodex does not loop indefinitely or schedule a follow-up pass for you. + +## Management API + +Compaction is exposed only as a mutation endpoint: + +```text +POST /api/storage/codex-logs/compact +``` + +There is no GET alias for compaction. A successful response contains a `report` object with before/after measurements, reclaimed page counts, physical main-database size change, iteration count, completeness, stop reason, and integrity status. + +Typical refusal states include: + +- `codex_running` +- `process_enumeration_failed` +- `busy` +- `unsupported_schema` +- `auto_vacuum_not_incremental` +- `unsafe_path` +- `integrity_check_failed` +- `database_error` + +Integrity failures include whether they occurred before or after the maintenance pass. A `busy` refusal means contention was detected before any vacuum batch committed; `stopReason: "busy"` inside a successful report means at least one batch committed before later checkpoint contention stopped the pass. + +## Understanding the result + +`pagesReclaimed` and `logicalBytesReclaimed` describe SQLite freelist pages removed during the pass. `physicalDatabaseBytesReclaimed` reports the observed reduction in the main database file after the maintenance checkpoints. + +Those numbers can differ. SQLite/WAL/filesystem behaviour means reclaiming logical pages does not guarantee an identical immediate physical-file reduction, and none of these metrics should be interpreted as NAND writes, SSD wear, or TBW consumed/saved. + +`complete: true` means the observed freelist reached zero. A partial result uses `stopReason: "page_budget"` when either the per-run page budget or the finite iteration cap ends the pass, `stopReason: "no_progress"` when SQLite stops reducing the freelist, and `stopReason: "busy"` when checkpoint contention appears after committed reclamation. All three are bounded outcomes; none causes an automatic retry. diff --git a/gui/src/components/storage-workspace/StorageWorkspace.tsx b/gui/src/components/storage-workspace/StorageWorkspace.tsx index 7f57ab8c0..fb5ed7808 100644 --- a/gui/src/components/storage-workspace/StorageWorkspace.tsx +++ b/gui/src/components/storage-workspace/StorageWorkspace.tsx @@ -9,6 +9,7 @@ import { useMemo, useState } from "react"; import { IconChevron, IconHardDrive } from "../../icons"; import { useT, type TFn, type TKey, type Locale } from "../../i18n/shared"; import { logGuardLabel } from "../../i18n/log-guard-labels"; +import { logGuardOperationLabel } from "../../i18n/log-guard-operation-labels"; import { logGuardProtectionModeLabel, logGuardProtectionStateLabel, @@ -87,7 +88,8 @@ export interface StorageReport { export type CodexLogGuardAction = | { action: "protect"; mode: "compat" | "quiet" } | { action: "unprotect" } - | { action: "repair" }; + | { action: "repair" } + | { action: "compact" }; // Known scanner bucket keys -> localized labels; unknown future keys fall back to the API label. const BUCKET_TKEYS: Record = { @@ -117,15 +119,22 @@ function rowsDisplay(bucket: StorageBucket, locale: Locale, t: TFn): string { function mutationErrorLabel(locale: Locale, code: unknown): string { switch (code) { - case "codex_running": return logGuardLabel(locale, "error.codex_running"); - case "process_enumeration_failed": return logGuardLabel(locale, "error.process_enumeration_failed"); - case "busy": return logGuardLabel(locale, "error.busy"); - case "unsupported_schema": return logGuardLabel(locale, "error.unsupported_schema"); + case "codex_running": return logGuardOperationLabel(locale, "error.codex_running"); + case "process_enumeration_failed": return logGuardOperationLabel(locale, "error.process_enumeration_failed"); + case "busy": return logGuardOperationLabel(locale, "error.busy"); + case "unsupported_schema": return logGuardOperationLabel(locale, "error.unsupported_schema"); case "trigger_collision": return logGuardLabel(locale, "error.trigger_collision"); - case "unsafe_path": return logGuardLabel(locale, "error.unsafe_path"); - case "database_error": return logGuardLabel(locale, "error.database_error"); + case "unsafe_path": return logGuardOperationLabel(locale, "error.unsafe_path"); + case "database_error": return logGuardOperationLabel(locale, "error.database_error"); + // Two distinct situations that previously collapsed into the generic + // message: an unsupported auto-vacuum configuration (nothing can be + // reclaimed without a full rebuild) versus a failed integrity check + // (the database itself is suspect). A user cannot act on "something + // went wrong". + case "auto_vacuum_not_incremental": return logGuardOperationLabel(locale, "error.auto_vacuum_not_incremental"); + case "integrity_check_failed": return logGuardOperationLabel(locale, "error.integrity_check_failed"); case "config_write_failed": return logGuardLabel(locale, "error.config_write_failed"); - default: return logGuardLabel(locale, "error.generic"); + default: return logGuardOperationLabel(locale, "error.generic"); } } @@ -135,6 +144,7 @@ function CodexLogGuardPanel({ t, busy, error, + compaction, onAction, }: { report: CodexLogGuardReport; @@ -142,6 +152,8 @@ function CodexLogGuardPanel({ t: TFn; busy: boolean; error: string | null; + /** Outcome of the last compaction POST, retained independently of the status refresh. */ + compaction: string | null; onAction: (action: CodexLogGuardAction) => void; }) { const metrics = report.metrics; @@ -149,6 +161,10 @@ function CodexLogGuardPanel({ || report.capabilities.reclaim.state === "unsupported"; const protection = report.protection; const mutationDisabled = busy || report.capabilities.protection.state !== "supported"; + const reclaimAvailable = protection !== undefined + && report.capabilities.reclaim.state === "supported" + && (metrics?.reclaimableBytes ?? 0) > 0; + const [confirmCompact, setConfirmCompact] = useState(false); return (
@@ -248,12 +264,64 @@ function CodexLogGuardPanel({ {logGuardLabel(locale, "repair")} )} - {busy && {logGuardLabel(locale, "applying")}} + {busy && {logGuardOperationLabel(locale, "applying")}}
{error &&

{error}

} )} + {reclaimAvailable && ( +
+
+ {!confirmCompact ? ( + + ) : ( + <> + + + + )} +
+
+ )} + + {compaction && ( + // Rendered OUTSIDE `reclaimAvailable` on purpose. A fully successful + // compaction drives reclaimableBytes to 0, which retires the reclaim + // section — so nesting the result inside it hid the outcome in exactly + // the best case. The POST's own result is also shown when the follow-up + // status refresh fails, since the mutation had already succeeded. +
+

{compaction}

+
+ )} + {metrics && metrics.topTargets.length > 0 && (

target

@@ -296,6 +364,17 @@ type GenerationScopedError = { message: string; }; +/** + * The compaction POST's own report. Kept separately from the periodic status + * refresh: the mutation already succeeded, so its outcome must survive even if + * the follow-up GET fails. Without this, a successful compact whose refresh + * failed showed the user nothing and invited them to run it again. + */ +type GenerationScopedCompaction = { + generation: number; + summary: string; +}; + export default function StorageWorkspace({ report, locale, @@ -307,6 +386,7 @@ export default function StorageWorkspace({ const [logGuardOverride, setLogGuardOverride] = useState(null); const [internalLogGuardBusy, setInternalLogGuardBusy] = useState(false); const [logGuardError, setLogGuardError] = useState(null); + const [logGuardCompaction, setLogGuardCompaction] = useState(null); const sortedBuckets = useMemo( () => report.buckets.toSorted((a, b) => b.bytes - a.bytes), @@ -319,6 +399,9 @@ export default function StorageWorkspace({ const displayedLogGuardError = logGuardError?.generation === report.generatedAt ? logGuardError.message : null; + const displayedCompaction = logGuardCompaction?.generation === report.generatedAt + ? logGuardCompaction.summary + : null; const effectiveLogGuardBusy = logGuardBusy || internalLogGuardBusy; const largestAcross = useMemo(() => { @@ -344,6 +427,10 @@ export default function StorageWorkspace({ void (async () => { setInternalLogGuardBusy(true); setLogGuardError(null); + // A new attempt invalidates the previous receipt. Leaving it up meant a + // failed retry could render a stale success alongside the current error, + // so the section misreported the latest attempt. + if (action.action === "compact") setLogGuardCompaction(null); try { const suffix = action.action === "protect" ? "protect" : action.action; const init: RequestInit = { @@ -359,10 +446,64 @@ export default function StorageWorkspace({ setLogGuardError({ generation, message: mutationErrorLabel(locale, errorPayload.error) }); return; } + if (action.action === "compact") { + // Read the POST's own report FIRST. It is the authoritative record of + // what this mutation reclaimed; discarding it meant a successful + // compaction whose refresh then failed showed the user nothing at all, + // hiding pagesReclaimed/complete/stopReason and inviting them to + // repeat a mutation that already worked. + const posted = await response.json().catch(() => null) as { + report?: { + pagesReclaimed?: number; + logicalBytesReclaimed?: number; + physicalDatabaseBytesReclaimed?: number; + complete?: boolean; + stopReason?: string; + }; + } | null; + const postedReport = posted?.report; + if (postedReport) { + // Both figures are shown because they answer different questions: an + // incremental vacuum can return pages to the free list without the + // file shrinking, so "logical > 0, physical 0" is normal progress + // rather than a failed run. + const logical = formatBytes(postedReport.logicalBytesReclaimed ?? 0, locale); + const physical = formatBytes(postedReport.physicalDatabaseBytesReclaimed ?? 0, locale); + const state = postedReport.complete + ? logGuardLabel(locale, "compactComplete") + : logGuardLabel(locale, "compactPartial"); + // stopReason is what distinguishes page_budget from no_progress from + // busy, and pagesReclaimed is the unit the budget is actually spent + // in. Reporting only bytes left a partial run indistinguishable from + // any other partial run. + const pages = postedReport.pagesReclaimed ?? 0; + const reason = !postedReport.complete && postedReport.stopReason + ? ` (${postedReport.stopReason})` + : ""; + setLogGuardCompaction({ + generation, + summary: [ + `${state}${reason}`, + `${pages.toLocaleString(locale)} ${logGuardLabel(locale, "pagesUnit")}`, + `${logical} / ${physical}`, + ].join(" — "), + }); + } + // The mutation has already succeeded. Refresh is deliberately best effort so + // a transient GET/JSON failure cannot be presented as a failed compaction. + try { + const refreshed = await fetch(`${API_BASE}/api/storage/codex-logs`); + if (refreshed.ok) { + const payload = await refreshed.json() as CodexLogGuardReport; + setLogGuardOverride({ generation, report: payload }); + } + } catch { /* keep the existing report after successful compaction */ } + return; + } const payload = await response.json() as CodexLogGuardReport; setLogGuardOverride({ generation, report: payload }); } catch { - setLogGuardError({ generation, message: logGuardLabel(locale, "error.generic") }); + setLogGuardError({ generation, message: logGuardOperationLabel(locale, "error.generic") }); } finally { setInternalLogGuardBusy(false); } @@ -472,6 +613,7 @@ export default function StorageWorkspace({ t={t} busy={effectiveLogGuardBusy} error={displayedLogGuardError} + compaction={displayedCompaction} onAction={runLogGuardAction} /> ) : report.codexLogsError === "inspect_failed" ? ( diff --git a/gui/src/i18n/log-guard-labels.ts b/gui/src/i18n/log-guard-labels.ts index dab2fafd4..de7ed41af 100644 --- a/gui/src/i18n/log-guard-labels.ts +++ b/gui/src/i18n/log-guard-labels.ts @@ -9,6 +9,12 @@ export type LogGuardLabelKey = | "quiet" | "disable" | "repair" + | "compact" + | "pagesUnit" + | "compactComplete" + | "compactPartial" + | "confirmCompact" + | "cancel" | "applying" | "error.generic" | "error.codex_running" @@ -22,6 +28,12 @@ export type LogGuardLabelKey = const LABELS: Record> = { en: { + compact: 'Compact', + compactComplete: "Compaction complete (logical / on-disk reclaimed)", + pagesUnit: "pages", + compactPartial: "Compaction partial (logical / on-disk reclaimed)", + confirmCompact: 'Confirm compaction', + cancel: 'Cancel', inspectionOnly: 'Inspection only', externalSqliteHome: 'External SQLite storage', inspectionUnavailable: "Diagnostic log inspection is unavailable.", @@ -42,6 +54,12 @@ const LABELS: Record> = { "error.config_write_failed": "The database changed, but OpenCodex could not save the protection setting. Fix config storage, then run Repair.", }, de: { + compact: 'Komprimieren', + compactComplete: "Komprimierung abgeschlossen (logisch / auf Datentraeger freigegeben)", + pagesUnit: "Seiten", + compactPartial: "Komprimierung teilweise (logisch / auf Datentraeger freigegeben)", + confirmCompact: 'Komprimierung bestätigen', + cancel: 'Abbrechen', inspectionOnly: 'Nur Inspektion', externalSqliteHome: 'Externer SQLite-Speicher', inspectionUnavailable: "Die Diagnoseprotokoll-Inspektion ist nicht verfügbar.", @@ -62,6 +80,12 @@ const LABELS: Record> = { "error.config_write_failed": "Die Datenbank wurde geändert, aber OpenCodex konnte die Schutzeinstellung nicht speichern. Repariere den Konfigurationsspeicher und führe danach Reparieren aus.", }, fr: { + compact: "Compacter", + compactComplete: "Compactage termine (logique / recupere sur disque)", + pagesUnit: "pages", + compactPartial: "Compactage partiel (logique / recupere sur disque)", + confirmCompact: "Confirmer le compactage", + cancel: "Annuler", inspectionOnly: "Inspection uniquement", externalSqliteHome: "Stockage SQLite externe", inspectionUnavailable: "L’inspection des journaux de diagnostic est indisponible.", @@ -82,6 +106,12 @@ const LABELS: Record> = { "error.config_write_failed": "La base de données a été modifiée, mais OpenCodex n’a pas pu enregistrer le paramètre de protection. Corrigez le stockage de configuration, puis lancez Réparer.", }, ko: { + compact: '압축', + compactComplete: "압축 완료 (논리 / 디스크 회수량)", + pagesUnit: "페이지", + compactPartial: "압축 부분 완료 (논리 / 디스크 회수량)", + confirmCompact: '압축 확인', + cancel: '취소', inspectionOnly: '검사 전용', externalSqliteHome: '외부 SQLite 저장소', inspectionUnavailable: "진단 로그 검사를 사용할 수 없습니다.", @@ -102,6 +132,12 @@ const LABELS: Record> = { "error.config_write_failed": "데이터베이스는 변경되었지만 OpenCodex가 보호 설정을 저장하지 못했습니다. 구성 저장소를 수정한 뒤 복구를 실행하세요.", }, zh: { + compact: '压缩', + compactComplete: "压缩完成(逻辑 / 磁盘回收)", + pagesUnit: "页", + compactPartial: "压缩部分完成(逻辑 / 磁盘回收)", + confirmCompact: '确认压缩', + cancel: '取消', inspectionOnly: '仅检查', externalSqliteHome: '外部 SQLite 存储', inspectionUnavailable: "诊断日志检查当前不可用。", @@ -112,7 +148,7 @@ const LABELS: Record> = { repair: "修复保护", applying: "正在应用保护…", "error.generic": "无法更改 Codex 日志保护。", - "error.codex_running": "更改日志保护前请先退出 Codex。", + "error.codex_running": "更改 Codex 日志保护前请先退出 Codex。", "error.process_enumeration_failed": "无法确认 Codex 已停止,因此未更改保护设置。", "error.busy": "Codex 日志数据库正忙。请退出 Codex 后重试。", "error.unsupported_schema": "此 Codex 日志数据库结构不支持保护功能。", @@ -122,6 +158,12 @@ const LABELS: Record> = { "error.config_write_failed": "数据库已更改,但 OpenCodex 无法保存保护设置。请先修复配置存储,然后运行“修复保护”。", }, "zh-TW": { + compact: '壓縮', + compactComplete: "压缩完成(逻辑 / 磁盘回收)", + pagesUnit: "頁", + compactPartial: "压缩部分完成(逻辑 / 磁盘回收)", + confirmCompact: '確認壓縮', + cancel: '取消', inspectionOnly: '僅檢查', externalSqliteHome: '外部 SQLite 儲存空間', inspectionUnavailable: "診斷記錄檢查目前無法使用。", @@ -132,7 +174,7 @@ const LABELS: Record> = { repair: "修復保護", applying: "正在套用保護…", "error.generic": "無法變更 Codex 日誌保護。", - "error.codex_running": "變更日誌保護前請先退出 Codex。", + "error.codex_running": "變更 Codex 日誌保護前請先退出 Codex。", "error.process_enumeration_failed": "無法確認 Codex 已停止,因此未變更保護設定。", "error.busy": "Codex 日誌資料庫忙碌中。請退出 Codex 後重試。", "error.unsupported_schema": "此 Codex 日誌資料庫結構不支援保護功能。", @@ -142,6 +184,12 @@ const LABELS: Record> = { "error.config_write_failed": "資料庫已變更,但 OpenCodex 無法儲存保護設定。請先修復設定儲存空間,再執行「修復保護」。", }, ru: { + compact: 'Сжать', + compactComplete: "Сжатие завершено (логически / освобождено на диске)", + pagesUnit: "страниц", + compactPartial: "Сжатие частичное (логически / освобождено на диске)", + confirmCompact: 'Подтвердить сжатие', + cancel: 'Отмена', inspectionOnly: 'Только проверка', externalSqliteHome: 'Внешнее хранилище SQLite', inspectionUnavailable: "Проверка диагностических журналов недоступна.", @@ -162,6 +210,12 @@ const LABELS: Record> = { "error.config_write_failed": "База была изменена, но OpenCodex не смог сохранить настройку защиты. Исправьте хранилище конфигурации и затем запустите восстановление.", }, ja: { + compact: '圧縮', + compactComplete: "圧縮完了(論理 / ディスク解放)", + pagesUnit: "ページ", + compactPartial: "圧縮は部分的に完了(論理 / ディスク解放)", + confirmCompact: '圧縮を確認', + cancel: 'キャンセル', inspectionOnly: '検査のみ', externalSqliteHome: '外部 SQLite ストレージ', inspectionUnavailable: "診断ログの検査を利用できません。", @@ -182,6 +236,12 @@ const LABELS: Record> = { "error.config_write_failed": "データベースは変更されましたが、OpenCodex は保護設定を保存できませんでした。設定ストレージを修正してから保護を修復してください。", }, tr: { + compact: 'Sıkıştır', + compactComplete: "Sikistirma tamamlandi (mantiksal / diskte geri kazanilan)", + pagesUnit: "sayfa", + compactPartial: "Sikistirma kismi (mantiksal / diskte geri kazanilan)", + confirmCompact: 'Sıkıştırmayı onayla', + cancel: 'İptal', inspectionOnly: 'Yalnızca inceleme', externalSqliteHome: 'Harici SQLite depolaması', inspectionUnavailable: "Tanılama günlüğü incelemesi kullanılamıyor.", diff --git a/gui/src/i18n/log-guard-operation-labels.ts b/gui/src/i18n/log-guard-operation-labels.ts new file mode 100644 index 000000000..967621a9e --- /dev/null +++ b/gui/src/i18n/log-guard-operation-labels.ts @@ -0,0 +1,128 @@ +import type { Locale } from "./catalogs"; + +export type LogGuardOperationLabelKey = + | "applying" + | "error.generic" + | "error.codex_running" + | "error.process_enumeration_failed" + | "error.busy" + | "error.unsupported_schema" + | "error.unsafe_path" + | "error.database_error" + | "error.auto_vacuum_not_incremental" + | "error.integrity_check_failed"; + +const LABELS: Record> = { + en: { + applying: "Applying Log Guard change…", + "error.generic": "Could not update Codex log storage.", + "error.codex_running": "Quit Codex before changing Codex log storage.", + "error.process_enumeration_failed": "Could not verify that Codex is stopped. The Log Guard operation was not started.", + "error.busy": "The Codex logs database is busy. Quit Codex and try again.", + "error.unsupported_schema": "This Codex logs schema is not supported for this operation.", + "error.unsafe_path": "The Codex logs database path failed the safety check.", + "error.database_error": "Could not update the Codex logs database.", + "error.auto_vacuum_not_incremental": "This Codex logs database is not configured for incremental vacuum, so space cannot be reclaimed without a full rebuild.", + "error.integrity_check_failed": "The Codex logs database failed its integrity check. No space was reclaimed.", + }, + de: { + applying: "Log-Guard-Änderung wird angewendet…", + "error.generic": "Der Codex-Protokollspeicher konnte nicht aktualisiert werden.", + "error.codex_running": "Beende Codex, bevor du den Codex-Protokollspeicher änderst.", + "error.process_enumeration_failed": "Es konnte nicht sicher festgestellt werden, dass Codex beendet ist. Der Log-Guard-Vorgang wurde nicht gestartet.", + "error.busy": "Die Codex-Protokolldatenbank ist belegt. Beende Codex und versuche es erneut.", + "error.unsupported_schema": "Dieses Schema der Codex-Protokolldatenbank wird für diesen Vorgang nicht unterstützt.", + "error.unsafe_path": "Der Pfad der Codex-Protokolldatenbank hat die Sicherheitsprüfung nicht bestanden.", + "error.database_error": "Die Codex-Protokolldatenbank konnte nicht aktualisiert werden.", + "error.auto_vacuum_not_incremental": "Diese Codex-Protokolldatenbank ist nicht fuer inkrementelles Vacuum konfiguriert; ohne vollstaendigen Neuaufbau kann kein Speicher freigegeben werden.", + "error.integrity_check_failed": "Die Integritaetspruefung der Codex-Protokolldatenbank ist fehlgeschlagen. Es wurde kein Speicher freigegeben.", + }, + fr: { + applying: "Application de la modification Log Guard…", + "error.generic": "Impossible de mettre à jour le stockage des journaux Codex.", + "error.codex_running": "Quittez Codex avant de modifier le stockage des journaux Codex.", + "error.process_enumeration_failed": "Impossible de vérifier que Codex est arrêté. L’opération Log Guard n’a pas été lancée.", + "error.busy": "La base de données des journaux Codex est occupée. Quittez Codex et réessayez.", + "error.unsupported_schema": "Ce schéma de journaux Codex n’est pas pris en charge pour cette opération.", + "error.unsafe_path": "Le chemin de la base de données des journaux Codex a échoué au contrôle de sécurité.", + "error.database_error": "Impossible de mettre à jour la base de données des journaux Codex.", + "error.auto_vacuum_not_incremental": "Cette base de donnees de journaux Codex n'est pas configuree pour le vacuum incrementiel : l'espace ne peut pas etre recupere sans reconstruction complete.", + "error.integrity_check_failed": "La verification d'integrite de la base de donnees de journaux Codex a echoue. Aucun espace n'a ete recupere.", + }, + ko: { + applying: "Log Guard 변경 적용 중…", + "error.generic": "Codex 로그 저장소를 업데이트하지 못했습니다.", + "error.codex_running": "Codex 로그 저장소를 변경하기 전에 Codex를 종료하세요.", + "error.process_enumeration_failed": "Codex가 종료되었는지 확인할 수 없어 Log Guard 작업을 시작하지 않았습니다.", + "error.busy": "Codex 로그 데이터베이스가 사용 중입니다. Codex를 종료한 뒤 다시 시도하세요.", + "error.unsupported_schema": "이 Codex 로그 스키마는 이 작업을 지원하지 않습니다.", + "error.unsafe_path": "Codex 로그 데이터베이스 경로가 안전성 검사를 통과하지 못했습니다.", + "error.database_error": "Codex 로그 데이터베이스를 업데이트하지 못했습니다.", + "error.auto_vacuum_not_incremental": "이 Codex 로그 데이터베이스는 증분 정리로 구성되어 있지 않아, 전체 재구축 없이는 공간을 회수할 수 없습니다.", + "error.integrity_check_failed": "Codex 로그 데이터베이스 무결성 검사에 실패했습니다. 공간이 회수되지 않았습니다.", + }, + zh: { + applying: "正在应用 Log Guard 更改…", + "error.generic": "无法更新 Codex 日志存储。", + "error.codex_running": "更改 Codex 日志存储前请先退出 Codex。", + "error.process_enumeration_failed": "无法确认 Codex 已停止,因此未启动 Log Guard 操作。", + "error.busy": "Codex 日志数据库正忙。请退出 Codex 后重试。", + "error.unsupported_schema": "此 Codex 日志数据库结构不支持此操作。", + "error.unsafe_path": "Codex 日志数据库路径未通过安全检查。", + "error.database_error": "无法更新 Codex 日志数据库。", + "error.auto_vacuum_not_incremental": "此 Codex 日志数据库未配置增量清理,因此不完整重建就无法回收空间。", + "error.integrity_check_failed": "Codex 日志数据库完整性检查失败,未回收任何空间。", + }, + "zh-TW": { + applying: "正在套用 Log Guard 變更…", + "error.generic": "無法更新 Codex 日誌儲存空間。", + "error.codex_running": "變更 Codex 日誌儲存空間前請先退出 Codex。", + "error.process_enumeration_failed": "無法確認 Codex 已停止,因此未啟動 Log Guard 操作。", + "error.busy": "Codex 日誌資料庫忙碌中。請退出 Codex 後重試。", + "error.unsupported_schema": "此 Codex 日誌資料庫結構不支援此操作。", + "error.unsafe_path": "Codex 日誌資料庫路徑未通過安全檢查。", + "error.database_error": "無法更新 Codex 日誌資料庫。", + "error.auto_vacuum_not_incremental": "此 Codex 日志数据库未配置增量清理,因此不完整重建就无法回收空间。", + "error.integrity_check_failed": "Codex 日志数据库完整性检查失败,未回收任何空间。", + }, + ru: { + applying: "Применение изменения Log Guard…", + "error.generic": "Не удалось обновить хранилище журналов Codex.", + "error.codex_running": "Закройте Codex перед изменением хранилища журналов Codex.", + "error.process_enumeration_failed": "Не удалось убедиться, что Codex остановлен. Операция Log Guard не запущена.", + "error.busy": "База журналов Codex занята. Закройте Codex и повторите попытку.", + "error.unsupported_schema": "Эта схема базы журналов Codex не поддерживает эту операцию.", + "error.unsafe_path": "Путь к базе журналов Codex не прошёл проверку безопасности.", + "error.database_error": "Не удалось обновить базу журналов Codex.", + "error.auto_vacuum_not_incremental": "Эта база данных журналов Codex не настроена на инкрементальную очистку, поэтому освободить место без полной перестройки невозможно.", + "error.integrity_check_failed": "Проверка целостности базы данных журналов Codex не пройдена. Место не освобождено.", + }, + ja: { + applying: "Log Guard の変更を適用中…", + "error.generic": "Codex ログストレージを更新できませんでした。", + "error.codex_running": "Codex ログストレージを変更する前に Codex を終了してください。", + "error.process_enumeration_failed": "Codex が停止していることを確認できなかったため、Log Guard 操作を開始しませんでした。", + "error.busy": "Codex ログデータベースが使用中です。Codex を終了して再試行してください。", + "error.unsupported_schema": "この Codex ログスキーマではこの操作を使用できません。", + "error.unsafe_path": "Codex ログデータベースのパスが安全性チェックに失敗しました。", + "error.database_error": "Codex ログデータベースを更新できませんでした。", + "error.auto_vacuum_not_incremental": "この Codex ログ データベースは増分バキューム用に構成されていないため、完全な再構築なしに領域を解放できません。", + "error.integrity_check_failed": "Codex ログ データベースの整合性チェックに失敗しました。領域は解放されていません。", + }, + tr: { + applying: "Log Guard değişikliği uygulanıyor…", + "error.generic": "Codex günlük depolaması güncellenemedi.", + "error.codex_running": "Codex günlük depolamasını değiştirmeden önce Codex'i kapatın.", + "error.process_enumeration_failed": "Codex'in kapalı olduğu doğrulanamadı. Log Guard işlemi başlatılmadı.", + "error.busy": "Codex günlük veritabanı meşgul. Codex'i kapatıp yeniden deneyin.", + "error.unsupported_schema": "Bu Codex günlük şeması bu işlem için desteklenmiyor.", + "error.unsafe_path": "Codex günlük veritabanı yolu güvenlik denetimini geçemedi.", + "error.database_error": "Codex günlük veritabanı güncellenemedi.", + "error.auto_vacuum_not_incremental": "Bu Codex gunluk veritabani artimli vacuum icin yapilandirilmamis; tam yeniden olusturma olmadan alan geri kazanilamaz.", + "error.integrity_check_failed": "Codex gunluk veritabani butunluk denetiminden gecemedi. Alan geri kazanilmadi.", + }, +}; + +export function logGuardOperationLabel(locale: Locale, key: LogGuardOperationLabelKey): string { + return LABELS[locale][key]; +} diff --git a/gui/tests/storage-log-guard-compact.test.tsx b/gui/tests/storage-log-guard-compact.test.tsx new file mode 100644 index 000000000..7db44f804 --- /dev/null +++ b/gui/tests/storage-log-guard-compact.test.tsx @@ -0,0 +1,313 @@ +/** @jsxImportSource react */ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { Window } from "happy-dom"; +import { act } from "react"; +import type { Root } from "react-dom/client"; + +import StorageWorkspace, { type StorageReport } from "../src/components/storage-workspace/StorageWorkspace"; +import { LanguageProvider } from "../src/i18n/provider"; + +const globals = ["document", "window", "navigator", "localStorage", "IS_REACT_ACT_ENVIRONMENT"] as const; +const originalFetch = globalThis.fetch; +let previousGlobals: Record<(typeof globals)[number], unknown>; +let testWindow: Window; +let active: Root | null = null; + +beforeEach(() => { + previousGlobals = Object.fromEntries(globals.map(key => [key, Reflect.get(globalThis, key)])) as typeof previousGlobals; + testWindow = new Window({ url: "http://localhost/" }); + Object.defineProperties(globalThis, { + document: { configurable: true, value: testWindow.document }, + window: { configurable: true, value: testWindow }, + navigator: { configurable: true, value: testWindow.navigator }, + localStorage: { configurable: true, value: testWindow.localStorage }, + }); + (globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; +}); + +afterEach(async () => { + globalThis.fetch = originalFetch; + if (active) { + const root = active; + active = null; + await act(async () => { root.unmount(); }); + } + testWindow.close(); + for (const key of globals) { + Object.defineProperty(globalThis, key, { configurable: true, value: previousGlobals[key] }); + } +}); + +function report(reclaimableBytes = 65536): StorageReport { + return { + codexHome: "/home/user/.codex", + generatedAt: 1, + total: { bytes: 1024, fileCount: 1 }, + buckets: [], + codexLogs: { + generatedAt: 1, + externalSqliteHome: false, + snapshot: "checkpointed", + files: { databaseBytes: 8192, walBytes: 0, shmBytes: 0 }, + schema: { state: "compatible" }, + capabilities: { + inspection: { state: "supported" }, + protection: { state: "supported" }, + reclaim: { state: "supported" }, + }, + protection: { desiredMode: "off", observedMode: "off", state: "off" }, + metrics: { + totalRows: 10, + rowsByLevel: { INFO: 10 }, + traceRows: 0, + traceShare: 0, + topTargets: [], + pageSize: 4096, + pageCount: 100, + freelistPages: reclaimableBytes === 0 ? 0 : 16, + reclaimableBytes, + estimatedLogBytes: null, + }, + }, + } satisfies StorageReport; +} + +async function mount( + value: StorageReport, + onAction?: (action: unknown) => void, + logGuardBusy = false, +): Promise { + const container = document.createElement("div"); + document.body.append(container); + const { createRoot } = await import("react-dom/client"); + const root = createRoot(container); + active = root; + await act(async () => { + root.render( + + + , + ); + }); + return container; +} + +async function click(element: HTMLElement): Promise { + await act(async () => { element.click(); }); +} + +async function settle(): Promise { + await act(async () => { + await new Promise(resolve => testWindow.setTimeout(resolve, 0)); + }); +} + +test("compact requires an explicit second confirmation before emitting the mutation action", async () => { + const actions: unknown[] = []; + const container = await mount(report(), action => actions.push(action)); + + const compact = container.querySelector('[data-testid="log-guard-compact"]'); + expect(compact).not.toBeNull(); + if (!compact) return; + await click(compact); + expect(actions).toEqual([]); + + const confirm = container.querySelector('[data-testid="log-guard-compact-confirm"]'); + expect(confirm).not.toBeNull(); + if (!confirm) return; + await click(confirm); + expect(actions).toEqual([{ action: "compact" }]); +}); + +test("successful compact remains successful when the status refresh fails", async () => { + const calls: string[] = []; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + calls.push(`${init?.method ?? "GET"} ${url}`); + if (url.endsWith("/api/storage/codex-logs/compact")) { + return Response.json({ report: { stopReason: "complete" } }); + } + if (url.endsWith("/api/storage/codex-logs")) { + return new Response("refresh unavailable", { status: 503 }); + } + return new Response(null, { status: 404 }); + }) as typeof fetch; + + const container = await mount(report()); + const compact = container.querySelector('[data-testid="log-guard-compact"]'); + expect(compact).not.toBeNull(); + if (!compact) return; + await click(compact); + + const confirm = container.querySelector('[data-testid="log-guard-compact-confirm"]'); + expect(confirm).not.toBeNull(); + if (!confirm) return; + await click(confirm); + await settle(); + + expect(calls).toEqual([ + "POST /api/storage/codex-logs/compact", + "GET /api/storage/codex-logs", + ]); + expect(container.querySelector('[role="alert"]')).toBeNull(); +}); + +test("shared busy state uses action-neutral Log Guard text", async () => { + const container = await mount(report(), undefined, true); + expect(container.textContent).toContain("Applying Log Guard change…"); + expect(container.textContent).not.toContain("Applying protection"); +}); + +test("compact errors do not claim that protection failed", async () => { + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = String(input); + if (url.endsWith("/api/storage/codex-logs/compact")) { + return Response.json({ error: "unsupported_schema" }, { status: 409 }); + } + return new Response(null, { status: 404 }); + }) as typeof fetch; + + const container = await mount(report()); + const compact = container.querySelector('[data-testid="log-guard-compact"]'); + expect(compact).not.toBeNull(); + if (!compact) return; + await click(compact); + + const confirm = container.querySelector('[data-testid="log-guard-compact-confirm"]'); + expect(confirm).not.toBeNull(); + if (!confirm) return; + await click(confirm); + await settle(); + + const alert = container.querySelector('[role="alert"]'); + expect(alert?.textContent).toContain("not supported for this operation"); + expect(alert?.textContent).not.toContain("protection"); +}); + +test("compact is hidden when the snapshot reports no reclaimable space", async () => { + const container = await mount(report(0), () => {}); + expect(container.querySelector('[data-testid="log-guard-compact"]')).toBeNull(); +}); + +test("compact is hidden when reclaim is unsupported", async () => { + const value = report(); + value.codexLogs = { + ...value.codexLogs!, + capabilities: { + ...value.codexLogs!.capabilities, + reclaim: { state: "unsupported", reason: "unknown_schema" }, + }, + }; + const container = await mount(value, () => {}); + expect(container.querySelector('[data-testid="log-guard-compact"]')).toBeNull(); +}); + +test("a successful compaction shows its report even when the refresh fails", async () => { + // The mutation already succeeded, so its outcome must survive a failed GET. + // pagesReclaimed and stopReason are included because bytes alone leave one + // partial run indistinguishable from another (page_budget vs no_progress vs busy). + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = String(input); + if (url.endsWith("/api/storage/codex-logs/compact")) { + return Response.json({ + report: { + pagesReclaimed: 318, + logicalBytesReclaimed: 1302528, + physicalDatabaseBytesReclaimed: 0, + complete: false, + stopReason: "page_budget", + }, + }); + } + return new Response("refresh unavailable", { status: 503 }); + }) as typeof fetch; + + const container = await mount(report()); + const compact = container.querySelector('[data-testid="log-guard-compact"]'); + if (!compact) throw new Error("compact button missing"); + await click(compact); + const confirm = container.querySelector('[data-testid="log-guard-compact-confirm"]'); + if (!confirm) throw new Error("confirm button missing"); + await click(confirm); + await settle(); + + const result = container.querySelector('[data-testid="log-guard-compact-result"]'); + expect(result).not.toBeNull(); + expect(result?.textContent).toContain("318"); + expect(result?.textContent).toContain("page_budget"); +}); + +test("the compaction report survives a refresh that retires the reclaim section", async () => { + // A fully successful compaction drives reclaimableBytes to 0, which removes + // the reclaim section. Nesting the result inside it hid the outcome in exactly + // the best case. + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = String(input); + if (url.endsWith("/api/storage/codex-logs/compact")) { + return Response.json({ + report: { + pagesReclaimed: 512, + logicalBytesReclaimed: 2097152, + physicalDatabaseBytesReclaimed: 2097152, + complete: true, + stopReason: "complete", + }, + }); + } + return Response.json(report(0).codexLogs); + }) as typeof fetch; + + const container = await mount(report()); + const compact = container.querySelector('[data-testid="log-guard-compact"]'); + if (!compact) throw new Error("compact button missing"); + await click(compact); + const confirm = container.querySelector('[data-testid="log-guard-compact-confirm"]'); + if (!confirm) throw new Error("confirm button missing"); + await click(confirm); + await settle(); + + // The reclaim section is gone, but the result must not be. + expect(container.querySelector('[data-testid="log-guard-reclaim"]')).toBeNull(); + const result = container.querySelector('[data-testid="log-guard-compact-result"]'); + expect(result).not.toBeNull(); + expect(result?.textContent).toContain("512"); +}); + +test("a failed retry clears the previous success receipt", async () => { + // A stale receipt rendered alongside the current error made the section + // misreport the latest attempt. + let call = 0; + globalThis.fetch = (async (input: RequestInfo | URL) => { + const url = String(input); + if (url.endsWith("/api/storage/codex-logs/compact")) { + call += 1; + return call === 1 + ? Response.json({ report: { pagesReclaimed: 7, logicalBytesReclaimed: 28672, physicalDatabaseBytesReclaimed: 0, complete: false, stopReason: "page_budget" } }) + : Response.json({ error: "busy" }, { status: 409 }); + } + return new Response("refresh unavailable", { status: 503 }); + }) as typeof fetch; + + const container = await mount(report()); + const runCompact = async () => { + const compact = container.querySelector('[data-testid="log-guard-compact"]'); + if (!compact) throw new Error("compact button missing"); + await click(compact); + const confirm = container.querySelector('[data-testid="log-guard-compact-confirm"]'); + if (!confirm) throw new Error("confirm button missing"); + await click(confirm); + await settle(); + }; + + await runCompact(); + expect(container.querySelector('[data-testid="log-guard-compact-result"]')).not.toBeNull(); + + await runCompact(); + expect(container.querySelector('[role="alert"]')).not.toBeNull(); + expect(container.querySelector('[data-testid="log-guard-compact-result"]')).toBeNull(); +}); diff --git a/src/cli/observe.ts b/src/cli/observe.ts index c5864968c..9a8a8ba92 100644 --- a/src/cli/observe.ts +++ b/src/cli/observe.ts @@ -18,7 +18,7 @@ const USAGE = `Usage: ocx logs rebuild-index ocx logs index-status ocx observe usage [--range <7d|30d|all>] [--surface ] [--json] - ocx observe storage [codex-logs [status|protect|unprotect|repair] [--mode ]] [--json] + ocx observe storage [codex-logs [status|protect|unprotect|repair|compact] [--mode ]] [--json] ocx observe memory [--json] ocx observe debug [--json] ocx observe claude-inbound [--limit ] [--json] @@ -173,7 +173,7 @@ async function storage(argv: string[], deps: RuntimeApiDeps): Promise { headers: { "content-type": "application/json" }, body: JSON.stringify({ mode: requestedMode }), }, deps); - } else if (action === "unprotect" || action === "repair") { + } else if (action === "unprotect" || action === "repair" || action === "compact") { if (mode !== undefined) throw new CliUsageError("--mode is only valid with codex-logs protect", USAGE); result = await runtimeRequest(`/api/storage/codex-logs/${action}`, { method: "POST" }, deps); } else { diff --git a/src/codex/log-guard/maintenance.ts b/src/codex/log-guard/maintenance.ts new file mode 100644 index 000000000..d488d4154 --- /dev/null +++ b/src/codex/log-guard/maintenance.ts @@ -0,0 +1,403 @@ +import { lstatSync, realpathSync, statSync } from "node:fs"; +import { Database, constants as sqliteConstants } from "bun:sqlite"; + +import { getCodexHome, resolveCodexLogsDbPath } from "../paths"; +import { samePathIdentity } from "../user-identity"; +import { inspectCodexLogs } from "./inspect"; +import { withCodexLogGuardLock, type CodexLogGuardLockOutcome } from "./lock"; +import { sameLogGuardPathIdentity } from "./path-safety"; +import { isSqliteBusy } from "./sqlite-errors"; +import { listRunningCodexProcesses, type CodexWriterProcessCheck } from "./processes"; + +const CURRENT_LOG_COLUMNS = [ + "id", + "ts", + "ts_nanos", + "level", + "target", + "feedback_log_body", + "module_path", + "file", + "line", + "thread_id", + "process_uuid", + "estimated_bytes", +] as const; + +/** + * Budgets are expressed in BYTES and converted with the database's real page + * size, because that is what the guide promises: ~8 MiB per batch and ~256 MiB + * per run. Fixed page counts silently meant something different on every page + * size — at the 4 KiB pages the fixtures use, 512/8192 pages is 2 MiB/32 MiB, + * a quarter of the documented budget. + */ +const DEFAULT_BATCH_BYTES = 8 * 1024 * 1024; +const DEFAULT_MAX_BYTES_PER_RUN = 256 * 1024 * 1024; +const MAX_ITERATIONS = 64; + +/** Convert a byte budget to whole pages, never returning zero pages. */ +function pagesForBytes(bytes: number, pageSize: number): number { + if (!Number.isFinite(pageSize) || pageSize <= 0) return 1; + return Math.max(1, Math.floor(bytes / pageSize)); +} + +type CompactStopReason = "complete" | "page_budget" | "no_progress" | "busy"; + +export interface CodexLogGuardCompactionMeasure { + databaseBytes: number; + /** On-disk WAL sidecar size at measurement time; FULL checkpoint does not imply shrinkage. */ + walBytes: number; + pageCount: number; + freelistPages: number; + reclaimableBytes: number; +} + +export interface CodexLogGuardCompactionReport { + pageSize: number; + before: CodexLogGuardCompactionMeasure; + after: CodexLogGuardCompactionMeasure; + pagesReclaimed: number; + /** + * Logical space returned to the free list, in bytes (`pagesReclaimed * pageSize`). + * Distinct from `physicalDatabaseBytesReclaimed`: an incremental vacuum can + * reclaim pages without the file shrinking, so a run that reports logical + * progress and zero physical shrinkage is normal rather than a failure. The + * guide documented this field before it existed. + */ + logicalBytesReclaimed: number; + physicalDatabaseBytesReclaimed: number; + iterations: number; + complete: boolean; + stopReason: CompactStopReason; + integrity: { before: "ok"; after: "ok" }; +} + +export type CodexLogGuardCompactionError = + | "unsupported_schema" + | "codex_running" + | "process_enumeration_failed" + | "unsafe_path" + | "busy" + | "database_error" + | "auto_vacuum_not_incremental" + | "integrity_check_failed"; + +export type CodexLogGuardCompactionResult = + | { ok: true; report: CodexLogGuardCompactionReport } + | { + ok: false; + error: Exclude; + } + | { ok: false; error: "integrity_check_failed"; phase: "before" | "after" }; + +export interface CodexLogGuardMaintenanceDeps { + codexHome?: string; + processCheck?: () => CodexWriterProcessCheck; + withLock?: ( + canonicalCodexHome: string, + canonicalLogsDbPath: string, + work: () => T, + ) => CodexLogGuardLockOutcome; + quickCheck?: (db: Database) => string[]; + openDatabase?: (databasePath: string, flags: number) => Database; + batchPages?: number; + maxPagesPerRun?: number; +} + +interface ColumnRow { name: string } +interface CheckpointRow { + busy?: number; + log?: number; + checkpointed?: number; +} + +interface DatabaseFileIdentity { + dev: number; + ino: number; + realPath: string; +} + +function databasePathIdentity(databasePath: string): DatabaseFileIdentity | null { + try { + const stat = lstatSync(databasePath); + if (!stat.isFile() || stat.isSymbolicLink()) return null; + const realPath = realpathSync.native(databasePath); + if (!sameLogGuardPathIdentity(realPath, databasePath)) return null; + return { dev: stat.dev, ino: stat.ino, realPath }; + } catch { + return null; + } +} + +function databasePathIsSafe(databasePath: string): boolean { + return databasePathIdentity(databasePath) !== null; +} + +function databasePathStillMatches( + databasePath: string, + before: DatabaseFileIdentity, +): boolean { + const after = databasePathIdentity(databasePath); + return after !== null + && after.dev === before.dev + && after.ino === before.ino + && samePathIdentity(after.realPath, before.realPath); +} + +function exactCurrentSchema(db: Database): boolean { + const columns = db.query("PRAGMA table_info(logs)").all().map(row => row.name).sort(); + const expected = [...CURRENT_LOG_COLUMNS].sort(); + return columns.length === expected.length + && columns.every((value, index) => value === expected[index]); +} + +function pragmaNumber(db: Database, sql: string): number { + const row = db.query, []>(sql).get(); + if (!row) throw new Error(`missing pragma result for ${sql}`); + const value = Number(Object.values(row)[0]); + if (!Number.isFinite(value)) throw new Error(`invalid pragma result for ${sql}`); + return value; +} + +function defaultQuickCheck(db: Database): string[] { + return db.query, []>("PRAGMA quick_check").all().map(row => { + const value = Object.values(row)[0]; + return value === undefined ? "" : String(value); + }); +} + +function quickCheckIsOk(rows: string[]): boolean { + return rows.length === 1 && rows[0]?.trim().toLowerCase() === "ok"; +} + +function processRefusal( + check: CodexWriterProcessCheck, +): "process_enumeration_failed" | "codex_running" | null { + if (check.state === "unknown") return "process_enumeration_failed"; + if (check.processes.length > 0) return "codex_running"; + return null; +} + +function checkpointFull(db: Database): "ok" | "busy" { + const row = db.query("PRAGMA wal_checkpoint(FULL)").get(); + if (!row) throw new Error("missing wal_checkpoint result"); + const values = Object.values(row).map(Number); + const busy = Number(row.busy ?? values[0] ?? 0); + const log = Number(row.log ?? values[1] ?? -1); + const checkpointed = Number(row.checkpointed ?? values[2] ?? -1); + if (busy !== 0) return "busy"; + // SQLite returns -1/-1 when the database is not in WAL mode or there are no + // WAL frames to report. Otherwise FULL must have copied every frame. + if (log >= 0 && checkpointed >= 0 && checkpointed < log) return "busy"; + return "ok"; +} + +function measure(databasePath: string, db: Database, pageSize: number): CodexLogGuardCompactionMeasure { + const databaseBytes = (() => { + try { + const stat = statSync(databasePath); + return stat.isFile() ? stat.size : 0; + } catch { + return 0; + } + })(); + const walBytes = (() => { + try { + const stat = statSync(`${databasePath}-wal`); + return stat.isFile() ? stat.size : 0; + } catch { + return 0; + } + })(); + const pageCount = pragmaNumber(db, "PRAGMA page_count"); + const freelistPages = pragmaNumber(db, "PRAGMA freelist_count"); + return { + databaseBytes, + walBytes, + pageCount, + freelistPages, + reclaimableBytes: pageSize * freelistPages, + }; +} + +function runCompaction( + databasePath: string, + deps: CodexLogGuardMaintenanceDeps, +): CodexLogGuardCompactionResult { + let db: Database | undefined; + let probeOpen = false; + let reportBusyPartial: (() => CodexLogGuardCompactionResult) | undefined; + try { + const beforeOpenIdentity = databasePathIdentity(databasePath); + if (!beforeOpenIdentity) return { ok: false, error: "unsafe_path" }; + const openDatabase = deps.openDatabase + ?? ((path: string, flags: number) => new Database(path, flags)); + db = openDatabase(databasePath, sqliteConstants.SQLITE_OPEN_READWRITE); + // The path is user-writable foreign state. Re-check its regular-file, + // canonical-path and st_dev/st_ino identity immediately after SQLite opens + // it, before issuing any pragma or write-capable statement. + if (!databasePathStillMatches(databasePath, beforeOpenIdentity)) { + return { ok: false, error: "unsafe_path" }; + } + db.exec("PRAGMA busy_timeout = 0"); + + if (!exactCurrentSchema(db)) return { ok: false, error: "unsupported_schema" }; + if (pragmaNumber(db, "PRAGMA auto_vacuum") !== 2) { + return { ok: false, error: "auto_vacuum_not_incremental" }; + } + + const quickCheck = deps.quickCheck ?? defaultQuickCheck; + if (!quickCheckIsOk(quickCheck(db))) { + return { ok: false, error: "integrity_check_failed", phase: "before" }; + } + + // Confirm no SQLite writer can acquire the file before the first checkpoint. + // BEGIN IMMEDIATE is intentionally released before PRAGMA wal_checkpoint, + // which cannot run while this same connection holds a write transaction. + db.exec("BEGIN IMMEDIATE"); + probeOpen = true; + db.exec("ROLLBACK"); + probeOpen = false; + + if (checkpointFull(db) === "busy") return { ok: false, error: "busy" }; + + const pageSize = pragmaNumber(db, "PRAGMA page_size"); + const before = measure(databasePath, db, pageSize); + // Derived from the byte budgets AFTER reading the real page size, so the + // documented ~8 MiB batch / ~256 MiB run hold at any page size. Explicit + // page-count overrides still win, which is what the tests use. + const batchPages = Math.max(1, Math.floor(deps.batchPages ?? pagesForBytes(DEFAULT_BATCH_BYTES, pageSize))); + const maxPages = Math.max(batchPages, Math.floor(deps.maxPagesPerRun ?? pagesForBytes(DEFAULT_MAX_BYTES_PER_RUN, pageSize))); + let previousFreelist = before.freelistPages; + let pagesReclaimed = 0; + let iterations = 0; + let stopReason: CompactStopReason = previousFreelist === 0 ? "complete" : "page_budget"; + + const finish = (reason: CompactStopReason): CodexLogGuardCompactionResult => { + const after = measure(databasePath, db!, pageSize); + if (!quickCheckIsOk(quickCheck(db!))) { + return { ok: false, error: "integrity_check_failed", phase: "after" }; + } + const complete = after.freelistPages === 0; + // If SQLITE_BUSY is thrown after an incremental_vacuum commit but before the + // loop can sample freelist_count, the before/after measurements still capture + // that committed logical reclamation. Never under-report already-landed work. + const observedPagesReclaimed = Math.max( + pagesReclaimed, + Math.max(0, before.freelistPages - after.freelistPages), + ); + return { + ok: true, + report: { + pageSize, + before, + after, + pagesReclaimed: observedPagesReclaimed, + logicalBytesReclaimed: observedPagesReclaimed * pageSize, + physicalDatabaseBytesReclaimed: Math.max(0, before.databaseBytes - after.databaseBytes), + iterations, + complete, + stopReason: reason === "busy" ? "busy" : complete ? "complete" : reason, + integrity: { before: "ok", after: "ok" }, + }, + }; + }; + reportBusyPartial = () => iterations > 0 ? finish("busy") : { ok: false, error: "busy" }; + + while (previousFreelist > 0 && pagesReclaimed < maxPages && iterations < MAX_ITERATIONS) { + const pageBudget = Math.min(batchPages, maxPages - pagesReclaimed, previousFreelist); + if (pageBudget <= 0) { + stopReason = "page_budget"; + break; + } + const priorFreelist = previousFreelist; + db.exec(`PRAGMA incremental_vacuum(${pageBudget})`); + iterations += 1; + const checkpoint = checkpointFull(db); + const currentFreelist = pragmaNumber(db, "PRAGMA freelist_count"); + const reclaimed = Math.max(0, priorFreelist - currentFreelist); + pagesReclaimed += reclaimed; + previousFreelist = currentFreelist; + + // incremental_vacuum has already committed by this point. A busy FULL + // checkpoint is therefore a partial-success stop, not an atomic refusal. + if (checkpoint === "busy") return finish("busy"); + if (currentFreelist === 0) { + stopReason = "complete"; + break; + } + if (currentFreelist >= priorFreelist) { + stopReason = "no_progress"; + break; + } + stopReason = "page_budget"; + } + + if (previousFreelist > 0 && iterations >= MAX_ITERATIONS && stopReason !== "no_progress") { + // MAX_ITERATIONS is a bounded-work limit, not evidence that vacuum stalled. + stopReason = "page_budget"; + } + + // The preceding incremental-vacuum iterations checkpoint after every batch. + // One final FULL checkpoint backfills any remaining WAL frames before the + // final main-database measurement. FULL does not reset or shrink the WAL + // sidecar, so `after.walBytes` is an observational size, not reclaimed WAL. + if (checkpointFull(db) === "busy") { + return iterations > 0 ? finish("busy") : { ok: false, error: "busy" }; + } + return finish(stopReason); + } catch (error) { + if (probeOpen) { + try { db?.exec("ROLLBACK"); } catch { /* close releases it */ } + } + if (isSqliteBusy(error)) { + // Mirror the explicit busy exits. Once at least one vacuum batch completed, + // a later thrown busy is a partial-success stop rather than a pure refusal. + if (reportBusyPartial) { + try { return reportBusyPartial(); } catch { /* fall through to refusal */ } + } + return { ok: false, error: "busy" }; + } + return { ok: false, error: "database_error" }; + } finally { + try { db?.close(); } catch { /* maintenance already settled */ } + } +} + +export function compactCodexLogs( + deps: CodexLogGuardMaintenanceDeps = {}, +): CodexLogGuardCompactionResult { + const codexHome = deps.codexHome ?? getCodexHome(); + const inspection = inspectCodexLogs({ codexHome }); + const databasePath = resolveCodexLogsDbPath({ codexHome }); + if (inspection.capabilities.reclaim.state !== "supported") { + return { ok: false, error: "unsupported_schema" }; + } + if (!databasePathIsSafe(databasePath)) return { ok: false, error: "unsafe_path" }; + + const checkProcesses = deps.processCheck ?? listRunningCodexProcesses; + const firstRefusal = processRefusal(checkProcesses()); + if (firstRefusal) return { ok: false, error: firstRefusal }; + + const withLock = deps.withLock ?? withCodexLogGuardLock; + let locked: CodexLogGuardLockOutcome; + try { + locked = withLock(codexHome, databasePath, () => { + const secondRefusal = processRefusal(checkProcesses()); + if (secondRefusal) return { ok: false as const, error: secondRefusal }; + return runCompaction(databasePath, deps); + }); + } catch { + return { ok: false, error: "database_error" }; + } + + if (locked.kind === "unavailable") { + return { + ok: false, + error: locked.reason === "busy" + ? "busy" + : locked.reason === "unsafe-path" ? "unsafe_path" : "database_error", + }; + } + return locked.value; +} diff --git a/src/server/management/context.ts b/src/server/management/context.ts index 190d7b9b4..9f01f213a 100644 --- a/src/server/management/context.ts +++ b/src/server/management/context.ts @@ -1,6 +1,7 @@ import type { OcxConfig } from "../../types"; import type { NativeProfileApiDeps } from "../../codex/native-profile-api"; import type { CodexLogGuardProtectionDeps } from "../../codex/log-guard/protection"; +import type { CodexLogGuardMaintenanceDeps } from "../../codex/log-guard/maintenance"; import type { StartupHealth } from "../../codex/autostart-health"; import type { StartupInstallAction } from "../startup-action-control"; import type { ManagementPrincipal } from "../management-auth"; @@ -78,6 +79,12 @@ export interface ManagementApiDeps { * or create lock/config state outside the fixture. */ codexLogGuardProtectionDeps?: CodexLogGuardProtectionDeps; + /** + * Log Guard maintenance seam. Production reuses the same fail-closed process + * enumerator and L namespace as Protect; route tests keep all maintenance + * state inside their temporary Codex home. + */ + codexLogGuardMaintenanceDeps?: CodexLogGuardMaintenanceDeps; } diff --git a/src/server/management/storage-log-guard-routes.ts b/src/server/management/storage-log-guard-routes.ts index 57efd10aa..324e746bd 100644 --- a/src/server/management/storage-log-guard-routes.ts +++ b/src/server/management/storage-log-guard-routes.ts @@ -1,4 +1,8 @@ import { resolveCodexHomeDir } from "../../codex/home"; +import { + compactCodexLogs, + type CodexLogGuardCompactionResult, +} from "../../codex/log-guard/maintenance"; import { getCodexLogGuardProtectionStatus, protectCodexLogs, @@ -38,6 +42,23 @@ function mutationStatus(result: CodexLogGuardMutationResult): number { } } +function compactStatus(result: CodexLogGuardCompactionResult): number { + if (result.ok) return 200; + switch (result.error) { + case "process_enumeration_failed": + return 503; + case "codex_running": + case "busy": + case "unsupported_schema": + case "auto_vacuum_not_incremental": + case "unsafe_path": + case "integrity_check_failed": + return 409; + case "database_error": + return 500; + } +} + function mutationResponse( result: CodexLogGuardMutationResult, ctx: ManagementContext, @@ -47,6 +68,23 @@ function mutationResponse( : jsonResponse({ error: result.error }, mutationStatus(result), ctx.req, ctx.config); } +function compactResponse( + result: CodexLogGuardCompactionResult, + ctx: ManagementContext, +): Response { + if (result.ok) { + return jsonResponse({ report: result.report }, 200, ctx.req, ctx.config); + } + return jsonResponse( + result.error === "integrity_check_failed" + ? { error: result.error, phase: result.phase } + : { error: result.error }, + compactStatus(result), + ctx.req, + ctx.config, + ); +} + async function readProtectMode(ctx: ManagementContext): Promise<"compat" | "quiet" | Response> { let body: unknown; try { @@ -66,7 +104,7 @@ async function readProtectMode(ctx: ManagementContext): Promise<"compat" | "quie return mode; } -/** Codex Log Guard diagnostics and explicit, opt-in protection mutations. */ +/** Codex Log Guard diagnostics plus explicit protection and maintenance mutations. */ export async function handleStorageLogGuardRoutes(ctx: ManagementContext): Promise { const { req, url, config, deps } = ctx; const protectionDeps = deps.codexLogGuardProtectionDeps; @@ -104,6 +142,11 @@ export async function handleStorageLogGuardRoutes(ctx: ManagementContext): Promi return mutationResponse(repairCodexLogGuardProtection(protectionDeps), ctx); } + if (url.pathname === "/api/storage/codex-logs/compact") { + if (req.method !== "POST") return null; + return compactResponse(compactCodexLogs(deps.codexLogGuardMaintenanceDeps), ctx); + } + if (url.pathname !== "/api/storage" || req.method !== "GET") return null; // Keep the existing CODEX_HOME scan as the primary storage contract. The Log Guard diff --git a/tests/api-codex-log-guard-compact.test.ts b/tests/api-codex-log-guard-compact.test.ts new file mode 100644 index 000000000..a4a234182 --- /dev/null +++ b/tests/api-codex-log-guard-compact.test.ts @@ -0,0 +1,122 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import type { CodexLogGuardMaintenanceDeps } from "../src/codex/log-guard/maintenance"; +import { handleManagementAPI } from "../src/server/management-api"; +import type { OcxConfig } from "../src/types"; +import { ManagementRequest } from "./helpers/management-auth"; + +const roots: string[] = []; +const originalCodexHome = process.env.CODEX_HOME; + +function createLogsDb(path: string): void { + const db = new Database(path); + db.exec(` + PRAGMA auto_vacuum=INCREMENTAL; + PRAGMA journal_mode=WAL; + CREATE TABLE logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + ts INTEGER NOT NULL, + ts_nanos INTEGER NOT NULL, + level TEXT NOT NULL, + target TEXT NOT NULL, + feedback_log_body TEXT, + module_path TEXT, + file TEXT, + line INTEGER, + thread_id TEXT, + process_uuid TEXT, + estimated_bytes INTEGER NOT NULL DEFAULT 0 + ); + CREATE INDEX idx_logs_ts ON logs(ts DESC, ts_nanos DESC, id DESC); + CREATE INDEX idx_logs_thread_id ON logs(thread_id); + CREATE INDEX idx_logs_thread_id_ts ON logs(thread_id, ts DESC, ts_nanos DESC, id DESC); + CREATE INDEX idx_logs_process_uuid_threadless_ts + ON logs(process_uuid, ts DESC, ts_nanos DESC, id DESC) + WHERE thread_id IS NULL; + CREATE TABLE reclaim_fixture (id INTEGER PRIMARY KEY, body BLOB NOT NULL); + `); + const insert = db.query("INSERT INTO reclaim_fixture (id, body) VALUES (?, zeroblob(8192))"); + for (let i = 0; i < 120; i += 1) insert.run(i + 1); + db.exec("DELETE FROM reclaim_fixture WHERE id <= 100; PRAGMA wal_checkpoint(FULL)"); + db.close(); +} + +function fixture(): { deps: CodexLogGuardMaintenanceDeps } { + const root = mkdtempSync(join(tmpdir(), "ocx-log-guard-api-compact-")); + roots.push(root); + const codexHome = join(root, "codex-home"); + mkdirSync(codexHome); + writeFileSync(join(codexHome, "config.toml"), ""); + createLogsDb(join(codexHome, "logs_2.sqlite")); + process.env.CODEX_HOME = codexHome; + return { + deps: { + codexHome, + processCheck: () => ({ state: "ok", processes: [] }), + withLock: (_home: string, _database: string, work: () => T) => ({ kind: "completed", value: work() }), + }, + }; +} + +function config(): OcxConfig { + return { port: 0, defaultProvider: "openai", providers: {} } as OcxConfig; +} + +async function request(path: string, deps: CodexLogGuardMaintenanceDeps): Promise { + const req = new ManagementRequest(`http://localhost${path}`, { method: "POST" }); + const response = await handleManagementAPI( + req, + new URL(req.url), + config(), + { codexLogGuardMaintenanceDeps: deps }, + ); + expect(response).not.toBeNull(); + return response!; +} + +afterEach(() => { + if (originalCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = originalCodexHome; + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +describe("Codex Log Guard compact management API", () => { + test("POST compact returns path-private before/after maintenance metrics", async () => { + const { deps } = fixture(); + const response = await request("/api/storage/codex-logs/compact", deps); + expect(response.status).toBe(200); + const body = await response.json(); + expect(body.report.before.freelistPages).toBeGreaterThan(0); + expect(body.report.after.freelistPages).toBeLessThan(body.report.before.freelistPages); + expect(body.report.pagesReclaimed).toBeGreaterThan(0); + expect(body.report.integrity).toEqual({ before: "ok", after: "ok" }); + expect(body.report).not.toHaveProperty("databasePath"); + expect(JSON.stringify(body)).not.toContain(deps.codexHome!); + }); + + test("GET compact is not a mutation alias", async () => { + const { deps } = fixture(); + const req = new ManagementRequest("http://localhost/api/storage/codex-logs/compact", { method: "GET" }); + const response = await handleManagementAPI( + req, + new URL(req.url), + config(), + { codexLogGuardMaintenanceDeps: deps }, + ); + expect(response).toBeNull(); + }); + + test("integrity refusal is surfaced without pretending maintenance succeeded", async () => { + const { deps } = fixture(); + const response = await request("/api/storage/codex-logs/compact", { + ...deps, + quickCheck: () => ["synthetic corruption"], + }); + expect(response.status).toBe(409); + expect(await response.json()).toEqual({ error: "integrity_check_failed", phase: "before" }); + }); +}); diff --git a/tests/cli-codex-log-guard-compact.test.ts b/tests/cli-codex-log-guard-compact.test.ts new file mode 100644 index 000000000..2a6be54bc --- /dev/null +++ b/tests/cli-codex-log-guard-compact.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, test } from "bun:test"; + +import { handleObserveCommand } from "../src/cli/observe"; + +describe("Codex Log Guard compact CLI", () => { + test("compact POSTs to the dedicated maintenance endpoint", async () => { + const seen: Array<{ url: string; method: string }> = []; + const fetchImpl: typeof fetch = async (input, init) => { + seen.push({ url: String(input), method: init?.method ?? "GET" }); + return new Response(JSON.stringify({ + report: { + before: { freelistPages: 10, reclaimableBytes: 40960 }, + after: { freelistPages: 2, reclaimableBytes: 8192 }, + pagesReclaimed: 8, + physicalDatabaseBytesReclaimed: 32768, + complete: false, + stopReason: "page_budget", + }, + }), { status: 200, headers: { "content-type": "application/json" } }); + }; + const originalLog = console.log; + console.log = () => {}; + try { + expect(await handleObserveCommand( + ["storage", "codex-logs", "compact", "--json"], + { baseUrl: "http://runtime", fetchImpl }, + )).toBe(0); + expect(seen).toEqual([{ + url: "http://runtime/api/storage/codex-logs/compact", + method: "POST", + }]); + } finally { + console.log = originalLog; + } + }); + + test("compact rejects protection-only mode flags before making a request", async () => { + let calls = 0; + const fetchImpl: typeof fetch = async () => { + calls += 1; + return new Response("{}", { status: 200 }); + }; + const originalError = console.error; + console.error = () => {}; + try { + expect(await handleObserveCommand( + ["storage", "codex-logs", "compact", "--mode", "quiet"], + { baseUrl: "http://runtime", fetchImpl }, + )).not.toBe(0); + expect(calls).toBe(0); + } finally { + console.error = originalError; + } + }); +}); diff --git a/tests/codex-log-guard-maintenance-coderabbit.test.ts b/tests/codex-log-guard-maintenance-coderabbit.test.ts new file mode 100644 index 000000000..be4684ee5 --- /dev/null +++ b/tests/codex-log-guard-maintenance-coderabbit.test.ts @@ -0,0 +1,260 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { + copyFileSync, + mkdirSync, + mkdtempSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + compactCodexLogs, + type CodexLogGuardMaintenanceDeps, +} from "../src/codex/log-guard/maintenance"; + +const roots: string[] = []; + +function createLogsSchema(db: Database): void { + db.exec(` + CREATE TABLE logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + ts INTEGER NOT NULL, + ts_nanos INTEGER NOT NULL, + level TEXT NOT NULL, + target TEXT NOT NULL, + feedback_log_body TEXT, + module_path TEXT, + file TEXT, + line INTEGER, + thread_id TEXT, + process_uuid TEXT, + estimated_bytes INTEGER NOT NULL DEFAULT 0 + ); + CREATE INDEX idx_logs_ts ON logs(ts DESC, ts_nanos DESC, id DESC); + CREATE INDEX idx_logs_thread_id ON logs(thread_id); + CREATE INDEX idx_logs_thread_id_ts ON logs(thread_id, ts DESC, ts_nanos DESC, id DESC); + CREATE INDEX idx_logs_process_uuid_threadless_ts + ON logs(process_uuid, ts DESC, ts_nanos DESC, id DESC) + WHERE thread_id IS NULL; + `); +} + +function fixture(): { codexHome: string; databasePath: string } { + const root = mkdtempSync(join(tmpdir(), "ocx-log-guard-cr-reclaim-")); + roots.push(root); + const codexHome = join(root, "codex-home"); + mkdirSync(codexHome); + writeFileSync(join(codexHome, "config.toml"), ""); + const databasePath = join(codexHome, "logs_2.sqlite"); + const db = new Database(databasePath); + db.exec("PRAGMA auto_vacuum=INCREMENTAL"); + db.exec("PRAGMA journal_mode=WAL"); + createLogsSchema(db); + db.exec("CREATE TABLE reclaim_fixture (id INTEGER PRIMARY KEY, body BLOB NOT NULL)"); + const fill = db.query("INSERT INTO reclaim_fixture (id, body) VALUES (?, zeroblob(8192))"); + for (let i = 0; i < 220; i += 1) fill.run(i + 1); + db.exec("DELETE FROM reclaim_fixture WHERE id <= 200"); + db.exec("PRAGMA wal_checkpoint(FULL)"); + db.close(); + return { codexHome, databasePath }; +} + +function scalar(path: string, pragma: string): number { + const db = new Database(path, { readonly: true }); + try { + const row = db.query, []>(pragma).get(); + if (!row) throw new Error(`missing ${pragma}`); + return Number(Object.values(row)[0]); + } finally { + db.close(); + } +} + +function deps( + codexHome: string, + extra: Partial = {}, +): CodexLogGuardMaintenanceDeps { + return { + codexHome, + processCheck: () => ({ state: "ok" as const, processes: [] }), + withLock: (_home: string, _database: string, work: () => T) => ({ + kind: "completed" as const, + value: work(), + }), + ...extra, + }; +} + +afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +describe("CodeRabbit Log Guard reclaim regressions", () => { + test("rejects a regular-file replacement between the pre-open check and SQLite open", () => { + const { codexHome, databasePath } = fixture(); + const backup = `${databasePath}.original`; + let opened = false; + + const result = compactCodexLogs(deps(codexHome, { + openDatabase: (path: string, flags: number) => { + opened = true; + renameSync(path, backup); + copyFileSync(backup, path); // same bytes, different filesystem identity + return new Database(path, flags); + }, + })); + + expect(opened).toBe(true); + expect(result).toEqual({ ok: false, error: "unsafe_path" }); + }); + + test("rechecks Codex processes after acquiring the Log Guard lock", () => { + const { codexHome } = fixture(); + let checks = 0; + const result = compactCodexLogs(deps(codexHome, { + processCheck: () => { + checks += 1; + return checks === 1 + ? { state: "ok" as const, processes: [] } + : { state: "ok" as const, processes: [{ pid: 42, commandLine: "codex exec" }] }; + }, + })); + + expect(result).toEqual({ ok: false, error: "codex_running" }); + expect(checks).toBe(2); + }); + + test("classifies continuous progress stopped by MAX_ITERATIONS as bounded work", () => { + const { codexHome } = fixture(); + const result = compactCodexLogs(deps(codexHome, { + batchPages: 1, + maxPagesPerRun: 100_000, + })); + + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.report.before.freelistPages).toBeGreaterThan(64); + expect(result.report.iterations).toBe(64); + expect(result.report.pagesReclaimed).toBeGreaterThan(0); + expect(result.report.after.freelistPages).toBeGreaterThan(0); + expect(result.report.stopReason).toBe("page_budget"); + }); + + test("refuses compaction when a WAL reader prevents the initial FULL checkpoint", () => { + const { codexHome, databasePath } = fixture(); + const reader = new Database(databasePath, { readonly: true }); + let result; + const beforeFreelist = scalar(databasePath, "PRAGMA freelist_count"); + try { + reader.exec("BEGIN"); + reader.query("SELECT count(*) AS n FROM logs").get(); + + const writer = new Database(databasePath); + try { + writer.query( + "INSERT INTO logs (ts, ts_nanos, level, target, feedback_log_body, estimated_bytes) VALUES (1, 0, 'INFO', 'test', NULL, 1)", + ).run(); + } finally { + writer.close(); + } + + result = compactCodexLogs(deps(codexHome)); + expect(result).toEqual({ ok: false, error: "busy" }); + expect(scalar(databasePath, "PRAGMA freelist_count")).toBe(beforeFreelist); + } finally { + try { reader.exec("ROLLBACK"); } catch { /* close releases the read transaction */ } + reader.close(); + } + }); + + test("reports a busy checkpoint as partial success after a vacuum batch commits", () => { + const { codexHome, databasePath } = fixture(); + const testDeps = deps(codexHome); + let reader: Database | undefined; + let getterReads = 0; + + // Injection point: runCompaction reads batchPages after its initial FULL + // checkpoint and before the first incremental-vacuum batch. Creating the + // blocking reader here makes only the mid-loop checkpoint busy. + Object.defineProperty(testDeps, "batchPages", { + enumerable: true, + get: () => { + getterReads += 1; + if (!reader) { + reader = new Database(databasePath, { readonly: true }); + reader.exec("BEGIN"); + reader.query("SELECT count(*) AS n FROM logs").get(); + const writer = new Database(databasePath); + try { + writer.query( + "INSERT INTO logs (ts, ts_nanos, level, target, feedback_log_body, estimated_bytes) VALUES (2, 0, 'INFO', 'after-initial-checkpoint', NULL, 1)", + ).run(); + } finally { + writer.close(); + } + } + return 1; + }, + }); + + try { + const result = compactCodexLogs(testDeps); + expect(getterReads).toBe(1); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.report.iterations).toBe(1); + expect(result.report.pagesReclaimed).toBeGreaterThan(0); + expect(result.report.stopReason).toBe("busy"); + expect(result.report.after.freelistPages).toBeLessThan(result.report.before.freelistPages); + } finally { + if (reader) { + try { reader.exec("ROLLBACK"); } catch { /* close releases the read transaction */ } + reader.close(); + } + } + }); + + test("reports thrown SQLITE_BUSY as partial success after an earlier batch commits", () => { + const { codexHome, databasePath } = fixture(); + let vacuumCalls = 0; + + const result = compactCodexLogs(deps(codexHome, { + batchPages: 1, + maxPagesPerRun: 100_000, + openDatabase: (path: string, flags: number) => { + const inner = new Database(path, flags); + return new Proxy(inner, { + get(target, property) { + if (property === "exec") { + return (sql: string) => { + if (/^PRAGMA incremental_vacuum\(/.test(sql)) { + vacuumCalls += 1; + if (vacuumCalls === 2) { + const error = new Error("database is locked") as Error & { code?: string }; + error.code = "SQLITE_BUSY"; + throw error; + } + } + return target.exec(sql); + }; + } + const value = Reflect.get(target, property, target); + return typeof value === "function" ? value.bind(target) : value; + }, + }) as Database; + }, + })); + + expect(vacuumCalls).toBe(2); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.report.iterations).toBe(1); + expect(result.report.pagesReclaimed).toBeGreaterThan(0); + expect(result.report.stopReason).toBe("busy"); + expect(result.report.after.freelistPages).toBeLessThan(result.report.before.freelistPages); + }); +}); diff --git a/tests/codex-log-guard-maintenance.test.ts b/tests/codex-log-guard-maintenance.test.ts new file mode 100644 index 000000000..047827c01 --- /dev/null +++ b/tests/codex-log-guard-maintenance.test.ts @@ -0,0 +1,287 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { mkdirSync, mkdtempSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +const roots: string[] = []; + +function makeRoot(): string { + const root = mkdtempSync(join(tmpdir(), "ocx-log-guard-reclaim-")); + roots.push(root); + return root; +} + +function createLogsSchema(db: Database): void { + db.exec(` + CREATE TABLE logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + ts INTEGER NOT NULL, + ts_nanos INTEGER NOT NULL, + level TEXT NOT NULL, + target TEXT NOT NULL, + feedback_log_body TEXT, + module_path TEXT, + file TEXT, + line INTEGER, + thread_id TEXT, + process_uuid TEXT, + estimated_bytes INTEGER NOT NULL DEFAULT 0 + ); + CREATE INDEX idx_logs_ts ON logs(ts DESC, ts_nanos DESC, id DESC); + CREATE INDEX idx_logs_thread_id ON logs(thread_id); + CREATE INDEX idx_logs_thread_id_ts ON logs(thread_id, ts DESC, ts_nanos DESC, id DESC); + CREATE INDEX idx_logs_process_uuid_threadless_ts + ON logs(process_uuid, ts DESC, ts_nanos DESC, id DESC) + WHERE thread_id IS NULL; + `); +} + +function fixture(options: { incremental?: boolean; withFreelist?: boolean } = {}) { + const root = makeRoot(); + const codexHome = join(root, "codex-home"); + mkdirSync(codexHome); + writeFileSync(join(codexHome, "config.toml"), ""); + const databasePath = join(codexHome, "logs_2.sqlite"); + const db = new Database(databasePath); + if (options.incremental !== false) db.exec("PRAGMA auto_vacuum=INCREMENTAL"); + db.exec("PRAGMA journal_mode=WAL"); + createLogsSchema(db); + db.exec("CREATE TABLE reclaim_fixture (id INTEGER PRIMARY KEY, body BLOB NOT NULL)"); + const logInsert = db.query( + "INSERT INTO logs (ts, ts_nanos, level, target, feedback_log_body, estimated_bytes) VALUES (?, 0, ?, ?, ?, ?)", + ); + for (let i = 0; i < 12; i += 1) { + logInsert.run(i + 1, i % 2 === 0 ? "INFO" : "TRACE", `target-${i % 3}`, `PRIVATE-${i}`, 32 + i); + } + const fill = db.query("INSERT INTO reclaim_fixture (id, body) VALUES (?, zeroblob(8192))"); + for (let i = 0; i < 180; i += 1) fill.run(i + 1); + if (options.withFreelist !== false) { + db.exec("DELETE FROM reclaim_fixture WHERE id <= 150"); + } + db.exec("PRAGMA wal_checkpoint(FULL)"); + db.close(); + return { codexHome, databasePath }; +} + +function scalar(db: Database, pragma: string): number { + const row = db.query, []>(pragma).get(); + if (!row) throw new Error(`no row for ${pragma}`); + return Number(Object.values(row)[0]); +} + +function logicalSnapshot(path: string) { + const db = new Database(path, { readonly: true }); + try { + return { + logs: db.query("SELECT * FROM logs ORDER BY id").all(), + fixture: db.query("SELECT id, length(body) AS bytes FROM reclaim_fixture ORDER BY id").all(), + triggers: db.query("SELECT name, sql FROM sqlite_master WHERE type='trigger' ORDER BY name").all(), + }; + } finally { + db.close(); + } +} + +function testDeps(codexHome: string, overrides: Record = {}) { + return { + codexHome, + processCheck: () => ({ state: "ok" as const, processes: [] }), + withLock: (_home: string, _database: string, work: () => T) => ({ + kind: "completed" as const, + value: work(), + }), + ...overrides, + }; +} + +afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +describe("Codex Log Guard reclaim", () => { + test("incrementally reclaims freelist pages while preserving every logical row and trigger", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + + const { codexHome, databasePath } = fixture(); + const db = new Database(databasePath); + db.exec("CREATE TRIGGER user_trigger BEFORE INSERT ON logs BEGIN SELECT 1; END;"); + db.close(); + const beforeLogical = logicalSnapshot(databasePath); + const beforeBytes = statSync(databasePath).size; + + const result = mod.compactCodexLogs(testDeps(codexHome)); + expect(result.ok).toBe(true); + if (!result.ok) return; + + expect(result.report.before.freelistPages).toBeGreaterThan(0); + expect(result.report.after.freelistPages).toBeLessThan(result.report.before.freelistPages); + expect(result.report.pagesReclaimed).toBeGreaterThan(0); + expect(result.report.after.databaseBytes).toBeLessThanOrEqual(beforeBytes); + expect(result.report.integrity).toEqual({ before: "ok", after: "ok" }); + expect(logicalSnapshot(databasePath)).toEqual(beforeLogical); + }); + + test("is a safe no-op when there is nothing reclaimable", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + + const { codexHome } = fixture({ withFreelist: false }); + const result = mod.compactCodexLogs(testDeps(codexHome)); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.report.pagesReclaimed).toBe(0); + expect(result.report.complete).toBe(true); + }); + + test("refuses databases that are not already auto_vacuum=INCREMENTAL", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + + const { codexHome, databasePath } = fixture({ incremental: false }); + const db = new Database(databasePath, { readonly: true }); + expect(scalar(db, "PRAGMA auto_vacuum")).not.toBe(2); + db.close(); + + expect(mod.compactCodexLogs(testDeps(codexHome))).toEqual({ + ok: false, + error: "auto_vacuum_not_incremental", + }); + }); + + test("refuses unknown log schemas before maintenance", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + + const { codexHome, databasePath } = fixture(); + const db = new Database(databasePath); + db.exec("ALTER TABLE logs ADD COLUMN future_field TEXT"); + db.close(); + + expect(mod.compactCodexLogs(testDeps(codexHome))).toEqual({ + ok: false, + error: "unsupported_schema", + }); + }); + + test("fails closed when Codex is running or process enumeration is uncertain", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + + const running = fixture(); + expect(mod.compactCodexLogs(testDeps(running.codexHome, { + processCheck: () => ({ state: "ok" as const, processes: [{ pid: 42, commandLine: "codex exec" }] }), + }))).toEqual({ ok: false, error: "codex_running" }); + + const unknown = fixture(); + expect(mod.compactCodexLogs(testDeps(unknown.codexHome, { + processCheck: () => ({ state: "unknown" as const, reason: "enumeration_failed" as const }), + }))).toEqual({ ok: false, error: "process_enumeration_failed" }); + }); + + test("fails closed when the Log Guard lock is busy", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + const { codexHome } = fixture(); + expect(mod.compactCodexLogs(testDeps(codexHome, { + withLock: () => ({ kind: "unavailable" as const, reason: "busy" as const }), + }))).toEqual({ ok: false, error: "busy" }); + }); + + test("requires quick_check before changing any pages", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + const { codexHome, databasePath } = fixture(); + const before = logicalSnapshot(databasePath); + const db = new Database(databasePath, { readonly: true }); + const freelist = scalar(db, "PRAGMA freelist_count"); + db.close(); + + const result = mod.compactCodexLogs(testDeps(codexHome, { + quickCheck: () => ["synthetic corruption"], + })); + expect(result).toEqual({ ok: false, error: "integrity_check_failed", phase: "before" }); + expect(logicalSnapshot(databasePath)).toEqual(before); + const afterDb = new Database(databasePath, { readonly: true }); + expect(scalar(afterDb, "PRAGMA freelist_count")).toBe(freelist); + afterDb.close(); + }); + + test("reports a post-maintenance quick_check failure explicitly", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + const { codexHome } = fixture(); + let checks = 0; + const result = mod.compactCodexLogs(testDeps(codexHome, { + quickCheck: () => { + checks += 1; + return checks === 1 ? ["ok"] : ["synthetic post failure"]; + }, + })); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.error).toBe("integrity_check_failed"); + expect(result.phase).toBe("after"); + }); + + test("stops at the per-run page budget and reports remaining work", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + expect(mod).not.toBeNull(); + if (!mod) return; + const { codexHome } = fixture(); + + const result = mod.compactCodexLogs(testDeps(codexHome, { + batchPages: 1, + maxPagesPerRun: 2, + })); + expect(result.ok).toBe(true); + if (!result.ok) return; + expect(result.report.pagesReclaimed).toBeLessThanOrEqual(2); + expect(result.report.complete).toBe(false); + expect(result.report.stopReason).toBe("page_budget"); + expect(result.report.after.freelistPages).toBeGreaterThan(0); + }); + test("derives batch budgets from the real page size, not a fixed page count", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + if (!mod) return; + // The guide promises ~8 MiB batches and ~256 MiB per run, converted with the + // database's page size. Fixed page counts meant something different at every + // page size: at 4 KiB pages the old 512/8192 was 2 MiB/32 MiB. + const { codexHome } = fixture({ incremental: true, withFreelist: true }); + const result = mod.compactCodexLogs(testDeps(codexHome)); + expect(result.ok).toBe(true); + if (!result.ok) return; + + const pageSize = result.report.pageSize; + expect(pageSize).toBeGreaterThan(0); + // 8 MiB / pageSize, and 256 MiB / pageSize, both at least one page. + const expectedBatch = Math.max(1, Math.floor((8 * 1024 * 1024) / pageSize)); + expect(expectedBatch * pageSize).toBeGreaterThanOrEqual(4 * 1024 * 1024); + }); + + test("reports logicalBytesReclaimed alongside the physical figure", async () => { + const mod = await import("../src/codex/log-guard/maintenance").catch(() => null); + if (!mod) return; + // Documented before it existed. It is deliberately distinct from the + // physical figure: an incremental vacuum can return pages to the free list + // without the file shrinking, so logical progress with zero physical + // shrinkage is normal rather than a failed run. + const { codexHome } = fixture({ incremental: true, withFreelist: true }); + const result = mod.compactCodexLogs(testDeps(codexHome)); + expect(result.ok).toBe(true); + if (!result.ok) return; + + expect(result.report.logicalBytesReclaimed) + .toBe(result.report.pagesReclaimed * result.report.pageSize); + expect(typeof result.report.physicalDatabaseBytesReclaimed).toBe("number"); + }); +});