Persist Electron logs and recover from renderer death across all window views#2428
Merged
Conversation
…h page Plans two improvements from the blank-screen-after-sleep diagnosis (Sentry bc6d78e9): a new rotated/gzipped electron.log capturing Electron main-process console output and uncaught exceptions (uploaded with bug reports, alongside rotation for minds.log), and a render-process-gone handler that shows a Chrome-style crash page with manual Reload and Report-a-bug buttons. Co-authored-by: Sculptor <sculptor@imbue.com>
…page
Implements blueprint/electron-log-and-crash-page.
Logging & rotation:
- New electron/logger.js tees the Electron main process's console.log/warn/error
into ~/.minds/logs/electron.log (timestamped) and records uncaught exceptions
/ unhandled rejections, without altering Sentry/exit behavior
(uncaughtExceptionMonitor). Initialized first in main.js.
- New electron/log-rotation.js: shared 100MB/keep-10 rotation that gzips each
rotated file; used by both electron.log and (via backend.js) minds.log, which
was previously an unbounded append stream.
Crash page:
- main.js handles content-view render-process-gone by showing a local
crashed.html ("Aw, Snap!"-style) with the crash reason, a manual Reload
button (no auto-reload; crash-loop-safe), and a Report-a-bug button. Navigation
is deferred (electron#19887); clean-exit and error-takeover teardown are ignored.
- content-relay-preload.js relays the new minds:reload-crashed-view message.
Upload plumbing:
- sentry.js attaches both minds.log and electron.log (plus newest jsonl) to
backend-down reports.
- utils/sentry/core.py splits the mislabeled *.log group into backend_logs /
electron_logs (+ gzipped rotated groups) with accurate names.
Co-authored-by: Sculptor <sculptor@imbue.com>
Problem: apps/minds/electron/logger.js wraps console.log/warn/error to tee into electron.log via writeLine -> logStream.write -> log-rotation.rotateIfNeeded. When rotation fails, rotateIfNeeded does not throw; it calls console.warn, which is the wrapped console, re-entering writeLine -> logStream.write -> rotateIfNeeded (size is never reset on failure, so it stays over threshold) -> console.warn -> ... unbounded recursion -> stack overflow crashing the Electron main process. The existing try/catch could not stop it because rotateIfNeeded swallows its own error instead of throwing. Fix: add a module-level reentrancy guard (isWritingLine) so a console.* call emitted while a log line is being written short-circuits writeLine instead of re-entering the rotating stream. The nested console.* still reaches stdout/stderr via the wrapper's original(...), so the rotation warning is not lost.
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
Match the codebase's existing log-writing precedent: the minds.log sink (the only streaming-log sink in electron/) used an async fs.createWriteStream, and the only *Sync file write is the tiny one-shot window-state.json snapshot. Convert createRotatingLogStream from synchronous fd writes back to an async WriteStream so writes never block the Electron main thread. Rotation stays correct: the stream is end()ed (flushing all buffered writes + closing the fd) BEFORE the rename, so the background gzip can't read a partially-flushed file; lines that arrive mid-rotation are buffered and replayed into the fresh stream. Also resets tracked size after a failed rename (bounded retry, no tight rotation loop) and swallows async stream 'error' events to stderr so a write error can't crash the main process. Unit tests updated to poll for the async flush. Document in logger.js why we hand-roll rather than use the electron-log library: we need on-disk gzipped rotations under fixed exact names so the Sentry LogAttachmentGroup globs upload the current file plus newest rotation with no extra transform, which the shared log-rotation.js provides while mirroring the Python jsonl sink's 100MB/keep-10 scheme. Co-authored-by: Sculptor <sculptor@imbue.com>
Problem: logger.js's isWritingLine guard comment referenced rotateIfNeeded (renamed to rotate in the same commit) and described a synchronous writeLine->logStream.write->console.warn->writeLine recursion that can no longer occur -- log-rotation.js now defers its rename-failure console.warn into an async WriteStream end() callback, where isRotating buffers it instead of re-entering writeLine on the same tick. Fix: rewrite the comment to state the guard's remaining defensive purpose (preventing a synchronous re-entry into writeLine) without naming the removed function or the now-impossible rotation example. Comment-only; no behavior change.
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
Problem: compressRotatedFile in apps/minds/electron/log-rotation.js piped the raw rotated file through gzip to a .gz destination with `source.pipe(gzip).pipe(dest)` and per-stream error handlers that only logged. Node's .pipe() does not destroy the other streams when one errors, so any read/gzip/write failure mid-compression leaked the still-open read and write file descriptors -- unbounded over the long-lived Electron main process's many rotations. Fix: use stream.pipeline, which destroys every stream (freeing all fds) on both success and error. The success path still unlinks the raw file and prunes old rotations; the error path still logs the same best-effort warning and leaves the raw rotation on disk.
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
…y reporting Broaden the blueprint plan beyond the content-view crash page to cover the window's other two renderers -- a miniaturized in-titlebar error strip with a manual Reload for the chrome view, and a silent warm reload for the overlay view -- and to close the observability gap by reporting abnormal renderer deaths (crashed/oom, labeled by view) to Sentry while deliberately not reporting sleep/external kills (killed). Updates the dev changelog to match. Co-authored-by: Sculptor <sculptor@imbue.com>
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
…to Sentry Extend the content-view crash-page recovery to the window's other two per-window views, which run in separate renderer processes and can die independently over a long sleep: - chrome (titlebar) view: on render-process-gone (non-clean-exit) show a compact local chrome-crashed.html strip anchored to the ~38px visible titlebar band, with a manual Reload button (reload-chrome IPC -> reload /_chrome, re-primed from the cached chrome state). The workspace content view is left running. Manual reload for loop-safety, matching the content crash page. - overlay/menu host view: on render-process-gone silently reload the warm host (it's hidden when idle), so the next sidebar/inbox/help open is fresh. Also close the observability gap that let the original blank-screen-after- sleep incident go unreported: broaden the @sentry/electron childProcess integration to capture 'oom' renderer deaths as events (labeled by view via getRendererName). 'crashed' is intentionally left out -- it already produces a minidump via SentryMinidump, so adding it would double-report -- and 'killed' stays breadcrumb-only (OS/sleep reaping is unactionable; it remains in electron.log). Co-authored-by: Sculptor <sculptor@imbue.com>
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
Replace the emoji glyph on the content-view crash page with the distressed Minds head (X-eye) logo, switch from the maroon takeover surface to a white background with black text, and update the copy to a 'Bummer' heading. Co-authored-by: Sculptor <sculptor@imbue.com>
Drop the filled brand-color background rect (and the rounded-tile framing) and recolor the head logo to a black-on-white negative so it sits cleanly on the white crash surface. Co-authored-by: Sculptor <sculptor@imbue.com>
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
chrome-crashed.html set -webkit-app-region: drag on the whole <body>, but the chrome view is a full-window underlay. macOS unions drag regions across all views in a window, so a body-wide drag region turned the entire window into a drag surface -- including the content view's crash-page Reload button when both renderers died at once, swallowing its clicks until the titlebar was reloaded (which dropped the region). Scope drag to the 38px .bar strip only, which never overlaps the content view, so both crash pages' Reload buttons are clickable independently. Co-authored-by: Sculptor <sculptor@imbue.com>
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
gnguralnick
changed the title
gnguralnick
commented
Jul 13, 2026
| // open-modal state and silently reload the warm host, so the next sidebar / inbox / | ||
| // help open lands on a fresh page instead of a dead one. Defer the reload a tick | ||
| // (never navigate synchronously in this handler -- electron#19887). | ||
| modal.webContents.on('render-process-gone', (_e, details) => { |
Contributor
Author
There was a problem hiding this comment.
i think it's reasonable for the modal webcontentsview to auto-reload since it's generally not even visible to the user but lmk if i should change that
gnguralnick
temporarily deployed
to
minds-ci-test
GitHub Actions
Inactive
Contributor
|
lgtm, but would love a post-hoc review from @hynek-urban as well in case I'm missing something |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
From kanjun's blank-screen-after-sleep incident (Sentry
bc6d78e9): on wake, a workspace window showed a pinned-white screen with no recovery, and the Electron main process logged nothing durable, so the failure was undiagnosable from uploaded logs. The root cause is that a window'sWebContentsViewrenderers can be reaped by the OS over a long sleep, and nothing handledrender-process-gone. Each window has three such renderers, in separate processes, that can die independently: the workspace content view, the chrome (titlebar) view, and the overlay (sidebar/inbox/help) view.This implements
blueprint/electron-log-and-crash-page/: durable main-process logging, recovery UI for all three views, and automatic Sentry reporting so these deaths are visible without waiting for a manual report.Changes
Logging & rotation
electron/logger.js— tees the main process'sconsole.log/warn/error(timestamped) into a new~/.minds/logs/electron.log, and records uncaught exceptions (viauncaughtExceptionMonitor, chosen so Sentry capture / process-exit behavior is unchanged) and unhandled rejections. Reentrancy-guarded so a rotation-failureconsole.warncan't recurse into the tee. Initialized first inmain.js.electron/log-rotation.js— shared 100 MB / keep-10 size rotation, gzipping each rotated file in the background. Used byelectron.logand (viabackend.js)minds.log, which was previously an unbounded append stream (the source of ~600 MB log files). Asyncfs.WriteStream(never blocks the main thread); on hitting the threshold it ends/flushes the stream before renaming so the background gzip never reads a partial file, replays any lines that arrive mid-rotation, and gzips viastream.pipelineso a compression failure can't leak fds.Renderer-crash recovery (all three views)
render-process-goneshows a localcrashed.html("Aw, Snap!"-style: crash reason/exit code, a manual Reload, and Report a bug). No auto-reload, by design (crash-loop safety, matching Chrome/Firefox/VS Code).content-relay-preload.jsrelays the allowlistedminds:reload-crashed-view;isCrashPageUrlguardsonContentNavigateso the crash page doesn't clobber the pre-crash URL that Reload re-loads.chrome-crashed.htmlstrip anchored to that band, with a manual Reload (reload-chromeIPC → reloads/_chrome, re-primed from the cached chrome state). The workspace content view keeps running. Drag region is scoped to just the 38 px strip: macOS unions-webkit-app-regionacross all views in a window, so a full-window drag surface was swallowing clicks on the content view's Reload button when both crashed at once.loadOverlayHost) on death; it's hidden when idle, so there's no visible error state, just a fresh host for the next sidebar/inbox/help open.clean-exit/ shutdown / error-takeover teardown and defer navigation viasetImmediate(electron#19887 — neverloadURLsynchronously in that handler).Sentry reporting of renderer deaths
sentry.jsbroadens the@sentry/electronchildProcess integration to captureoomrenderer deaths as events (its default only eventsabnormal-exit/launch-failed/integrity-failure), and setsgetRendererNameso events are labeled by which view died (chrome/content/modal).crashedis deliberately not added — a native crash already produces a minidump viaSentryMinidump, so adding it would double-report.killedstays breadcrumb-only (OS/sleep reaping is unactionable and noisy); it remains inelectron.log. Auto-gated by the existingreport_unexpected_errorsopt-in.Upload plumbing
sentry.js— backend-down reports now attach bothminds.logandelectron.log(plus the newest jsonl), so a report filed while the backend is dead shows both why it died and what the shell did.imbue/minds/utils/sentry/core.py— splits the previously-mislabeled*.logattachment group into accurately-namedbackend_logs(minds.log) andelectron_logs(electron.log), each with a gzipped rotated group;core_test.pyupdated.Screenshots
This shows the result of killing both the chrome and content view renderer processes. Each state can show independently.
Testing
Automated
node --test apps/minds/test/unit/*.test.js→ 16 passed (addslog-rotation.test.js)just test-quick apps/minds/imbue/minds/utils/sentry/core_test.py/autofixand/verify-conversationgates cleanReal-app verification (done on a staging build)
oom/killed-path events reach Sentry labeled by view — three distinct issues ('chrome'/'content'/'modal'process exited), carryinggit_sha/release/environmenttags and structured{exitCode, reason}breadcrumbs. (killedwas temporarily un-exempted only to drive this test, then reverted — it does not ship as a reported reason.)/helpbug report uploads bothminds.logandelectron.logto S3 under their accurateuploaded_files_backend_logs/uploaded_files_electron_logscontext keys. (The*_rotated_logsgzip groups are correctly absent until a log crosses 100 MB.)