fix: implement component shutdown panic handling

[jiangzhe]

Closes #942

Summary by CodeRabbit

• Bug Fixes

  • Improved shutdown reliability by completing cleanup even when background workers or components fail.
  • Shutdown failures are now reported consistently rather than treated as successful.
  • Prevented secondary cleanup errors from obscuring the original failure.
  • Improved startup rollback and transaction-worker recovery during failures.

• Diagnostics

  • Added clearer shutdown and worker-failure reporting with useful panic context.

• Tests

  • Expanded coverage for shutdown ordering, failure propagation, cleanup, and resource release.

[coderabbitai]

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (2)

• docs/backlogs/closed/000174-atomic-index-metadata-publication-and-panic-safe-shutdown.md is excluded by none and included by none
• docs/tasks/000256-component-shutdown-panic-containment.md is excluded by none and included by none

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: d7f57ee8-3633-49fb-bb70-3f5fff53e0cb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Component shutdown now contains hook and worker panics, completes reverse-order cleanup, tracks degraded owners and guards, and propagates or suppresses the first panic after teardown. Engine shutdown preserves lifecycle completion and root-release ordering.

Changes

Shutdown panic containment

Layer / File(s)	Summary
Component shutdown aggregation doradb-storage/src/component.rs, doradb-storage/src/quiescent.rs	Component hooks catch and aggregate panics, preserve the first payload, track degraded state and outstanding guards, and conditionally retain unsafe owners. Tests cover ordering, payload handling, suppression, and cleanup.
Worker join and resource cleanup doradb-storage/src/runtime/mandatory.rs, doradb-storage/src/trx/purge.rs, doradb-storage/src/trx/sys.rs, doradb-storage/src/file/fs.rs, doradb-storage/src/buffer/evictor.rs	Worker shutdown joins all handles, logs panic payloads, completes required cleanup, and defers first-panic propagation. Comments document unsupported arbitrary worker-body unwinds.
Engine teardown integration doradb-storage/src/engine.rs, doradb-storage/src/root.rs	Engine shutdown now returns and handles ComponentShutdownOutcome. Root release remains after subordinate shutdown hooks.
Shutdown ownership documentation doradb-storage/src/buffer/mod.rs, doradb-storage/src/catalog/mod.rs, doradb-storage/src/lock/mod.rs, doradb-storage/src/poison.rs	Passive hooks document component shutdown ownership, ordering, and panic-safety assumptions.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Engine
  participant ComponentRegistry
  participant WorkerComponents
  participant StorageRootLease
  Engine->>ComponentRegistry: start reverse-order shutdown
  ComponentRegistry->>WorkerComponents: invoke shutdown hooks
  WorkerComponents-->>ComponentRegistry: complete cleanup and return panic outcome
  ComponentRegistry-->>Engine: return ComponentShutdownOutcome
  Engine->>StorageRootLease: release root lease
  Engine-->>Engine: propagate or suppress first shutdown panic

Loading

Possibly related PRs

• jiangzhe/doradb#923: Introduced related mandatory runtime and component shutdown paths.
• jiangzhe/doradb#818: Modified overlapping component, filesystem, evictor, and transaction shutdown handling.
• jiangzhe/doradb#422: Introduced the quiescent guard primitive used for shutdown owner inspection.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: implementing panic handling during component shutdown.
Linked Issues check	✅ Passed	The changes address issue #942 objectives, including panic containment, continued teardown, worker joining, resource release, degraded cleanup, ordering, and documented panic boundaries.
Out of Scope Changes check	✅ Passed	The changes remain within issue #942 scope and add only supporting implementation, tests, cleanup logic, and shutdown-safety documentation.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch shutdown-panic

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 1 medium

Results:
1 new issue

Category	Results
Complexity	1 medium

View in Codacy

🟢 Metrics 77 complexity · 26 duplication

Metric	Results
Complexity	77
Duplication	26

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[greptile-apps]

Greptile Summary

The PR makes component shutdown panic-contained while preserving reverse teardown order and terminal engine lifecycle semantics.

• Aggregates component-hook panics, continues later shutdown hooks, and propagates or suppresses the first payload after cleanup.
• Adds degraded owner release based on shutdown failure and outstanding quiescent guards.
• Completes multi-worker joins and required resource release for mandatory-runtime, purge, and redo workers.
• Documents shutdown authority, ordering, containment limits, and regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains in the eligible follow-up scope.

Important Files Changed

Filename	Overview
doradb-storage/src/component.rs	Adds per-hook panic containment, aggregate outcomes, once-only dispatch, and guard-aware degraded owner release without an eligible unresolved follow-up finding.
doradb-storage/src/engine.rs	Publishes terminal lifecycle state and releases the shutdown mutex before applying the aggregate panic policy.
doradb-storage/src/runtime/mandatory.rs	Preserves live runners when callers remain and otherwise joins all runners before exposing the first terminal failure.
doradb-storage/src/trx/sys.rs	Ensures purge and redo shutdown complete all joins and required resource release before panic propagation.
doradb-storage/src/quiescent.rs	Adds acquire-ordered guard-count observation for terminal degraded owner-release decisions.
docs/engine-component-lifetime.md	Documents the panic-containment boundary, shutdown order, degraded release policy, and unsupported arbitrary worker-body unwinds.

_{Reviews (3): Last reviewed commit: "resolve task" | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 91.25767% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.46%. Comparing base (d9684e3) to head (a001a71).

Files with missing lines	Patch %	Lines
doradb-storage/src/component.rs	88.07%	34 Missing ⚠️
doradb-storage/src/engine.rs	93.70%	8 Missing ⚠️
doradb-storage/src/runtime/mandatory.rs	94.11%	8 Missing ⚠️
doradb-storage/src/trx/sys.rs	90.16%	6 Missing ⚠️
doradb-storage/src/trx/purge.rs	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #943      +/-   ##
==========================================
+ Coverage   93.42%   93.46%   +0.03%     
==========================================
  Files         149      149              
  Lines      127741   128332     +591     
==========================================
+ Hits       119347   119948     +601     
+ Misses       8394     8384      -10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@doradb-storage/src/runtime/mandatory.rs`:
- Around line 930-945: Update the mandatory runner shutdown flow around
FirstPanic and signal_stop so it does not signal or join runners while admission
callers remain. When callers != 0, fail immediately after recording the existing
panic; only drain internal admission, call signal_stop, and proceed to
handle.join once caller admission is confirmed drained.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: a041f684-660e-4b11-9bc0-8d6c81e46918

📥 Commits

Reviewing files that changed from the base of the PR and between d9684e3 and 5371a6f.

⛔ Files ignored due to path filters (3)

• docs/engine-component-lifetime.md is excluded by none and included by none
• docs/tasks/000256-component-shutdown-panic-containment.md is excluded by none and included by none
• docs/tasks/next-id is excluded by none and included by none

📒 Files selected for processing (13)

• doradb-storage/src/buffer/evictor.rs
• doradb-storage/src/buffer/mod.rs
• doradb-storage/src/catalog/mod.rs
• doradb-storage/src/component.rs
• doradb-storage/src/engine.rs
• doradb-storage/src/file/fs.rs
• doradb-storage/src/lock/mod.rs
• doradb-storage/src/poison.rs
• doradb-storage/src/quiescent.rs
• doradb-storage/src/root.rs
• doradb-storage/src/runtime/mandatory.rs
• doradb-storage/src/trx/purge.rs
• doradb-storage/src/trx/sys.rs