fix: stop the OOM/502 outage at the root — PDS restore guard, bounded native memory

[nperez0111]

Every one of the 66 requests over 30s in a measured 6h production window carried oauth_restore: "failed" — one user's PDS started blackholing packets, oauthClient.restore() had no timeout, and it hung holding the cross-process refresh lock while every other request for that DID in every worker burned the lock's full 37.5s poll budget at three synchronous SQLite statements per poll, until workers stopped calling accept() (Caddy's 166,450 dial tcp: i/o timeout, the dominant class of the outage's 171,145 502s). This PR adds guardedRestore — a 5s timeout plus a circuit breaker keyed by authorization-server host, resolved from a local KV read — bounds the lock wait to 3s with exponential backoff, and stops destroying sessions on transient failures; then removes the unbounded native memory that made the ~148 OOM kills possible: DB_MMAP_SIZE now defaults to 0, the OG cache is deleted outright (Cloudflare already serves every repeat from the edge, so over 48h the origin saw 1,189 requests across 1,134 distinct cards — a perfect origin cache could have served 4%), etag() no longer buffers whole downloads, uploads check size before arrayBuffer(), library re-sync fans out at 3 instead of one search per record, and the KV is VACUUMed on startup (measured at 1.94 GB on disk holding 34.7 MB of live rows).

It also makes the failures visible for the first time — worker_exit classification moves into a tested server/worker-exit.ts, since bunfig's test root was src and that is exactly how a number-keyed lookup against Bun's string signalCode shipped "SIGSIGKILL" and a permanently false likely_oom through every one of those kills — and adds external/array_buffers gauges and a per-worker label, without which the SO_REUSEPORT workers alias into one silently-alternating series. The two hot scans over 356k rows are indexed (FTS5 for search; a hive_book_author join table for author lookup, taking /authors/:author from ~511ms to 1ms with identical counts), and the enrich queue gains the terminal state it needed to converge — previously a row at max attempts was deleted without recording anything on the book, so the next crawler page view re-enqueued it forever. Tracing turned out to have been live the whole time (13.7M spans): the repo's compose.yaml pointed OPEN_OBSERVE_URL at localhost while the deployment used openobserve, which is how the pipeline was wrongly written off as dead, so that config is fixed and route spans are now named per matched route rather than all sharing "hono-middleware".

Two open items deliberately left out: fetchGoodreadsViaWaf books both the plain-HTTP fetch and the WAF solve against a single circuit breaker, so a degraded solver starves a path that still succeeded 4,671 times in 24h; and mem_limit can go back from 6g to 3g once the acceptance criteria in the plan hold.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added faster, more accurate book and author search.
  • Added conditional downloads for unchanged personal books.
  • Added clearer enrichment retry handling and improved library synchronization.
  • Added richer operational diagnostics and monitoring.

• Bug Fixes

  • Prevented oversized personal-book uploads with a 100 MB limit.
  • Improved resilience when authentication services are unavailable.
  • Reduced duplicate cover-image rendering and improved service connectivity handling.
  • Improved worker failure reporting and memory diagnostics.

[coderabbitai]

Warning

Review limit reached

@nperez0111, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 39 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0510ddf6-b831-4ea7-b88c-d128492b6a89

📥 Commits

Reviewing files that changed from the base of the PR and between d66d4fc and 362bf68.

📒 Files selected for processing (5)

• AGENTS.md
• server/cluster.ts
• src/context.ts
• src/middleware/otel-middleware.ts
• src/pages/bookInfo.tsx

📝 Walkthrough

Walkthrough

This PR adds worker exit and memory diagnostics, OAuth restore protection, FTS5 and author indexing, enrichment cooldowns, KV vacuuming, upload limits, conditional downloads, OG render deduplication, tracing updates, structured logging, and updated operational documentation.

Changes

Worker supervision and memory diagnostics

Layer / File(s)	Summary
Worker-exit classification server/worker-exit.ts, server/worker-exit.test.ts	Adds signal normalization, OOM detection, process-memory parsing, structured exit events, and tests.
Cluster worker integration server/cluster.ts, Dockerfile	Samples worker memory, classifies exits, logs restart details, and copies the classifier into the runtime image.
Memory diagnostics and worker metrics src/routes/debug.ts, src/metrics.ts	Adds /memory diagnostics and worker-scoped runtime metrics.

OAuth restore resilience

Layer / File(s)	Summary
Refresh-lock timeout and backoff src/auth/refresh-lock.ts, src/auth/refresh-lock.test.ts	Uses deadline-based exponential polling and validates timeout behavior.
Restore circuit breaker src/auth/restore-guard.ts, src/auth/restore-guard.test.ts	Adds bounded per-host breakers, restore timeouts, error classification, cooldowns, and state inspection.
Guarded restoration integration src/auth/storage.ts, src/context.ts, src/auth/pds-outage.test.ts	Derives issuer-host keys, protects restores, records telemetry, and preserves sessions for transient failures.

Book indexing and search

Layer / File(s)	Summary
FTS5, author, and enrichment schema src/db.ts, src/types.ts, src/db.authors.test.ts	Adds FTS5, normalized author rows, enrichment state columns, triggers, and schema tests.
Normalized author query paths src/pages/*, src/routes/og.tsx, src/utils/authorMatching.ts, src/utils/authorMatching.test.ts	Replaces serialized author pattern matching with exact indexed joins.
FTS query and resynchronization search src/utils/ftsQuery.ts, src/utils/ftsQuery.test.ts, src/routes/lib.ts	Adds FTS query validation, indexed local search, bounded refetch concurrency, and tests.

SQLite KV maintenance

Layer / File(s)	Summary
KV vacuum lifecycle src/sqlite-kv.ts, src/sqlite-kv.vacuum.test.ts, src/context.ts, src/workers/*	Returns the raw SQLite handle, adds full and incremental vacuuming, wires maintenance into startup and cleanup, and updates KV consumers.

Enrichment queue convergence

Layer / File(s)	Summary
Enrichment cooldown and queue metrics src/utils/enrichQueue.ts, src/utils/enrichQueue.test.ts, src/scrapers/goodreads.ts, src/types.ts, src/db.ts	Tracks attempts and failures, applies a seven-day cooldown, publishes queue statistics, and tests retry convergence.

Upload and streaming controls

Layer / File(s)	Summary
Personal-library upload validation src/utils/personalLibrary.ts, src/utils/personalLibrary.test.ts, src/routes/library.tsx, src/routes/opds.ts, src/xrpc/router.ts	Adds the shared 100 MiB limit, validates sizes before buffering, and supports conditional downloads with ETags.
Streaming response ETag behavior src/app.ts	Excludes ebook and import routes from global ETag middleware.

OG rendering and shedding

Layer / File(s)	Summary
In-flight OG rendering src/routes/og.tsx, src/workers/og-render/client.ts, src/metrics.ts	Removes persistent OG caching, shares concurrent renders, and records queue shedding.

Tracing, logging, and build configuration

Layer / File(s)	Summary
Tracing and request events server/plugins/otel-sdk.ts, src/middleware/otel-middleware.ts, src/middleware/wide-event.ts	Refines instrumentation, route span names, HTTP attributes, and worker event labels.
OpenObserve worker logging src/workers/open-observe-worker.ts, compose.yaml	Uses the container endpoint and emits structured failure and disable events.
Test and runtime configuration bunfig.toml, package.json, src/env.ts, src/workers/*	Expands test discovery, updates memory configuration, and adapts runtime KV initialization.
AGENTS.md operational documentation AGENTS.md	Documents the current architecture, routes, storage, indexing, OAuth, tracing, worker behavior, scrapers, and test commands.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• nperez0111/bookhive#78: Both PRs modify src/routes/lib.ts search and refetch flows.
• nperez0111/bookhive#118: Both PRs extend worker observability and failure handling.
• nperez0111/bookhive#191: Both PRs modify the cross-process OAuth refresh lock.

Poem

A rabbit checks the worker’s flight,
And guards the PDS through day and night.
FTS finds books in tidy rows,
While queues cool their retry woes.
ETags whisper, “Not modified!”
Logs hop neatly, well-structured.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 52.46% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary outage fixes: PDS restore protection and bounded native memory usage.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/bound-refetchbooks-fanout

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 22

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@AGENTS.md`:
- Around line 615-618: Update the Architecture at a Glance diagram near the
scraper overview to remove Google and ISBNdb from the active scraper list, or
explicitly label them as historical, while keeping only currently supported
scrapers as active runtime guidance.

In `@compose.yaml`:
- Around line 97-101: In compose.yaml, add a healthcheck to the openobserve
service and update the server service’s dependency condition to require
openobserve.service_healthy before startup. Preserve the existing
OPEN_OBSERVE_URL and ensure the dependency uses the healthcheck result rather
than simple container startup.

In `@server/cluster.ts`:
- Around line 90-94: Update logWorkerExit to return the classified worker-exit
event produced by classifyWorkerExit, then use its likely_oom field when
composing the restart message instead of recomputing the SIGKILL/null-exit
condition. Keep the structured and human-readable logs consistent with the
single classification result.

In `@src/app.ts`:
- Around line 121-128: Update ETAG_EXCLUDED_PREFIXES and the etag middleware
guard to explicitly bypass both "/import" and "/import/" paths, regardless of
route registration order. Add a regression test covering the import SSE response
and verifying it does not pass through etagMiddleware.

In `@src/auth/restore-guard.ts`:
- Around line 43-75: Ensure getBreaker always enforces MAX_BREAKERS by selecting
an eviction candidate even when no breaker is closed. Remove the closed-state
filter or add a bounded fallback that evicts the least recently used entry when
all breakers are open or half_open, while preserving the existing
least-recently-used selection.

In `@src/context.ts`:
- Around line 397-404: Update restoreGuardKey to avoid exposing the raw DID when
getStoredSessionIssuerHost returns no value: derive a hashed or truncated
fallback key before it is used in SESSION_FAIL breaker state and returned by
restoreGuardStates(). Preserve the existing issuer-host key behavior for stored
sessions and keep the result stable for the same DID.

In `@src/db.ts`:
- Around line 710-760: Update the post-VACUUM flow in server/cluster.ts and
src/context.ts to rebuild the external-content hive_book_fts index after
sqlite.exec("VACUUM") runs on env.DB_PATH. Execute INSERT INTO
hive_book_fts(hive_book_fts) VALUES ('rebuild') on the same database connection,
or exclude the main database from VACUUM; preserve migration 019’s FTS setup.

In `@src/middleware/otel-middleware.ts`:
- Around line 20-23: Update the initial span-name fallback in the middleware to
use only ctx.req.method, rather than combining the method with ctx.req.path;
retain ctx.req.path unchanged for ATTR_URL_PATH and preserve the existing
matched-route renaming via updateName.

In `@src/pages/bookInfo.tsx`:
- Around line 208-211: Trim the first author value at its source in the author
parsing flow, using parseAuthors or equivalent trimming before assigning
firstAuthor. Reuse that normalized value for both the hive_book_author equality
filter and the /authors/:author link so related-book queries and
getBooksByAuthor receive the same trimmed author.

In `@src/routes/lib.ts`:
- Around line 27-33: Configure the module-scoped searchSlots Semaphore with
finite maxPending and acquireTimeoutMs values so refetchBooks cannot accumulate
unbounded waiters. Update the promises pushed by refetchBooks to catch and
swallow semaphore shed/timeout rejections, preserving Promise.all completion for
the remaining book searches instead of aborting the entire re-sync.

In `@src/routes/library.tsx`:
- Around line 78-83: Add a zValidator("form", ...) middleware before the /upload
handler, defining the form schema with the required file validation and size
constraint. Keep the existing body-size limiter registered before this
validator, and update the handler to obtain the validated fields through
c.req.valid("form") instead of manually parsing FormData or performing ad-hoc
file checks.
- Around line 78-83: Enforce upload limits before request bodies are
materialized: in src/routes/library.tsx at lines 78-83, add Hono bodyLimit with
maxSize MAX_FILE_SIZE and the requested error response before c.req.formData(),
while retaining the file.size and bytes.length checks; in src/xrpc/router.ts at
lines 1656-1668, reject oversized bodies before request.arrayBuffer() using a
bounded stream/body-limit path or the app-level bodyLimit, while retaining
existing size checks.

In `@src/routes/og.tsx`:
- Around line 383-403: Update the totalRow count query to inner join hive_book
using the same hive_book_author.hiveId-to-hive_book.id relationship as the
avgRow and books queries, while preserving the existing author filter and count
behavior.

In `@src/sqlite-kv.ts`:
- Around line 140-146: Update incrementalVacuumKv to coerce pages to an integer
before interpolating it into the PRAGMA statement, and add an optional log
callback following the existing vacuumKvIfBloated pattern. Invoke the callback
from the catch with the reclaim error while preserving the current no-op
behavior for unsupported auto_vacuum modes.

In `@src/sqlite-kv.vacuum.test.ts`:
- Around line 64-66: Add a test for vacuumKvIfBloated that forces the database
exec/VACUUM operation to throw, then asserts the error is swallowed and noop
receives the "kv VACUUM failed" log message. Keep the existing empty-database
test unchanged and use the current test helpers or mocking approach.

In `@src/utils/enrichQueue.ts`:
- Around line 278-290: Update the exhausted-count query in the enrich queue
metrics flow to count rows in hive_book where enrichFailedAt is set, rather than
deleted enrich_queue rows; if the metric represents active cooldowns, apply the
established cooldown cutoff. Preserve the existing enrichQueueDepth label update
and add a regression test that exhausts a book and verifies a nonzero exhausted
metric.

In `@src/utils/ftsQuery.ts`:
- Around line 24-26: Update the comment above the cleaned assignment to describe
only collapsing internal whitespace and trimming surrounding whitespace; remove
the claim that FTS5 token-separator characters are stripped from the ends, while
leaving the input.replace behavior unchanged.

In `@src/utils/ogCache.test.ts`:
- Around line 92-101: Add a symmetric test beside the existing “cache write
fails” case in the test covering cachedOgRender: configure the storage’s get
operation to reject, invoke cachedOgRender with a renderer returning known
bytes, and assert the image is still returned with the expected byte length.
Keep the test focused on the read-failure fallback path.

In `@src/utils/ogCache.ts`:
- Around line 58-72: Make swallowed KV failures observable: in
src/utils/ogCache.ts lines 58-72, increment the existing failure counter from
both kv.get and kv.set catch paths while preserving best-effort behavior. In
src/sqlite-kv.ts lines 140-146, update incrementalVacuumKv to accept an optional
log callback and invoke it with the caught error before returning, so
maintenance failures are recorded without failing the caller.
- Around line 85-92: Update publishOgCacheStats to retain the cheap row count
but stop calculating ogCacheBytes with sum(length(value)) from og_cache. Use the
existing byte-accounting mechanism or metadata maintained during cache writes to
populate the byte metric without scanning cached payloads or synchronously
reading every row.

In `@src/utils/personalLibrary.ts`:
- Around line 84-91: Update streamPersonalBook and its download consumers to
evaluate If-None-Match before opening or streaming the ebook; when it matches
contentHash, return a 304 response containing the existing ETag and
Cache-Control headers, otherwise preserve the current streaming response.

In `@src/workers/og-render/client.ts`:
- Around line 64-67: Change the shedding log in the queue-backpressure path to
use logger.warn instead of logger.error, while preserving the existing
ogRenderShedTotal increment and log context. Leave the genuine og_render_timeout
error logging unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 63fad0d2-a88c-425a-9da6-39ca2f6acdfb

📥 Commits

Reviewing files that changed from the base of the PR and between 80a700f and 2248fe6.

📒 Files selected for processing (52)

• AGENTS.md
• Dockerfile
• bunfig.toml
• compose.yaml
• package.json
• server/cluster.ts
• server/plugins/otel-sdk.ts
• server/worker-exit.test.ts
• server/worker-exit.ts
• src/app.ts
• src/auth/pds-outage.test.ts
• src/auth/refresh-lock.test.ts
• src/auth/refresh-lock.ts
• src/auth/restore-guard.test.ts
• src/auth/restore-guard.ts
• src/auth/storage.ts
• src/context.ts
• src/db.authors.test.ts
• src/db.ts
• src/env.ts
• src/metrics.ts
• src/middleware/otel-middleware.ts
• src/middleware/wide-event.ts
• src/pages/authorBooks.tsx
• src/pages/authorDirectory.tsx
• src/pages/bookInfo.tsx
• src/routes/debug.ts
• src/routes/lib.ts
• src/routes/library.tsx
• src/routes/og.tsx
• src/scrapers/goodreads.ts
• src/scrapers/google.ts
• src/scrapers/index.ts
• src/scrapers/isbndb.ts
• src/sqlite-kv.ts
• src/sqlite-kv.vacuum.test.ts
• src/types.ts
• src/utils/authorMatching.test.ts
• src/utils/authorMatching.ts
• src/utils/enrichQueue.test.ts
• src/utils/enrichQueue.ts
• src/utils/ftsQuery.test.ts
• src/utils/ftsQuery.ts
• src/utils/imageProxy.ts
• src/utils/ogCache.test.ts
• src/utils/ogCache.ts
• src/utils/personalLibrary.ts
• src/workers/import/context.ts
• src/workers/ingester-worker.ts
• src/workers/og-render/client.ts
• src/workers/open-observe-worker.ts
• src/xrpc/router.ts

💤 Files with no reviewable changes (4)

• src/scrapers/index.ts
• src/utils/authorMatching.ts
• src/scrapers/isbndb.ts
• src/scrapers/google.ts

[coderabbitai]

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '30,115p' compose.yaml

for file in src/workers/open-observe-worker.ts server/plugins/otel-sdk.ts src/logger/index.ts; do
  if [ -f "$file" ]; then
    printf '\n--- %s ---\n' "$file"
    rg -n -C 8 'OPEN_OBSERVE_URL|ECONNREFUSED|connection refused|retry|disable|fetch|OTLP' "$file" || true
  fi
done

Repository: nperez0111/bookhive

Length of output: 9227

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- compose services relevant ---'
python3 - <<'PY'
import yaml
from pathlib import Path
p=Path('compose.yaml')
data=yaml.safe_load(p.read_text())
for name in ['server','openobserve','otel-collector']:
    svc=data.get('services',{}).get(name,{})
    print(name, {
        'depends_on': svc.get('depends_on'),
        'healthcheck': svc.get('healthcheck'),
        'ports': svc.get('ports'),
    })
PY

printf '\n--- openobserve worker flush tail ---\n'
sed -n '85,140p' src/workers/open-observe-worker.ts

printf '\n--- otel plugin lifecycle tail ---\n'
sed -n '20,120p' server/plugins/otel-sdk.ts

printf '\n--- logger setup context ---\n'
sed -n '1,80p' src/logger/index.ts

Repository: nperez0111/bookhive

Length of output: 308

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- compose services relevant ---'
python3 - <<'PY'
import json
from pathlib import Path

path = Path('compose.yaml')
text = path.read_text().splitlines()
# extract relevant service blocks as YAML is not available; simple state machine
state = []
for i,line in enumerate(text,1):
    if line.startswith('  server:') or line.startswith('  openobserve:') or line.startswith('  otel-collector:'):
        state.append([i, line.strip().rstrip(':'), {}])
    elif len(state):
        k=line.strip().split(':', 1)[0]
        if k in ('depends_on','healthcheck') and state[-1][1] in ('server','openobserve','otel-collector'):
            state[-1][2][k]=True
        elif k and k[0] != ',' and k[0] not in '-#':
            if state[-1][1] == 'otel-collector' and state[-1][2] == {}:
                # avoid config keys after depends_on line if parsed inline
                pass
for item in state:
    if not item[2]:
        print(item[1], {'depends_on': None, 'healthcheck': None})
    elif item[2].get('healthcheck'):
        print(item[1], {'depends_on': item[2].get('depends_on'), 'healthcheck': True})
    else:
        print(item[1], {'depends_on': item[2].get('depends_on'), 'healthcheck': False})
PY

printf '\n--- openobserve worker flush tail ---\n'
sed -n '85,140p' src/workers/open-observe-worker.ts

printf '\n--- otel plugin lifecycle tail ---\n'
sed -n '20,120p' server/plugins/otel-sdk.ts

printf '\n--- logger setup context ---\n'
sed -n '1,80p' src/logger/index.ts

Repository: nperez0111/bookhive

Length of output: 4942

---

Tie server startup to OpenObserve readiness.

server targets http://openobserve:5080, but openobserve has no healthcheck and server does not depend on it; otel-collector uses dependent so it can start before server and openobserve are both running. Add a healthcheck to openobserve:latest and make server depend on openobserve.service_healthy, or use an explicit startup health-check wrapper, so logs and traces are not configured against a service that is too early to accept connections.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@compose.yaml` around lines 97 - 101, In compose.yaml, add a healthcheck to
the openobserve service and update the server service’s dependency condition to
require openobserve.service_healthy before startup. Preserve the existing
OPEN_OBSERVE_URL and ensure the dependency uses the healthcheck result rather
than simple container startup.

[coderabbitai]

📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add the symmetric read-failure case.

This case covers a failing kv.set. cachedOgRender also swallows a failing kv.get at line 58 of src/utils/ogCache.ts, and no case covers that path. A read fault is the more likely production fault, because it happens on every request rather than only on a miss.

💚 Proposed test

   it("still returns the image when the cache write fails", async () => {
     const broken = createStorage({ driver: memoryDriver() });
     broken.set = async () => {
       throw new Error("disk full");
     };
     const key = ogCacheKey("book", { id: "bk_nowrite" });
 
     const result = await cachedOgRender(broken, key, 60, async () => bytesOf(12));
     expect(result.byteLength).toBe(12);
   });
+
+  it("still returns the image when the cache read fails", async () => {
+    const broken = createStorage({ driver: memoryDriver() });
+    broken.get = async () => {
+      throw new Error("no such table: og_cache");
+    };
+    const key = ogCacheKey("book", { id: "bk_noread" });
+
+    const result = await cachedOgRender(broken, key, 60, async () => bytesOf(9));
+    expect(result.byteLength).toBe(9);
+  });

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	it("still returns the image when the cache write fails", async () => {
	const broken = createStorage({ driver: memoryDriver() });
	broken.set = async () => {
	throw new Error("disk full");
	};
	const key = ogCacheKey("book", { id: "bk_nowrite" });
	const result = await cachedOgRender(broken, key, 60, async () => bytesOf(12));
	expect(result.byteLength).toBe(12);
	});
	it("still returns the image when the cache write fails", async () => {
	const broken = createStorage({ driver: memoryDriver() });
	broken.set = async () => {
	throw new Error("disk full");
	};
	const key = ogCacheKey("book", { id: "bk_nowrite" });
	const result = await cachedOgRender(broken, key, 60, async () => bytesOf(12));
	expect(result.byteLength).toBe(12);
	});
	it("still returns the image when the cache read fails", async () => {
	const broken = createStorage({ driver: memoryDriver() });
	broken.get = async () => {
	throw new Error("no such table: og_cache");
	};
	const key = ogCacheKey("book", { id: "bk_noread" });
	const result = await cachedOgRender(broken, key, 60, async () => bytesOf(9));
	expect(result.byteLength).toBe(9);
	});

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/ogCache.test.ts` around lines 92 - 101, Add a symmetric test beside
the existing “cache write fails” case in the test covering cachedOgRender:
configure the storage’s get operation to reject, invoke cachedOgRender with a
renderer returning known bytes, and assert the image is still returned with the
expected byte length. Keep the test focused on the read-failure fallback path.

[coderabbitai]

🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Check whether the dbstat virtual table is available in the Bun SQLite build,
# and confirm how ogCacheBytes / ogCacheEntries are declared.

set -uo pipefail

echo "=== metrics declarations ==="
fd -t f 'metrics.ts' src | xargs rg -n -C3 'ogCacheBytes|ogCacheEntries|ogRenderShedTotal'

echo
echo "=== bun version ==="
bun --version 2>/dev/null || echo "bun not available in sandbox"

echo
echo "=== dbstat availability ==="
cat > /tmp/dbstat-probe.ts <<'EOF'
import { Database } from "bun:sqlite";
const db = new Database(":memory:");
db.exec("CREATE TABLE og_cache (id TEXT PRIMARY KEY, value TEXT NOT NULL)");
db.exec("INSERT INTO og_cache VALUES ('a', 'b')");
try {
  console.log("dbstat OK:", db.query("SELECT sum(pgsize) AS b FROM dbstat WHERE name='og_cache'").get());
} catch (e) {
  console.log("dbstat UNAVAILABLE:", (e as Error).message);
}
EOF
bun /tmp/dbstat-probe.ts 2>&1 || echo "probe could not run"

Repository: nperez0111/bookhive

Length of output: 969

---

🏁 Script executed:

#!/bin/bash
set -uo pipefail

echo "=== locate relevant files ==="
git ls-files | rg '(^|/)src/(utils/ogCache\.ts|context\.ts|sqlite-kv\.ts|.*metrics.*)$' || true

echo
echo "=== ogCache declarations and publishOgCacheStats ==="
sed -n '1,140p' src/utils/ogCache.ts

echo
echo "=== publishOgCacheStats call sites ==="
rg -n -C 6 'publishOgCacheStats' src || true

echo
echo "=== sqlite helper/table definitions ==="
rg -n -C 4 'og_cache|dbstat|page_count|page_size' src || true

echo
echo "=== package/runtime config hints ==="
for f in package.json bun.lockb bun.lockb.bak; do
  if [ -f "$f" ]; then
    echo "--- $f"
    if [ "$f" = package.json ]; then cat "$f" | sed -n '1,220p'; else ls -l "$f"; fi
  fi
done

echo
echo "=== read-only SQLite probe using node-sqlite3 or better-sqlite3 if available, or sqlite3 CLI ==="
node - <<'JS'
const sqlite3 = require('better-sqlite3');
const fs = require('fs');
const db = new sqlite3(':memory:');
db.exec(`CREATE TABLE og_cache (id TEXT PRIMARY KEY, value TEXT NOT NULL)`);
const buf = Buffer.alloc(102400, 0x42);
db.exec(`INSERT INTO og_cache VALUES ('x', ${JSON.stringify(buf.toString('base64'))})`);
console.log(hasDbstat(db));
db.close();
function hasDbstat(db) {
  const stmt = db.prepare('SELECT sum(pgsize) AS b FROM dbstat WHERE name=?');
  try { return stmt.get('og_cache'); } catch (e) { return e && e.message; }
}
JS
sqlite3 :memory: 'CREATE TABLE og_cache(id TEXT PRIMARY KEY, value TEXT NOT NULL); INSERT INTO og_cache VALUES(\'a\',\'b\'); SELECT sum(pgsize) AS b FROM dbstat WHERE name = \'og_cache\';' 2>&1 || true

Repository: nperez0111/bookhive

Length of output: 17125

---

🌐 Web query:

Bun sqlite database module dbstat virtual table SQLITE_ENABLE_DBSTAT_VTAB

💡 Result:

The bun:sqlite module in Bun includes the dbstat virtual table, as it is compiled with the SQLITE_ENABLE_DBSTAT_VTAB option enabled [1][2]. The dbstat virtual table is a read-only eponymous virtual table used to retrieve information about disk space usage in an SQLite database [3]. Because it is eponymous, you do not need to execute CREATE VIRTUAL TABLE to instantiate it; you can query the dbstat module directly as if it were a table [3]. While Bun's bun:sqlite documentation mentions support for CREATE VIRTUAL TABLE [4][5], general support for custom virtual tables (where a JavaScript generator yields rows to SQLite) is a separate, requested feature currently tracked in the Bun issue tracker [6]. However, the built-in dbstat virtual table provided by the underlying SQLite library is available for use within Bun [1][2]. To use it, you can execute queries against the dbstat table directly in your SQL: sql SELECT * FROM dbstat; This will return information such as the table name, page number, page type, and page size for the database file [3].

Citations:

• 1: https://github.com/oven-sh/bun/blob/6618e7f7/scripts/build/deps/sqlite.ts
• 2: node:sqlite: implement the module and pass the Node v26.3.0 test suite oven-sh/bun#32498
• 3: https://sqlite.org/dbstat.html
• 4: https://bun.sh/reference/bun/sqlite
• 5: https://bun.sh/reference/bun/sqlite/Database
• 6: Virtual table support for bun:sqlite oven-sh/bun#23063

---

🌐 Web query:

Bun sqlite synchronous query documentation

💡 Result:

The bun:sqlite module in Bun provides a synchronous API for interacting with SQLite databases [1][2]. There is no native asynchronous API for bun:sqlite; all database operations executed through this module are blocking and run synchronously on the main thread [1][3][4]. Key points regarding the synchronous nature of Bun's SQLite implementation: API Design: The API is designed to be high-performance and synchronous, inspired by better-sqlite3 [1][2]. Blocking Behavior: Because the API is synchronous, long-running queries will block the event loop, which can prevent other tasks (such as UI updates or handling other requests) from executing during that time [3][4]. Workarounds: Users who require non-blocking behavior often move SQLite operations to separate worker threads in user space to offload the blocking I/O from the main event loop [3][4]. Methods: Standard methods such as.query,.run,.get, and.all all operate synchronously [5][6][2]. For further details, you can consult the official Bun documentation for the bun:sqlite module [1][2].

Citations:

• 1: https://bun.com/docs/runtime/sqlite
• 2: https://bun.com/reference/bun/sqlite
• 3: Asynchronous SQLite API oven-sh/bun#978
• 4: Async SQLite oven-sh/bun#2326
• 5: https://bun.com/reference/bun/sqlite/Database/query
• 6: https://bun.sh/reference/bun/sqlite/Database

---

Don’t sum length(value) to compute og_cache bytes

sum(length(value)) reads every cached OG row and value on the 15-minute timer. This blocks the primary worker because bun:sqlite is synchronous, and it drags the cached image payloads through the SQLite page cache while this PR is focused on page-cache pressure.

Keep the cheap row count. Use byte accounting instead of a length(value) scan.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/ogCache.ts` around lines 85 - 92, Update publishOgCacheStats to
retain the cheap row count but stop calculating ogCacheBytes with
sum(length(value)) from og_cache. Use the existing byte-accounting mechanism or
metadata maintained during cache writes to populate the byte metric without
scanning cached payloads or synchronously reading every row.

[coderabbitai]

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

src/middleware/otel-middleware.ts (1)

42-53: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Close spans from a finally path.

If next() throws, the post-processing block is skipped and span.end() at line 71 is never reached, so error spans can remain unfinished and miss route/status enrichment. Move span finalization into a finally block while keeping the exception recording from catch and rethrowing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/middleware/otel-middleware.ts` around lines 42 - 53, Update the
middleware’s span lifecycle so the route/status enrichment and span.end()
execute from a finally block even when next() throws. Preserve the catch block’s
exception recording and rethrow the original error after recording it.

src/utils/personalLibrary.ts (2)

79-126: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract a shared response helper for PersonalBookDownload results. Both download routes translate a PersonalBookDownload result into an HTTP response with the identical three-branch pattern (missing → 404, notModified → 304, else stream). Centralizing this in src/utils/personalLibrary.ts prevents the two routes from drifting if this response logic changes later (for example, adding a header to the 304 branch).

• src/utils/personalLibrary.ts#L79-L126: add an exported helper, e.g. respondToDownload(c, download, notFoundBody), that implements the shared branch logic and returns the appropriate c.body(...) call.
• src/routes/library.tsx#L247-L257: replace the manual !download/notModified/stream branches with a call to the new shared helper.
• src/routes/opds.ts#L528-L535: replace the manual !download/notModified/stream branches with a call to the new shared helper, passing its own not-found body.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/personalLibrary.ts` around lines 79 - 126, In
src/utils/personalLibrary.ts lines 79-126, add an exported helper such as
respondToDownload that handles missing downloads with the supplied not-found
body, notModified results with a 304 response and headers, and stream results
with their headers. In src/routes/library.tsx lines 247-257 and
src/routes/opds.ts lines 528-535, replace each duplicated three-branch response
flow with the shared helper, passing the route-specific not-found body.

---

10-17: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Add the XRPC personal-book download route for GET /books/:hash/download.

src/routes/library.tsx and src/routes/opds.ts both use streamPersonalBook(...) with the new PersonalBookDownload contract, including if-none-match/304 handling. src/xrpc/router.ts does not define this endpoint, so XRPC clients can request these books while the route is still missing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/personalLibrary.ts` around lines 10 - 17, Add the missing XRPC GET
endpoint for /books/:hash/download in the router, using the PersonalBookDownload
contract and streamPersonalBook(...) consistently with the implementations in
library.tsx and opds.ts. Preserve their if-none-match handling, including
returning 304 when the resource is unchanged, and return the streamed
personal-book response for valid requests.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@AGENTS.md`:
- Line 139: Update the GET /:hiveId documentation to remove the claim that
?force-refresh=true performs inline enrichment; describe forced refresh as
queued enrichment instead, consistent with the existing route contract and
enqueueEnrichment usage.

In `@server/cluster.ts`:
- Around line 45-65: Clear the worker index entry in lastMemory immediately
after logWorkerExit captures the exit event, preventing samples from being
reused by a restarted process. Update the restart message near the exit handling
around logWorkerExit to read anon_kb from the captured exit event rather than
directly from lastMemory.

In `@src/context.ts`:
- Around line 161-165: Update the catch block around the hive_book_fts rebuild
after VACUUM to rethrow err immediately after logger.error, ensuring startup
aborts when rebuilding fails and the worker cannot serve with a stale
external-content index.
- Around line 222-225: Update the DELETE flow in the surrounding context method
to await the asynchronous DELETE operation before calling
incrementalVacuumKv(kvSqlite). Invoke incrementalVacuumKv only after the DELETE
completes successfully, preserving the existing vacuum behavior and avoiding
execution when the DELETE fails.

In `@src/pages/bookInfo.tsx`:
- Around line 144-149: Normalize all tab-separated author segments once near
firstAuthor into a trimmed author array, then derive firstAuthor from that array
and use the same array when rendering author links. Preserve the canonical
tab-separated source format and ensure every generated link uses the trimmed
author value.

In `@src/sqlite-kv.ts`:
- Around line 143-154: Make incremental-vacuum errors observable from the
production timer by updating the incrementalVacuumKv call in the context timer
to pass the available application logger as its log callback. Preserve the
existing optional callback behavior and error message in incrementalVacuumKv,
ensuring SQLITE_BUSY and disk failures are emitted through the production
logger.

---

Outside diff comments:
In `@src/middleware/otel-middleware.ts`:
- Around line 42-53: Update the middleware’s span lifecycle so the route/status
enrichment and span.end() execute from a finally block even when next() throws.
Preserve the catch block’s exception recording and rethrow the original error
after recording it.

In `@src/utils/personalLibrary.ts`:
- Around line 79-126: In src/utils/personalLibrary.ts lines 79-126, add an
exported helper such as respondToDownload that handles missing downloads with
the supplied not-found body, notModified results with a 304 response and
headers, and stream results with their headers. In src/routes/library.tsx lines
247-257 and src/routes/opds.ts lines 528-535, replace each duplicated
three-branch response flow with the shared helper, passing the route-specific
not-found body.
- Around line 10-17: Add the missing XRPC GET endpoint for /books/:hash/download
in the router, using the PersonalBookDownload contract and
streamPersonalBook(...) consistently with the implementations in library.tsx and
opds.ts. Preserve their if-none-match handling, including returning 304 when the
resource is unchanged, and return the streamed personal-book response for valid
requests.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 749ef9aa-573e-4046-a60a-a6956bf35c8e

📥 Commits

Reviewing files that changed from the base of the PR and between 2248fe6 and 81e0029.

📒 Files selected for processing (22)

• AGENTS.md
• compose.yaml
• server/cluster.ts
• src/app.ts
• src/auth/restore-guard.test.ts
• src/auth/restore-guard.ts
• src/context.ts
• src/metrics.ts
• src/middleware/otel-middleware.ts
• src/pages/bookInfo.tsx
• src/routes/lib.ts
• src/routes/library.tsx
• src/routes/og.tsx
• src/routes/opds.ts
• src/sqlite-kv.ts
• src/sqlite-kv.vacuum.test.ts
• src/utils/enrichQueue.test.ts
• src/utils/enrichQueue.ts
• src/utils/ftsQuery.ts
• src/utils/personalLibrary.test.ts
• src/utils/personalLibrary.ts
• src/workers/og-render/client.ts

💤 Files with no reviewable changes (1)

• src/metrics.ts

[coderabbitai]

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Remove the inline-enrichment claim from the route documentation.

Line 55 says enrichment is queued and never inline. This line says ?force-refresh=true enriches inline. The conflicting guidance can reintroduce request-bound scraper work. Document forced refresh as a queued operation, or update both sections and the route implementation to the same contract.

As per coding guidelines, routes must not perform inline enrichment when queued enrichment is appropriate; use enqueueEnrichment or enqueueEnrichmentBatch instead.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@AGENTS.md` at line 139, Update the GET /:hiveId documentation to remove the
claim that ?force-refresh=true performs inline enrichment; describe forced
refresh as queued enrichment instead, consistent with the existing route
contract and enqueueEnrichment usage.

Source: Coding guidelines

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Stop startup when the FTS rebuild fails.

If the rebuild fails after VACUUM, this code logs the error and continues. The external-content index can then reference stale hive_book rowids and return incorrect books from search. Rethrow the error after logging it so the worker does not serve with an invalid index.

As per coding guidelines, “Rebuild hive_book_fts after the startup VACUUM, on the primary worker, because VACUUM may change implicit rowids used by the external-content index.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/context.ts` around lines 161 - 165, Update the catch block around the
hive_book_fts rebuild after VACUUM to rethrow err immediately after
logger.error, ensuring startup aborts when rebuilding fails and the worker
cannot serve with a stale external-content index.

Source: Coding guidelines

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@AGENTS.md`:
- Around line 314-320: The documentation in AGENTS.md incorrectly says the
main-database VACUUM uses the same freelist ratio as the KV. Update that phrase
to state it is gated on the main database’s freelist ratio, while preserving the
clarification that the KV VACUUM remains unconditional and the main-database
VACUUM must stay conditional.

In `@src/sqlite-kv.ts`:
- Around line 158-160: Remove the unused exported readPragma helper from
sqlite-kv.ts. Do not alter the currently used SQL-building helpers or make
changes based solely on the documented src/context.ts callers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e214ea6b-3f21-4354-a132-4c45b9b05279

📥 Commits

Reviewing files that changed from the base of the PR and between 81e0029 and d66d4fc.

📒 Files selected for processing (3)

• AGENTS.md
• src/context.ts
• src/sqlite-kv.ts

[coderabbitai]

🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

🧩 Analysis chain

🏁 Script executed:

rg -n -C 2 '\breadPragma\s*\(' --glob '*.ts'

Repository: nperez0111/bookhive

Length of output: 157

---

🏁 Script executed:

#!/bin/bash
set -u

echo "--- tracked files matching sqlite-kv.ts / context.ts ---"
git ls-files | rg '(^|/)(sqlite-kv|context)\.ts$|sqlite'

echo
echo "--- locate readPragma text in tracked TS files ---"
rg -n -C 2 'readPragma|PRAGMA' --glob '*.ts' || true

echo
echo "--- file snippets if sqlite-kv exists ---"
for f in $(git ls-files | rg '(^|/)sqlite-kv\.ts$'); do
  echo "### $f"
  wc -l "$f"
  sed -n '1,220p' "$f" | cat -n
done

Repository: nperez0111/bookhive

Length of output: 412

---

🏁 Script executed:

#!/bin/bash
set -u

echo "--- src/sqlite-kv.ts line count and relevant section ---"
wc -l src/sqlite-kv.ts
sed -n '130,180p' src/sqlite-kv.ts | cat -n

echo
echo "--- src/context.ts relevant sqlite-kv imports/calls section ---"
wc -l src/context.ts
sed -n '1,220p' src/context.ts | cat -n

echo
echo "--- all tracked occurrences of identifiers in src/sqlite-kv.ts ---"
python3 - <<'PY'
import re, pathlib
p=pathlib.Path('src/sqlite-kv.ts')
text=p.read_text()
print("contains readPragma:", "readPragma" in text)
for name in ["readPragma","writePragma","PRAGMA","page_count","freelist_count","auto_vacuum"]:
    print(f"name={name}: count={text.count(name)}")
PY

Repository: nperez0111/bookhive

Length of output: 13635

---

Remove the unused readPragma helper before relying on the current callers.

The helper is exported but has no current callsite, so the documented src/context.ts callers do not justify it. Keep unused exported SQL-building helpers unless they are reused.

🧰 Tools

🪛 ast-grep (0.45.0)

[error] 158-158: Avoid SQL injection
Context: sqlite.query(PRAGMA ${name})
Note: [CWE-89] Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection').

(sql-injection-typescript)

🪛 OpenGrep (1.26.0)

[ERROR] 159-159: SQL query built via string concatenation or template literal passed to query()/execute(). Use parameterized queries instead.

(coderabbit.sql-injection.raw-query-concat-js)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/sqlite-kv.ts` around lines 158 - 160, Remove the unused exported
readPragma helper from sqlite-kv.ts. Do not alter the currently used
SQL-building helpers or make changes based solely on the documented
src/context.ts callers.

Source: Linters/SAST tools