perf: make the /explore aggregates index-only and cache them with SWR

AGENTS.md

@@ -57,6 +57,8 @@ Worker threads (bundled to .output/server/workers/):
 - **Enrichment is queued, never inline**: routes call `enqueueEnrichment`/`enqueueEnrichmentBatch` (`src/utils/enrichQueue.ts`). The primary worker drains it every 5s at concurrency 3, with exponential backoff. `enrichBookWithDetailedData` holds its own semaphore (4) + 45s deadline. That drain interval × concurrency is also the **only** rate limit on requests to Goodreads — 36 fetches/min, healthy or not. Don't add a second one.
 - **`enrich_queue.attempts` counts answers from Goodreads, not failures.** A run reports `enrich_retry` in the wide-event bag: `retry` spends an attempt, `defer` costs nothing and re-queues on a decaying schedule, `dead` tombstones the book immediately. Anything the app decided on its own — a WAF challenge, a timeout, a transport error — is a `defer`. Getting this wrong is expensive: when refusals counted as attempts, one 6h window wrote off 2,854 books for 7 days apiece **without sending a single request on their behalf** (98% of everything the queue gave up on). The bound on defers is `MAX_QUEUE_AGE_MS` (7d from `enqueuedAt`, which survives re-enqueue), not the attempt counter.
 - **Author lookups use `hive_book_author` join** (mig 020), not `LIKE`. This is exact identity, not text search.
+- **The `/explore` aggregates say `INDEXED BY idx_hive_book_stats`, and that is not decoration.** Each groups the whole of `hive_book_author`/`hive_book_genre` joined to `hive_book`, and migration 024 added `hive_book(id, ratingsCount, rating, language)` so that join can be index-only. But **this database has never been `ANALYZE`d** — with no `sqlite_stat1` the planner prefers the UNIQUE `sqlite_autoindex_hive_book_1` for an `id = ?` equality and fetches the whole row from the 1.62 GB table anyway, so the index does nothing unless the query names it. Measured at 350k books: `/explore/authors` 2742ms → 260ms, `/explore`'s genre list 209ms → 21ms. Don't "clean up" the hint, and don't reach for `ANALYZE` instead — it would re-plan every query in an app whose indexes were all tuned against the no-stats planner. `src/utils/authorStats.test.ts` asserts the plans.
+- **`bun:sqlite` is synchronous, so a slow query is a whole-worker outage.** `stmt.all()` blocks the event loop, and production runs 3 processes — a 3s aggregate on a request path stalls a third of _all_ traffic, not just that route. Hence the explore aggregates are cached with **stale-while-revalidate** (`ttl: 24h, revalidateAfter: 1h`), not a plain TTL: a plain TTL makes every expiry a synchronous cliff for whichever request draws the short straw. The caching lives _inside_ `src/utils/authorStats.ts` / `src/utils/exploreGenres.ts`, not at the call sites — the three consumers (`/explore`, `/explore/authors`, XRPC `getExplore`) used to wrap the same query in three different policies, one of which was no cache at all.
 - **Library re-sync** fans out at most `REFETCH_SEARCH_CONCURRENCY` (3) searches.
 
 **The app shell scroller — never put `overflow-*-auto` on `<main>`.** The `jsxRenderer` in
@@ -123,6 +125,10 @@ into a 304. Without it an e-reader re-downloads every book on every sync.
 
 **Anonymous page cache** (`src/middleware/anon-page-cache.ts`): serves GET requests without a `sid` cookie on `/books/*`, `/explore*`, `/authors/*` from KV (gzipped HTML, 1h TTL). Prod-only.
 
+**The cache key percent-encodes the query; it must never be joined with a literal `?`.** unstorage's `normalizeKey` is `key.split("?")[0].replace(/[/\\]/g, ":")…` — it _discards the query string_. A `?`-joined key therefore collapsed every variant of a path onto one entry, which made `ALLOWED_QUERY_PARAMS` and the sorted-query construction dead code and served visibly wrong pages: `/explore?lang=French` got the English render, `/authors/X?page=2` got page 1, `/explore/genres/Y?sort=relevance` got the popularity sort. The key is now `page:{pathname}:q:{encodeURIComponent(query)}` — `encodeURIComponent` escapes `?`, `/` and `\`, the only characters `normalizeKey` touches. Changing the key format orphans the old rows; the 15-minute sweep in `src/context.ts` clears them at 2× TTL.
+
+**The size limit is measured on the bytes we store, i.e. after gzip** (`MAX_STORED_BYTES`, 256 KB), with a separate 4 MB ceiling on the uncompressed buffer purely to bound memory. It used to compare the _uncompressed_ body against 512 KB and then store the gzipped form, which rejected pages that would have cost ~25 KB of KV. That matters because production inlines the whole CSS bundle into `<head>` (`getInlineCss`, `src/utils/manifest.ts`) and `/explore/authors` renders 500 author rows on top of it — near enough to the old ceiling to fall off it, and the failure is silent (a rejected page sets no `x-page-cache` header at all, which is the diagnostic: `curl -sD- <url>` and look for it).
+
 **Caching policy** lives in one place: `src/utils/cacheHeaders.ts`. One rule —
 **signed in (`sid` cookie) → `private, no-store` on every path; signed out →
 cache aggressively** so Cloudflare absorbs the scraper load. Three layers apply
@@ -186,9 +192,12 @@ Two traps this encodes, both of which caused real bugs:
 - `/app` → `src/pages/app.tsx` — iOS app landing
 - `/import` → `src/pages/import.tsx` — CSV import page, SSE progress
 - `/search` → `src/pages/searchResults.tsx` (zValidator query `q`/`page`/`lang`)
-- `/explore` → `src/pages/explore.tsx` — explore hub
+- `/explore` → `src/pages/explore.tsx` — explore hub (`?lang=`)
 - `/explore/genres` → `src/pages/genres.tsx`; `/explore/genres/:genre` → `src/pages/genreBooks.tsx`
-- `/explore/authors` → `src/pages/authorDirectory.tsx`
+- `/explore/authors` → `src/pages/authorDirectory.tsx` (`?lang=`)
+
+**`?lang=` is validated against `getAvailableLanguages`, never passed through** (`resolveLanguage`, `src/utils/getLanguages.ts`). It keys a cached 356k-row aggregate _and_ is in the anon page cache's `ALLOWED_QUERY_PARAMS`, so an arbitrary string is an unbounded KV-cardinality and CPU amplifier. Related: `/explore/authors` used to ignore `lang` entirely while `/explore` linked to it _with_ `lang` and the page cache keyed on it — every language a crawler found became its own cache entry holding byte-identical HTML, each paying its own cold render.
+
 - `/authors/:author` → `src/pages/authorBooks.tsx`
 - `/genres`, `/genres/:genre` → 301 redirects to `/explore/genres`
 - `/.well-known/atproto-did` → returns DID constant
@@ -455,7 +464,9 @@ Client hooks/utils: `useSearchBooks.ts`, `useDebounce.ts`, `icons.tsx`, `debounc
 
 ### Database (`src/db.ts`)
 
-SQLite via Kysely. Schema + all migrations (001–023) in one file. `createDb` sets WAL/perf PRAGMAs. `mmap_size` defaults to 0 (see `DB_MMAP_SIZE` in `src/env.ts`). Kysely talks to `bun:sqlite` through `src/bun-sqlite-kysely.ts`, which rewrites `begin` to `BEGIN IMMEDIATE` (deferred transactions fail with `SQLITE_BUSY_SNAPSHOT` across cluster processes).
+SQLite via Kysely. Schema + all migrations (001–024) in one file. `createDb` sets WAL/perf PRAGMAs. `mmap_size` defaults to 0 (see `DB_MMAP_SIZE` in `src/env.ts`). Kysely talks to `bun:sqlite` through `src/bun-sqlite-kysely.ts`, which rewrites `begin` to `BEGIN IMMEDIATE` (deferred transactions fail with `SQLITE_BUSY_SNAPSHOT` across cluster processes).
+
+That wrapper also decides `statement.reader`, which is how Kysely picks `all()` (rows) over `run()` (changes). **It asks SQLite — `stmt.columnNames` is empty for anything that doesn't produce rows — rather than pattern-matching the SQL text.** The old regex was anchored on a leading `SELECT`, so `WITH cte AS (…) SELECT …` was classified as a write and Kysely got **zero rows with no error of any kind**; the author-directory cover lookup is a window function over a CTE and silently returned nothing. `columnNames` also gets the converse right, which a regex struggles with: `WITH cte AS (…) INSERT INTO …` is not a reader.
 
 | Table                 | Purpose                   | Key columns                                                                                                                                                                                              |
 | --------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -520,7 +531,9 @@ SQLite-backed unstorage. Mounts: `search:` (in-memory LRU), `profile:`, `identit
 | `circuitBreaker.ts`     | Three-state breaker — **`auth/restore-guard.ts` only.** Right when refusing is cheaper for a waiting user than failing; wrong for scraping, where the queue can defer instead. See the note under Scrapers                                                                                                                                                                                                                    |
 | `bookIdentifiers.ts`    | ISBN/ID normalization + persistence                                                                                                                                                                                                                                                                                                                                                                                           |
 | `bookProgress.ts`       | BookProgress serialization                                                                                                                                                                                                                                                                                                                                                                                                    |
-| `readThroughCache.ts`   | KV read-through with TTL                                                                                                                                                                                                                                                                                                                                                                                                      |
+| `readThroughCache.ts`   | KV read-through with TTL + optional `revalidateAfter` (stale-while-revalidate). Prefer SWR for anything expensive — a plain TTL makes every expiry a blocking recompute on a request path. The entry is stamped **after** the fetch resolves, so a slow fetch isn't born stale                                                                                                                                                |
+| `authorStats.ts`        | `getAuthorStats` / `getFeaturedAuthors` — the `/explore` author aggregates, SWR-cached inside the helper, `INDEXED BY idx_hive_book_stats`. Featured is a strict prefix of the directory list, not its own query                                                                                                                                                                                                              |
+| `exploreGenres.ts`      | `getTopGenres` — same, for genres. Only joins `hive_book` when a language is given                                                                                                                                                                                                                                                                                                                                            |
 | `csv.ts`                | Goodreads/StoryGraph CSV parsers                                                                                                                                                                                                                                                                                                                                                                                              |
 | `lists.ts`              | Book list (shelf) CRUD against PDS                                                                                                                                                                                                                                                                                                                                                                                            |
 | `readingStats.ts`       | Reading stats aggregation by year                                                                                                                                                                                                                                                                                                                                                                                             |

src/bun-sqlite-kysely.test.ts

@@ -1,6 +1,6 @@
 import { describe, it, expect, beforeEach } from "bun:test";
 import { Database as DatabaseSync } from "bun:sqlite";
-import { Kysely, SqliteDialect, type Generated } from "kysely";
+import { Kysely, SqliteDialect, sql, type Generated } from "kysely";
 
 import { toImmediateTransaction, wrapBunSqliteForKysely } from "./bun-sqlite-kysely";
 
@@ -86,6 +86,35 @@ describe("wrapBunSqliteForKysely", () => {
     expect(rows[0]?.note).toBe("n");
   });
 
+  it("returns rows from a query fronted by a CTE", async () => {
+    // `WITH ... SELECT` used to be classified as a non-reader (the check was
+    // anchored on a leading SELECT), so Kysely called run() and the query
+    // yielded zero rows with no error whatsoever. The author-directory cover
+    // lookup is a window function over a CTE, and it silently returned nothing.
+    await db.insertInto("thing").values({ name: "a", note: "n" }).execute();
+
+    const rows = await sql<{ name: string }>`
+      WITH ranked AS (
+        SELECT name, ROW_NUMBER() OVER (ORDER BY id) AS rn FROM thing
+      )
+      SELECT name FROM ranked WHERE rn = 1
+    `.execute(db);
+
+    expect(rows.rows.map((r) => r.name)).toEqual(["a"]);
+  });
+
+  it("still reports affected rows for a write fronted by a CTE", async () => {
+    // The converse: a leading CTE does not make a statement a reader.
+    await db.insertInto("thing").values({ name: "a", note: null }).execute();
+
+    const res = await sql`
+      WITH targets AS (SELECT id FROM thing WHERE name = 'a')
+      UPDATE thing SET name = 'b' WHERE id IN (SELECT id FROM targets)
+    `.execute(db);
+
+    expect(Number(res.numAffectedRows)).toBe(1);
+  });
+
   it("runs transactions", async () => {
     await db.transaction().execute(async (trx) => {
       await trx.insertInto("thing").values({ name: "in-txn", note: null }).execute();

src/bun-sqlite-kysely.ts

@@ -25,9 +25,21 @@ export interface KyselySqliteStatement {
  * rows must report true — including `INSERT`/`UPDATE`/`DELETE ... RETURNING`
  * (SQLite >= 3.35), which otherwise executes fine but hands Kysely zero rows, so
  * `.returning(...).executeTakeFirstOrThrow()` fails with "no result".
+ *
+ * SQLite answers this exactly, so ask it rather than pattern-matching the text:
+ * a prepared statement's `columnNames` is empty for anything that doesn't
+ * produce rows. The regex below is only a fallback for a runtime that doesn't
+ * expose it.
+ *
+ * This used to be `/^\s*SELECT\b/` plus a RETURNING check, which got
+ * `WITH cte AS (...) SELECT ...` wrong — a leading CTE is extremely common for
+ * window-function queries, and misclassifying one produces **zero rows with no
+ * error at all**. `columnNames` also gets the converse right, where a regex
+ * struggles: `WITH cte AS (...) INSERT INTO ...` is not a reader.
  */
-function isReaderStatement(sql: string): boolean {
-  return /^\s*SELECT\b/i.test(sql) || /\bRETURNING\b/i.test(sql);
+function isReaderStatement(stmt: { columnNames?: string[] }, sql: string): boolean {
+  if (Array.isArray(stmt.columnNames)) return stmt.columnNames.length > 0;
+  return /^\s*(?:SELECT|WITH)\b/i.test(sql) || /\bRETURNING\b/i.test(sql);
 }
 
 /**
@@ -55,7 +67,7 @@ export function wrapBunSqliteForKysely(db: DatabaseSync): KyselySqliteDatabase {
     prepare(rawSql: string): KyselySqliteStatement {
       const sql = toImmediateTransaction(rawSql);
       const stmt = db.prepare(sql);
-      const reader = isReaderStatement(sql);
+      const reader = isReaderStatement(stmt, sql);
       return {
         get reader() {
           return reader;

src/db.ts

@@ -1006,6 +1006,66 @@ migrations["023"] = {
   },
 };
 
+/**
+ * Covering indexes for the /explore family's author and genre aggregates.
+ *
+ * Every one of them is a `GROUP BY` over the whole of `hive_book_author` (or
+ * `hive_book_genre`) joined to `hive_book`, and every one of them planned as
+ * `SEARCH b USING INDEX sqlite_autoindex_hive_book_1 (id=?)` — an index probe
+ * to get a rowid, then a fetch of the whole 356k-row, 1.62 GB `hive_book` row
+ * just to read `ratingsCount`/`rating`/`language`. Against the 16 MB
+ * `cache_size` with `mmap_size = 0` that is 356k random reads; `/explore` took
+ * 6-9s and `/explore/authors` 9-14.5s, and `bun:sqlite` is synchronous, so
+ * each one froze a whole worker's event loop.
+ *
+ * `idx_hive_book_stats` is ~18 MB and holds exactly the columns those
+ * aggregates read, so the join becomes index-only and the working set fits in
+ * the page cache. Deliberately WITHOUT `thumbnail` — URLs are 60-100 bytes a
+ * row and would triple the index, evicting the thing we are trying to keep
+ * resident. The handful of thumbnails the featured row needs are fetched by id
+ * afterwards (see `src/utils/authorStats.ts`).
+ *
+ * IMPORTANT: the index alone does nothing. This database has never been
+ * ANALYZEd, and with no `sqlite_stat1` the planner prefers the UNIQUE
+ * `sqlite_autoindex_hive_book_1` for an `id = ?` equality and goes right back
+ * to the table. The queries therefore say `INDEXED BY idx_hive_book_stats`
+ * explicitly. Shipping `ANALYZE` instead would re-plan every other query in an
+ * app whose indexes were all hand-tuned against the no-stats planner — far too
+ * much blast radius for an index migration. If you drop this index, the
+ * `INDEXED BY` clauses become hard errors rather than silent 9s regressions,
+ * which is the intent.
+ */
+migrations["024"] = {
+  async up(db: Kysely<unknown>) {
+    // `IF NOT EXISTS` throughout (as migration 012 does): a half-applied state
+    // — a backup restored without its WAL, an index created by hand while
+    // debugging — otherwise makes this throw `index already exists` on every
+    // boot, and since migrations run inside the startup barrier that is a
+    // permanent crash loop rather than a degraded page.
+    await sql`CREATE INDEX IF NOT EXISTS idx_hive_book_stats ON hive_book(id, ratingsCount, rating, language)`.execute(
+      db,
+    );
+
+    // `idx_hive_book_author_first` (migration 020) is a strict prefix of this.
+    // Adding `hiveId` makes the `WHERE position = 0 GROUP BY author` side of
+    // the aggregate index-only too — the join key no longer costs a rowid
+    // fetch per row. Every other consumer of this table filters on `author`
+    // alone and is served by `idx_hive_book_author_author`, so the old index
+    // has no remaining reader.
+    await sql`CREATE INDEX IF NOT EXISTS idx_hive_book_author_first_cover ON hive_book_author(position, author, hiveId)`.execute(
+      db,
+    );
+    await sql`DROP INDEX IF EXISTS idx_hive_book_author_first`.execute(db);
+  },
+  async down(db: Kysely<unknown>) {
+    await sql`CREATE INDEX IF NOT EXISTS idx_hive_book_author_first ON hive_book_author(position, author)`.execute(
+      db,
+    );
+    await sql`DROP INDEX IF EXISTS idx_hive_book_author_first_cover`.execute(db);
+    await sql`DROP INDEX IF EXISTS idx_hive_book_stats`.execute(db);
+  },
+};
+
 // APIs
 
 export const createDb = (location: string): { db: Database; sqlite: DatabaseSync } => {