fix(chapters): zip-level TOC fallback + extraction-attempt marker#126
Merged
Conversation
Prod's first Word Counts run left 8 EPUBs permanently "pending" and chapterless — real books (Tolkien, Overlord, an omnibus) whose TOCs chapter extraction couldn't see. Two defects: - Extraction depended entirely on ebooklib's nav parsing, which fails three ways on real files: EPUB2 books whose TOC lives only in the NCX; files whose broken/EMPTY nav shadows a good NCX (book.toc comes back as a bare empty Link); and malformed navs (no <ol>) that crash read_epub outright. Fix: the fraction math is factored out (_build_chapter_map), ebooklib's toc shapes are normalized (bare Link, node tuples, a single wrapping Section unwraps to its children), and a zip-level parser (_chapters_from_zip — container→OPF→spine word counts, TOC from NCX navPoints or the EPUB3 nav) takes over whenever ebooklib yields nothing or refuses the file. Rescues 5 of dev's 7 holdouts; the other two are genuinely TOC-less files (empty <navMap/>, single "Start" entry). - TOC-less EPUBs re-queued in the backfill forever, because "done" was inferred from having BookChapter rows they can never gain. New Book.chapters_extracted_at stamps every attempt (found or not); replace_book_chapters distinguishes None (never ran — non-EPUB callers) from [] (ran, nothing there — stamp only, keep existing rows) from [...] (replace + stamp). The migration marks already-chaptered books done so upgraded installs don't re-extract their whole library. Suite green (5 new tests: NCX-only EPUB2 fixture end-to-end, the []-in-meta ingest contract, marker semantics incl. stale-TOC protection, the pending predicate). Verified against the actual failing prod/dev files: Overlord volumes now yield 8-15 chapters, Cloud FinOps 34. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prod's first Word Counts run surfaced 8 EPUBs stuck permanently "pending" with no chapter maps — Tolkien, four Overlord volumes, an omnibus. Two defects from the time-per-chapter work:
ebooklib-only TOC parsing misses real-world files
Three observed failure shapes: EPUB2 books whose TOC lives only in the NCX; files whose broken/empty nav shadows a good NCX (
book.toccomes back as a bare emptyLink); and malformed navs (no<ol>) that crashread_epuboutright.Fix: fraction math factored into
_build_chapter_map; ebooklib toc shapes normalized (bare Link, node tuples, single wrapping Section unwraps to its children); and a zip-level parser (_chapters_from_zip: container → OPF → spine word counts, TOC from NCX navPoints or the EPUB3 nav) takes over whenever ebooklib yields nothing or refuses the file — the same fallback strategy word counting already uses.Rescues 5 of the 7 remaining dev holdouts (Overlord ×4 now 8–15 chapters, Cloud FinOps 34). The last two are genuinely TOC-less files (empty
<navMap/>; a single "Start" entry) — correctly no chapters.TOC-less EPUBs re-queued in the backfill forever
"Done" was inferred from having
BookChapterrows, which a TOC-less book can never gain — so it re-parsed on every run and the pending counter never reached zero (the "8 Counted / 0 Words found" screenshot). NewBook.chapters_extracted_atstamps every attempt;replace_book_chaptersdistinguishesNone(never ran) /[](ran, nothing there — stamp, keep existing rows) /[...](replace + stamp). The migration marks already-chaptered books done, so upgraded installs don't re-extract their library.Verification
Suite green (5 new tests: NCX-only EPUB2 fixture end-to-end, the
[]-in-meta ingest contract, marker semantics incl. stale-TOC protection, the pending predicate). Verified directly against the failing prod/dev files.After merge + redeploy: one more Admin → Word Counts run picks up the rescued books.