feat(transfer): carry Knowledge Pages tree + regenerate mental-model search state on import (#3308, #3323) - #3330
Conversation
|
I compared this against #3315; this PR is the better, smaller lane and covers the material transfer behavior (typed tree, target-bank remap, target embeddings/search projection, parent-first restore, history/config/webhook paths already present on main). One safety delta from #3315 seems worth folding in before merge: validate the typed Knowledge Page graph and backing-model references before the first import write. Today A narrow preflight can reject, before any write/provider call:
The corresponding regression should assert a corrupt archive causes zero writes and zero embedding calls. This does not require changing the valid parent-first restore path or broadening this PR. |
…l search state on import (#3308, #3323) Whole-bank export/import previously dropped the Knowledge Pages tree (knowledge_pages was in _SKIP_TABLES because its self-referential parent_id FK needs a topological restore) and restored mental models without an embedding or lexical search state — leaving imported knowledge pages disconnected and unsearchable, on every text-search backend. Export: - Add a typed TransferKnowledgePage model (no raw dicts across phases) and carry the folder/page tree in knowledge_pages.json, parent-first, preserving id, parent_id, mental_model_id, managed, sort_order, name and timestamps. - Remove knowledge_pages from _SKIP_TABLES; classify it under a new KNOWLEDGE_TABLES bucket (coverage guard updated). Import: - Regenerate each restored mental model's embedding with the TARGET model (same "{name} {content}" text create_mental_model embeds), off-connection so no DB conn is held across the embedding call. - Rebuild backend-specific lexical state via the shared pg_search_vector_expr (vchord's bm25vector column; native's is GENERATED and repopulates on insert; pg_search/pg_textsearch/pgroonga index base columns). - Restore the tree after its backing mental models exist and parents-first (topological order tolerant of cycles/dangling parents), ON CONFLICT DO NOTHING. Tests: whole-bank roundtrip asserts the nested tree restores exactly (ids, parents, mm refs, managed) and pages are searchable after import with no NULL mental-model embeddings; plus non-DB unit tests for the topological ordering.
909c645 to
da148d4
Compare
Fixes #3308. Closes #3323.
Problem
Whole-bank export/import produced a logically incomplete bank:
knowledge_pagessat in_SKIP_TABLESbecause its self-referentialparent_idFK needs a parents-first restore the generic per-row restorer didn't provide. Folders, page nodes,parent_id,mental_model_id,managed, andsort_orderwere all lost; backing mental models survived but were disconnected from any page.embedding/search_vector(target-derived), but import restored the rows without regenerating either. So imported knowledge pages had a NULL embedding (vector arm empty on every backend) and, under vchord, empty lexical state — i.e. unsearchable (this is Bank import does not recompute mental_models embedding / search_vector — knowledge search degraded after import #3323).Fix
Export (
transfer/export.py,schema.py)TransferKnowledgePagemodel (no raw dicts across phases, per the issue) and carry the tree inknowledge_pages.json, emitted parent-first via a recursive walk, preservingid,parent_id,mental_model_id,managed,sort_order,name, and timestamps.knowledge_pagesfrom_SKIP_TABLES; classify it under a newKNOWLEDGE_TABLESbucket (thetest_export_bank_covers_schemaguard is updated so a future table still can't be silently dropped).Import (
transfer/importer.py)"{name} {content}"textcreate_mental_modeluses. Computed off-connection so no DB connection is held across the embedding call (the established retain-path rule).pg_search_vector_expr— the single source of truth the live mental-model writes and the memory-recall path use. Only vchord needs an explicit write (itsbm25vectorcolumn); native'ssearch_vectoris GENERATED and repopulates on insert, and pg_search/pg_textsearch/pgroonga index the basename/contentcolumns.parent_idself-FK), via a topological order that tolerates cycles/dangling parents (emits them so the DB FK — not a silent drop — surfaces a corrupt export).ON CONFLICT DO NOTHINGkeeps the import idempotent.import_bankreports a newknowledge_pages_importedcount (surfaced by the admin CLI). No HTTP surface changed — whole-bank import is admin-CLI only, so no OpenAPI/client regen.Tests
test_bank_roundtrip_carries_knowledge_pages): builds nested folders + pages (including a root-level page and amanagedfolder), exports → deletes → imports, then asserts the tree restores exactly (ids,parent_id,mental_model_id,managed), that pages are searchable after import, and that no restored mental model has a NULL embedding.test_document_transfer.pysuite (28 tests) green on an isolated pg0 instance.Scope note
Verified searchable-after-import end-to-end on the native backend (the default test backend). The vchord/base-column lexical regeneration goes through the same
pg_search_vector_exprdispatch already unit-tested per backend in #3268; a live vchord backend isn't part of the transfer test matrix.