Skip to content

feat(transfer): carry Knowledge Pages tree + regenerate mental-model search state on import (#3308, #3323) - #3330

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/whole-bank-transfer-knowledge-pages-3308
Aug 10, 2026
Merged

feat(transfer): carry Knowledge Pages tree + regenerate mental-model search state on import (#3308, #3323)#3330
nicoloboschi merged 1 commit into
mainfrom
fix/whole-bank-transfer-knowledge-pages-3308

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #3308. Closes #3323.

Problem

Whole-bank export/import produced a logically incomplete bank:

  1. The Knowledge Pages tree was dropped entirelyknowledge_pages sat in _SKIP_TABLES because its self-referential parent_id FK needs a parents-first restore the generic per-row restorer didn't provide. Folders, page nodes, parent_id, mental_model_id, managed, and sort_order were all lost; backing mental models survived but were disconnected from any page.
  2. Restored mental models had no derived search state — export strips embedding/search_vector (target-derived), but import restored the rows without regenerating either. So imported knowledge pages had a NULL embedding (vector arm empty on every backend) and, under vchord, empty lexical state — i.e. unsearchable (this is Bank import does not recompute mental_models embedding / search_vector — knowledge search degraded after import #3323).

Fix

Export (transfer/export.py, schema.py)

  • Add a typed TransferKnowledgePage model (no raw dicts across phases, per the issue) and carry the tree in knowledge_pages.json, emitted parent-first via a recursive walk, preserving id, parent_id, mental_model_id, managed, sort_order, name, and timestamps.
  • Remove knowledge_pages from _SKIP_TABLES; classify it under a new KNOWLEDGE_TABLES bucket (the test_export_bank_covers_schema guard is updated so a future table still can't be silently dropped).

Import (transfer/importer.py)

  • Regenerate each restored mental model's embedding with the target model, embedding the same "{name} {content}" text create_mental_model uses. Computed off-connection so no DB connection is held across the embedding call (the established retain-path rule).
  • Rebuild backend-specific lexical state through the shared pg_search_vector_expr — the single source of truth the live mental-model writes and the memory-recall path use. Only vchord needs an explicit write (its bm25vector column); native's search_vector is GENERATED and repopulates on insert, and pg_search/pg_textsearch/pgroonga index the base name/content columns.
  • Restore the tree after its backing mental models exist (page→mm FK) and parents-first (parent_id self-FK), via a topological order that tolerates cycles/dangling parents (emits them so the DB FK — not a silent drop — surfaces a corrupt export). ON CONFLICT DO NOTHING keeps the import idempotent.

import_bank reports a new knowledge_pages_imported count (surfaced by the admin CLI). No HTTP surface changed — whole-bank import is admin-CLI only, so no OpenAPI/client regen.

Tests

  • Whole-bank roundtrip (test_bank_roundtrip_carries_knowledge_pages): builds nested folders + pages (including a root-level page and a managed folder), exports → deletes → imports, then asserts the tree restores exactly (ids, parent_id, mental_model_id, managed), that pages are searchable after import, and that no restored mental model has a NULL embedding.
  • Non-DB unit tests for the topological ordering (parent-first; tolerant of cycles/dangling parents).
  • Ran the full test_document_transfer.py suite (28 tests) green on an isolated pg0 instance.

Scope note

Verified searchable-after-import end-to-end on the native backend (the default test backend). The vchord/base-column lexical regeneration goes through the same pg_search_vector_expr dispatch already unit-tested per backend in #3268; a live vchord backend isn't part of the transfer test matrix.

@JiehoonKwak

Copy link
Copy Markdown
Contributor

I compared this against #3315; this PR is the better, smaller lane and covers the material transfer behavior (typed tree, target-bank remap, target embeddings/search projection, parent-first restore, history/config/webhook paths already present on main).

One safety delta from #3315 seems worth folding in before merge: validate the typed Knowledge Page graph and backing-model references before the first import write.

Today _load_knowledge_pages() walks only from roots, so a cyclic component would be omitted from the archive. On import, _topological_page_order() deliberately emits dangling/cyclic leftovers and relies on the FK to fail, but that happens after the bank/documents/facts have already been restored (and their embeddings paid for), leaving a partially imported target.

A narrow preflight can reject, before any write/provider call:

  • duplicate/missing node ids, dangling parents, and parent cycles;
  • non-folder parents;
  • folders carrying mental_model_id;
  • pages without a carried backing mental model.

The corresponding regression should assert a corrupt archive causes zero writes and zero embedding calls. This does not require changing the valid parent-first restore path or broadening this PR.

…l search state on import (#3308, #3323)

Whole-bank export/import previously dropped the Knowledge Pages tree
(knowledge_pages was in _SKIP_TABLES because its self-referential parent_id
FK needs a topological restore) and restored mental models without an
embedding or lexical search state — leaving imported knowledge pages
disconnected and unsearchable, on every text-search backend.

Export:
- Add a typed TransferKnowledgePage model (no raw dicts across phases) and
  carry the folder/page tree in knowledge_pages.json, parent-first, preserving
  id, parent_id, mental_model_id, managed, sort_order, name and timestamps.
- Remove knowledge_pages from _SKIP_TABLES; classify it under a new
  KNOWLEDGE_TABLES bucket (coverage guard updated).

Import:
- Regenerate each restored mental model's embedding with the TARGET model
  (same "{name} {content}" text create_mental_model embeds), off-connection so
  no DB conn is held across the embedding call.
- Rebuild backend-specific lexical state via the shared pg_search_vector_expr
  (vchord's bm25vector column; native's is GENERATED and repopulates on insert;
  pg_search/pg_textsearch/pgroonga index base columns).
- Restore the tree after its backing mental models exist and parents-first
  (topological order tolerant of cycles/dangling parents), ON CONFLICT DO NOTHING.

Tests: whole-bank roundtrip asserts the nested tree restores exactly (ids,
parents, mm refs, managed) and pages are searchable after import with no NULL
mental-model embeddings; plus non-DB unit tests for the topological ordering.
@nicoloboschi
nicoloboschi force-pushed the fix/whole-bank-transfer-knowledge-pages-3308 branch from 909c645 to da148d4 Compare August 10, 2026 12:01
@nicoloboschi
nicoloboschi merged commit fd29422 into main Aug 10, 2026
209 of 210 checks passed
@nicoloboschi
nicoloboschi deleted the fix/whole-bank-transfer-knowledge-pages-3308 branch August 10, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants