Skip to content

fix: wrap extraction transcript turns in speaker tags to stop assistant misattribution#964

Open
gorkem2020 wants to merge 3 commits into
CortexReach:masterfrom
gorkem2020:fix/extraction-speaker-attribution
Open

fix: wrap extraction transcript turns in speaker tags to stop assistant misattribution#964
gorkem2020 wants to merge 3 commits into
CortexReach:masterfrom
gorkem2020:fix/extraction-speaker-attribution

Conversation

@gorkem2020

Copy link
Copy Markdown
Contributor

Problem

The extraction transcript renders turns as User:/Assistant: line prefixes, so only the FIRST line of a message carries a speaker marker. A multi-paragraph assistant reply sheds its marker after the first paragraph, and the extractor attributes assistant-authored plans, preferences, and self-descriptions to the user and stores them as user memories. Live example that motivated this: an assistant's multi-paragraph explanation of how memory layers work came back as "User believes manual notes are only for rare big items" style rows.

Change

  • Each message is wrapped wholly in <user_message>/<assistant_message> blocks, built from the capture hook's role-tagged message-loop order, so every line has an unambiguous owner.
  • The extraction prompt becomes a {system, user} split: the transcript-format teaching and grounding rules ride the system half, the tagged transcript rides the user half. completeJson gains an optional per-call system prompt to carry it (the default generic system message is unchanged for all other callers).
  • Two prompt modes, matching captureAssistant: with false (default) assistant lines never enter the transcript and memories may only be grounded in <user_message> blocks; with true assistant blocks are attributable sources with explicit attribution rules ("attribute every memory to whoever actually said it").
  • Literal speaker tags typed INSIDE a message are neutralized with guillemets, so content can neither fake a block boundary nor defeat tag-boundary trimming.
  • extractMaxChars truncation snaps to a tag boundary, so a sliced transcript never opens with headless text.

Tests

New test/extraction-transcript-speaker-tags.test.mjs (16 tests): whole-message wrapping, multi-paragraph containment, chronology, spoof neutralization, boundary trimming, turn assembly (watermark tail-slice, mixed-role alignment, pair-aware skip, ingress replay fallback), and prompt teaching in both modes. Existing extraction suites updated for the new prompt shape; registered in the test chain and CI manifest.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The speaker-tag direction is sound, and both the full suite and a clean source/dist build pass. One user-facing regression blocks merge:

  • For the supported two-step flow where a user states a fact and then says remember this, the code prepends the prior message only to texts. buildConversationTurnsForExtraction() still receives the original newTexts, and SmartExtractor prefers any nonempty conversationTurns over the flat conversationText. The actual extraction prompt therefore contains only remember this, not the fact it references; if no candidate is produced, the fallback is skipped as well.

Please build the tagged turns from the final text sequence (or explicitly reconstruct the prepended prior turn) and add an integration assertion against the real extraction prompt for this flow.

Two follow-ups are worth covering in the same area: session compression currently changes texts after the turn list is built, so the tagged prompt can bypass the compressor selection; and a single turn longer than extractMaxChars returns a suffix with a closing tag but no opening speaker tag. Neither should be allowed to undermine the new attribution invariant.

@gorkem2020

Copy link
Copy Markdown
Contributor Author

All three items are addressed on the new head:

  1. Remember-this flow: the prepended prior message is now reconstructed as a tagged user turn (the tagged transcript carries the same final sequence as the flat text path), so the real extraction prompt contains both the referenced fact and the command. A new integration test asserts against the captured extraction prompt for exactly this two-step flow.
  2. Compression selection: the tagged transcript now mirrors the final extraction input via a single kept-set filter (a user turn survives only if its text survived session compression and the noise filter alike), so selector-dropped texts can no longer re-enter through the tags. Covered by an integration test where a compression-dropped filler must not reach the prompt.
  3. Truncation: a single turn longer than the budget is re-headed with the opener matching its closing tag, so the attribution invariant survives truncation; genuinely untagged input still passes through unchanged (existing behavior pinned by the prior test).

All three regressions fail on the previous head and pass on this one; tsc, the touched suites, and the full chain are green, and dist is rebuilt at the head.

…ant misattribution

With "User:"/"Assistant:" line prefixes only the first line of a message
carries a speaker marker, so a multi-paragraph assistant reply sheds its
marker after the first paragraph and the extractor attributes
assistant-authored plans and preferences to the user. Wrap every message
wholly in <user_message>/<assistant_message> blocks built from the hook's
role-tagged message-loop order, neutralize literal speaker tags typed inside
message content, and snap extractMaxChars truncation to a tag boundary so a
sliced transcript never leads with headless text.

The extraction prompt becomes a {system, user} split so the format teaching
and grounding rules ride the system half while the transcript rides the user
half: captureAssistant=false grounds memories exclusively in user blocks
(assistant lines never enter the transcript); captureAssistant=true keeps
assistant blocks as attributable sources with explicit attribution rules.
completeJson gains an optional per-call system prompt to carry the split.
…on, and truncation re-heading into the tagged transcript
@gorkem2020
gorkem2020 force-pushed the fix/extraction-speaker-attribution branch from f1a8f32 to f56499e Compare July 24, 2026 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants