Skip to content

Let the model provider compact long conversations - #1907

Draft
fazxes wants to merge 6 commits into
mainfrom
fazxes/provider-written-compaction
Draft

fazxes wants to merge 6 commits into
mainfrom
fazxes/provider-written-compaction

Conversation

@fazxes

@fazxes fazxes commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • On models whose provider can compact a conversation, fx asks the provider to compact it, and later requests send the provider's block, then the newest 4 turns unchanged with their reasoning.
  • If the provider's compaction fails, fx writes its own summary and keeps no turns after it.
  • After a model switch, the earlier model's kept steps are sent as text.
  • Models without provider compaction keep fx's own compactor.
  • Stopping a compaction while the provider writes its block leaves the conversation as it was.
  • A session killed after the block is saved resumes with the block and the kept turns.
  • A running turn that doesn't fit is compacted whole, and only its message follows the block.
  • A request the provider refuses for its size is compacted with fx's summary and sent again.
  • /compact on these models runs right before the next message is sent.
  • When the provider rejects a compaction because the sign-in expired, fx renews the sign-in once and asks again, as it does for its own summary.
  • The compaction row keeps moving while the provider writes its block.
  • Resuming a session compacted in the middle of a turn no longer drops the steps saved after the compaction.

Why

  • These models bind each reply's reasoning to everything sent before it. fx's own summary changes that start, so the turns fx kept after its summary lost their reasoning. The provider's own block is what the provider reads in place of the older turns, so the newest turns can follow it unchanged.
  • Keeping no turns after fx's summary keeps the reasoning valid, but it drops the newest turns at every compaction. It is the fallback when the provider writes no block, not the main path.
  • The checkpoint saves the provider's block, how many turns it kept, and the system messages those turns were sent with. A resumed session has to send the same request, or the kept turns' reasoning no longer follows what it was written after.

Before and after

  • Before: every compaction sent fx's summary, then the newest turns, whose reasoning no longer matched what came before it.
  • After, on these models: the provider's block, then the newest 4 turns unchanged with their reasoning. Any other model reads fx's exact text of the same turns.
  • /compact on these models now shows "Context will be compacted before your next message." and compacts right before that message is sent, the one request the provider can compact exactly as it was sent.

Tests

  • tests/e2e/provider-compaction.test.ts drives the built binary through a fake AI Gateway: both providers' blocks, each fallback (error statuses, no block, an answer, empty, cut and oversized replies), a model switch, /compact, a stopped compaction, a kill and resume, 30-turn sessions, a running turn compacted whole, and a request refused for its size. compaction-policy.test.ts now counts the provider's compaction request.
  • tests/tla/compaction/ProviderCompaction.tla models the cut, the block, the fallback and the kept turns. check.sh runs TLC on it, shows that the old behavior breaks NoDroppedReasoning, and checks that four planted bugs each break a property. tests/tla/drivers/compaction/ walks the built binary along the model.
  • Two fuzz targets cover the block check and its replay, and the checkpoint's provider block round trip.
  • CI: the PGSO corpus lists the E2E file as a verification scenario, and it has a shard weight.

@fazxes fazxes added the type: improvement Improves existing user-facing behavior label Nov 12, 2026
@fazxes
fazxes force-pushed the fazxes/provider-written-compaction branch 2 times, most recently from d72e656 to cff4c8f Compare November 12, 2026 22:59
- On models whose provider can compact a conversation, fx asks the provider to compact it, and later requests send the provider's block, then the newest 4 turns unchanged with their reasoning.
- If the provider's compaction fails, fx writes its own summary and keeps no turns after it.
- After a model switch, the earlier model's kept steps are sent as text.
- Models without provider compaction keep fx's own compactor.
- Stopping a compaction while the provider writes its block leaves the conversation as it was.
- A session killed after the block is saved resumes with the block and the kept turns.
- A running turn that doesn't fit is compacted whole, and only its message follows the block.
- A request the provider refuses for its size is compacted with fx's summary and sent again.
- `/compact` on these models runs right before the next message is sent.
- Resuming a session compacted in the middle of a turn no longer drops the steps saved after the compaction.
@fazxes
fazxes force-pushed the fazxes/provider-written-compaction branch from e615086 to e0517b3 Compare November 22, 2026 20:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: improvement Improves existing user-facing behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant