[codex] support configurable fast compaction routing#4689
Conversation
LLM summary compaction previously only had per-model overrides plus the active and fallback model chain, which made a dedicated global compaction target awkward and gave OpenAI Codex sessions no safe fast default. This adds modelRoles.compaction and an inferred provider-local Spark candidate while preserving currentModel.compactionModel as the most specific override. The inferred fast candidate is filtered before execution when the prepared input exceeds usable context or needs unsupported image input, and preparation is rerun after filtering so remote preserve-data reuse matches the actual execution candidates.
…omics Provider-native remote compaction replays the session's exact native history, which the session model serves mostly at the ~10x cheaper cacheRead rate; prefix caches do not transfer across models, and the inferred Spark candidate shares the session model's list price. Defaulting metered API-key sessions onto it silently traded a warm cached replay for a full-price cold prefill. The inferred default fast candidate is now only used when the compaction wire is the flat-rate openai-codex-responses subscription backend, when the session model would not use provider-native replay anyway (the local summary prompt never shared a prefix with the session cache), or when the candidate's catalog input price does not exceed the session model's cacheRead price. Explicit compactionModel and modelRoles.compaction targets keep bypassing the gate, and the existing input-fit and modality filters are unchanged.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
roboomp
left a comment
roboomp
left a comment
There was a problem hiding this comment.
P2: scoped feature work, but it changes default compaction model routing, so it needs maintainer sign-off on the default flip.
Found one blocking correctness gap: image-bearing fileMention entries are not detected before selecting the text-only inferred Spark candidate. Also flagged the default-enabled setting as a maintainer-call.
Verification: attempted bun test packages/coding-agent/test/compaction-prefer-current-model.test.ts packages/coding-agent/test/agent-session-goal-midrun-compaction.test.ts; it failed before running tests because packages/coding-agent/src/export/html/index.ts could not resolve ./tool-views.generated.js.
Thanks for the detailed cache-economics writeup and focused tests.
| ].some(message => this.#messageHasImageContent(message)); | ||
| } | ||
|
|
||
| #messageHasImageContent(message: AgentMessage): boolean { |
There was a problem hiding this comment.
blocking: #messageHasImageContent() does not detect fileMention image attachments (message.files[].image). convertToLlm() later turns those into ImageContent blocks (see packages/coding-agent/src/session/messages.ts fileMention case), so an inferred text-only Spark candidate can still be selected for a compaction input that includes @image files. That violates this PR's "skip before any provider request when the prepared compact input needs unsupported image input" contract and will route image-bearing compaction to a text-only model.
| "compaction.preferFastModel": { | ||
| type: "boolean", | ||
| default: true, |
There was a problem hiding this comment.
should-fix / maintainer-call: default: true changes default compaction routing from the active session model to an inferred fast model in subscription, remote-disabled, and zero-cost cases. Repo review guidance treats default-behaviour changes as requiring explicit maintainer sign-off; if this is intentional, the maintainer should confirm the default flip rather than shipping it only as an opt-out setting.
|
Already using remote compaction so do we need it? |
3 participants
Summary
Adds configurable routing for LLM summary compaction so users can set a dedicated
modelRoles.compactiontarget while retaining the existing per-modelcompactionModeloverride.Also adds
compaction.preferFastModel, enabled by default, to try a provider-local fast OpenAI Codex Spark model such asgpt-5.3-codex-sparkbefore falling back to the active session model. The inferred fast candidate is skipped before any provider request when the prepared compact input exceeds its usable context, needs unsupported image input, or when rerouting would forfeit cached input pricing on a metered provider.Supersedes #4519 (auto-closed pre-vouch, can't be reopened by the author); includes the fix for the prompt-caching concern raised in the vouch discussion (#4520).
Why
OpenAI Codex sessions should be able to use a cheaper/faster compact target by default without hardcoding generated catalog policy or forcing users to duplicate model-selection semantics under a separate setting.
modelRolesalready handles custom providers, auth-aware resolution, canonical model ids, and role aliases.Prompt-cache safety
Provider-native remote compaction (V1
/responses/compactand V2 streaming) replays the session's exact native history — input the session model serves mostly at the ~10x cheapercacheReadrate — and prefix caches don't transfer across models. Spark's list price matches codex's, so defaulting metered API-key sessions onto it would have traded a warm cached replay for a full-price cold prefill.The inferred fast candidate is therefore only used when at least one of these holds:
openai-codex-responses(flat-rate ChatGPT subscription; spark is a pure latency win there),cacheReadprice (covers zero-cost catalogs, and auto-unlocks if spark pricing ever drops below cached-input rates).Explicit
compactionModel/modelRoles.compactiontargets bypass the gate — routing compaction to a specific model is the user accepting the trade-off. The live session's own cache is untouched either way: compaction is a side request, and the post-compaction cold start on the next turn is inherent to rewriting the transcript, not this change.Implementation
modelRoles.compactionas a built-in role.compaction.preferFastModelwith defaulttrue.compact()'s V2-then-V1 remote gating for the session model.docs/compaction.md.Validation
bun test packages/coding-agent/test/compaction-prefer-current-model.test.ts packages/coding-agent/test/compaction.test.ts packages/coding-agent/test/agent-session-goal-midrun-compaction.test.ts— 67 pass; new coverage for metered-blocked, subscription-allowed, remote-disabled, zero-cost catalog, explicit-override, and mid-turn metered/subscription casesbun run checkinpackages/coding-agentbun run checkinpackages/agentbun run check:tsat repo root