Skip to content

fix(llm): remove top-level cache_control from Claude provider (fixes #2512) #2521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nankingjing wants to merge 1 commit into
base: main
Choose a base branch
from
+0 −1
Open

fix(llm): remove top-level cache_control from Claude provider (fixes #2512) #2521

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion src/llm/providers/claude.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ def generate(self, input: LLMInput) -> LLMOutput:
"model": model,

@greptile-apps greptile-apps Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Prompt Caching Becomes Unreachable

When generate() builds Claude requests, callers only pass plain LLMInput messages and Message.to_dict() does not expose a way to add per-block cache_control. Removing the provider-level cache setting means existing Claude calls lose the only automatic cache breakpoint in this path, so long repeated prompts can stop producing cache token usage and pay full input cost on every request.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@nankingjing nankingjing Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against the SDK floor this repo pins (anthropic>=0.111.0): top-level cache_control on messages.create() is the documented automatic prompt caching parameter on the first-party Anthropic API - the SDK auto-places the breakpoint on the last cacheable block - so it is accepted, not rejected, on every SDK version the project supports. That makes this finding substantively right: the removal trades away automatic caching to fix an error that does not reproduce on a pin-compliant install. The failure reported in #2512 points to an out-of-range environment (an SDK older than the pin, or a platform without automatic caching such as Bedrock/Vertex). Deliberately not adding a regression test asserting the parameter's absence, since that would enshrine the caching regression; flagging for maintainers to confirm the #2512 reproduction environment before merging this or #2515.

"messages": api_messages,
"max_tokens": input.max_tokens if input.max_tokens else 16000,
"cache_control": {"type": "ephemeral"},
}
if system_parts:
params["system"] = "\n\n".join(system_parts)
Expand Down