fix(models): correct retry semantics and Anthropic streaming - #1651
Merged
Conversation
Always execute model calls at least once when retries is zero or unset, bypass retries for non-recoverable client errors, and retry Anthropic stream creation and consumption as one operation. Add synchronous and asynchronous regression coverage across the affected backends.
git-jxj
marked this pull request as ready for review
August 27, 2026 09:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes three retry issues in model API calls:
retries=0returnedNonewithout invoking the wrapped call, whileretries=NoneraisedTypeError.The changes cover synchronous and asynchronous paths across the affected OpenAI-compatible, OpenAI Responses, Anthropic, and LiteLLM backends.
Root Cause
retry_call()andasync_retry_call()iterated directly overrange(retries). Zero or negative values produced no attempts, andNonewas not accepted byrange().The helpers caught every
Exception, including deterministic 400/401/403/404/422 client failures. Backend-specific handlers only saw those exceptions after all configured attempts and delays had been exhausted.For Anthropic streaming,
messages.create()ran inside the retry helper butcollect_stream_response()ran afterward. Oncecreate()returned a stream object, an interrupted stream was outside the retry boundary.Changes
retriesas the total attempt count, defaultNoneto 3, and clamp values below 1 to one attempt.no_retry_exceptionssupport to synchronous and asynchronous retry helpers.Reproduction
On the unmodified
mainbranch:The new regression suite reports 15 failures against the old source, including all three behaviors above.
After this change:
Validation
Targeted regression tests:
Result:
Model-layer regression suite:
Result:
CI smoke test:
Result:
All configured pre-commit checks also passed:
Scope
This change does not alter request payloads, configured retry intervals, or the number of attempts for values greater than zero. Rate limits and transient transport/server failures remain retryable. The new stream-consumption retry boundary is limited to Anthropic; unrelated backend streaming behavior is unchanged.