Skip to content

fix(models): correct retry semantics and Anthropic streaming - #1651

Merged
Yunnglin merged 1 commit into
modelscope:mainfrom
git-jxj:fix/retry-semantics
Aug 28, 2026
Merged

fix(models): correct retry semantics and Anthropic streaming#1651
Yunnglin merged 1 commit into
modelscope:mainfrom
git-jxj:fix/retry-semantics

Conversation

@git-jxj

@git-jxj git-jxj commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes three retry issues in model API calls:

  • retries=0 returned None without invoking the wrapped call, while retries=None raised TypeError.
  • Non-recoverable client errors were retried with the normal delay before reaching backend error handling.
  • Anthropic streaming requests were only protected while creating the stream; failures while consuming it escaped without retrying.

The changes cover synchronous and asynchronous paths across the affected OpenAI-compatible, OpenAI Responses, Anthropic, and LiteLLM backends.

Root Cause

retry_call() and async_retry_call() iterated directly over range(retries). Zero or negative values produced no attempts, and None was not accepted by range().

The helpers caught every Exception, including deterministic 400/401/403/404/422 client failures. Backend-specific handlers only saw those exceptions after all configured attempts and delays had been exhausted.

For Anthropic streaming, messages.create() ran inside the retry helper but collect_stream_response() ran afterward. Once create() returned a stream object, an interrupted stream was outside the retry boundary.

Changes

  • Treat retries as the total attempt count, default None to 3, and clamp values below 1 to one attempt.
  • Add no_retry_exceptions support to synchronous and asynchronous retry helpers.
  • Bypass retries for non-recoverable 400/401/403/404/422 SDK errors.
  • Keep rate limits, conflicts, timeouts, connection failures, and server errors retryable.
  • Apply the exception policy to OpenAI-compatible, OpenAI Responses, Anthropic, and LiteLLM call sites.
  • Move Anthropic stream creation and complete stream consumption into one retryable operation.
  • Discard partial output from failed Anthropic attempts before issuing a fresh request.
  • Add synchronous and asynchronous regression tests for helper semantics, backend exception policies, and interrupted Anthropic streams.

Reproduction

On the unmodified main branch:

retry_call(lambda: "x", retries=0)    -> None
retry_call(lambda: "x", retries=None) -> TypeError
BadRequest with retries=5              -> 5 attempts
Authentication/NotFound, retries=5     -> 5 attempts
Anthropic interrupted stream           -> ConnectionError after attempt 1

The new regression suite reports 15 failures against the old source, including all three behaviors above.

After this change:

retries=0                              -> one call, returns "x"
retries=None                           -> uses the default attempt count
non-recoverable client error           -> one attempt
transient connection error             -> retries to the configured attempt count
Anthropic interrupted stream           -> attempt 2 returns the complete response

Validation

Targeted regression tests:

python -m pytest \
  tests/test_function_utils.py \
  tests/models/test_no_retry_exceptions.py \
  tests/models/test_anthropic_stream_retry.py -q

Result:

58 passed

Model-layer regression suite:

python -m pytest tests/models -q

Result:

78 passed

CI smoke test:

python -m pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings

Result:

1 passed

All configured pre-commit checks also passed:

flake8, ruff check, yapf, trailing whitespace, YAML, end-of-file,
requirements, quote normalization, merge-conflict, and line-ending checks

Scope

This change does not alter request payloads, configured retry intervals, or the number of attempts for values greater than zero. Rate limits and transient transport/server failures remain retryable. The new stream-consumption retry boundary is limited to Anthropic; unrelated backend streaming behavior is unchanged.

Always execute model calls at least once when retries is zero or unset, bypass retries for non-recoverable client errors, and retry Anthropic stream creation and consumption as one operation. Add synchronous and asynchronous regression coverage across the affected backends.
@git-jxj
git-jxj marked this pull request as ready for review August 27, 2026 09:53

@Yunnglin Yunnglin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Yunnglin
Yunnglin merged commit bdf1892 into modelscope:main Aug 28, 2026
3 checks passed
@git-jxj
git-jxj deleted the fix/retry-semantics branch August 29, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants