🔄 Upstream Sync: LiteLLM v1.95.0 - #128
Conversation
…itellm_/wonderful-northcutt-14b37d
…o shadcn Replaces antd and Tremor with shadcn primitives across the 18 files these three routes exclusively own. Markup only: no behaviour, data flow or copy changed, and no shared or form-bearing component is touched, so the blast radius stops at these pages. The 12 tests covering these components are unchanged from the previous commit and still pass, which is the evidence that the rewrite preserved behaviour. Also prunes the six antd no-restricted-imports suppressions these files no longer need.
A template with no LLM enrichment rendered every parameter field twice: the shared list already covers them, because nonEnrichmentParams is the full parameter list when there is no enrichment, and a second no-enrichment branch mapped the same list again. Predates the shadcn migration and was carried forward by it. The test now asserts exactly one field per parameter, and fails if the duplicate branch comes back.
…itellm_/blissful-torvalds-5a5be3
…emory-9b2c09 refactor(ui): migrate memory page to shadcn
The suggested CIDR chip was a click-only span both before and after the shadcn migration, so keyboard users could not reach or activate it. Render it as a Button, which brings focus and Enter/Space activation with it, and cover the keyboard path with a test that fails against the old span.
25 test functions across three files pass unchanged when every function they execute is mutated; the owning file killed zero of their scored mutants. Four zero-kill tests tied to the fix in BerriAI#31288 are kept for rewrite instead of removal.
…itellm_/test-coverage-mutation-analysis-e42223
…in-3d4c69 refactor(ui): migrate budgets, skills, ui-theme to shadcn
…0e7ee6 refactor(ui): migrate access-groups, vector-stores, organizations to shadcn
…hcutt-14b37d refactor(ui): migrate logging-and-alerts, caching, policies to shadcn
…andalone feat(ui): standalone /connect route for MCP OAuth, decoupled from Chat UI flag
…lds-5a5be3 refactor(ui): migrate mcp-servers, tag-management, tool-policies to shadcn
Registers claude-opus-5 across the cost maps and provider lists so the model prices, reports its real 1M/128K limits, and advertises its capabilities instead of falling through the generalization patterns at zero cost. Adds the first-party entry plus the Bedrock (base, global, us, eu, au, jp), Vertex AI, and Azure AI variants. Pricing matches Opus 4.8 at $5/$25 per MTok with the usual 1.1x regional premium on the cross-region inference profiles, and fast mode is priced at 2x through provider_specific_entry on the first-party entry only. Two fields deliberately differ from Opus 4.8: prompt_cache_min_tokens drops to 512, and bedrock_output_config_effort_ceiling is omitted because Bedrock accepts output_config.effort="max" for Opus 5.
…orks (BerriAI#34512) The a2a completion-bridge tests registered the agent with only custom_llm_provider and model, so the bridge's litellm.acompletion had no api_key and relied on the gateway resolving ANTHROPIC_API_KEY from its ambient env. When that env var is absent, POST /a2a/{id} returns 500 with "Missing Anthropic API Key" and every message/send test (completion bridge, pinned v0.3/v1.0 message shapes, semver serves) fails while the register/discovery/rejection tests still pass. Give A2ABridgeParams an optional api_key and register the bridge agent with api_key="os.environ/ANTHROPIC_API_KEY", matching how the rest of the suite wires anthropic-backed models (e.g. the ratelimit redis tests). The agent now carries its provider key explicitly instead of depending on ambient gateway env.
…itellm_/replace-gpt5-codex-test-27520b
…ss_response (BerriAI#34390) (BerriAI#34405) * fix(guardrails/model_armor): handle None metadata in post_call _process_response On batch routes data["metadata"] is normalized to None (present key, None value), so request_data.get("metadata", {}) returned None and _process_response raised 'NoneType' object has no attribute 'get', 500ing every /v1/batches create with a post_call Model Armor guardrail (regression from v1.93.0 activating the post_call hook). Coalesce a falsy metadata to {} Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Clean up test case documentation Remove regression comment from test_process_response_with_none_metadata_does_not_crash. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
feat(anthropic): add Claude Opus 5
chore(ci): promote internal staging to main
…async_failure_handler (BerriAI#34306) The async streaming error paths fired the sync failure_handler in a thread and the async_failure_handler via create_task at the same time, so both mutated the shared logging object concurrently and could crash pydantic-core. Route failure logging through a single guarded dispatch_failure_handlers, so the sync handler only runs after the async one completes.
…event multi-value wedge (BerriAI#34320) The accumulated-JSON fallback ran json.loads over the whole buffer after every fragment and, on failure, kept the buffer without resetting it. A buffer that ever held more than one concatenated JSON value could never parse (json raises on trailing data), so it returned None on every subsequent chunk while growing without bound - an unrecoverable per-request CPU spin. Parse one value at a time from the front with raw_decode and keep the remainder, draining trailing values on later calls and at end of stream.
…odex-test-27520b test: replace deprecated gpt-5-codex with gpt-5.3-codex
…erriAI#34426) The gateway image installed --extra proxy/proxy-runtime/extra_proxy/semantic-router but not bedrock-realtime, so aws-sdk-bedrock-runtime was absent. Bedrock Nova Sonic speech-to-speech uses InvokeModelWithBidirectionalStream (which boto3 cannot do) via that package, so realtime requests failed at startup with 'Missing aws_sdk_bedrock_runtime'. Add the extra to both uv sync stages; it is already in uv.lock so --frozen resolves, and the package/marker (python>=3.12) matches the python3.13 image.
litellm already supports Google, Microsoft and generic OIDC SSO through fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an unused enum value. This adds real SAML 2.0 single sign-on for the admin UI. A new SAMLAuthHandler validates signed assertions with the OneLogin python3-saml toolkit and maps them onto a CustomOpenID, then reuses the shared post-login path every other provider goes through, so provisioning, role/team mapping and the UI session JWT are unchanged. Both SP-initiated and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are bound to the browser that started them via an HttpOnly state cookie plus a cached AuthnRequest id, and the ACS rejects any response whose InResponseTo doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected by a consumed-assertion guard whose lifetime tracks each assertion's NotOnOrAfter, and both the replay guard and the login-state binding go through the proxy's shared in-memory + Redis cache for multi-instance deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the free-SSO-user Enterprise gate after the assertion is validated, so an unvalidated POST can no longer drive the billable-user count query. SAML is configurable from the admin UI SSO settings (IdP metadata URL or inline XML, SP entity ID, and an allow-unsolicited toggle), which persists the SAML_* environment variables the handler reads, exactly like the Google, Microsoft and generic OIDC providers. python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels bundle the native libraries so no system packages are required, and the import is guarded so the proxy still starts without the package with the SAML routes returning a clear 501. Resolves LIT-4016
…ays stream completed responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…and post_call guardrails (BerriAI#33770) * feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails Pre-call guardrails run sequentially because each may mutate the request payload and later guardrails depend on earlier mutations. Deployments with several slow block-only pre_call guardrails (external moderation, Bedrock, LLM-judge) therefore pay the sum of their latencies. during_call guardrails run concurrently but alongside the LLM call, so a violating payload has already been sent, which is unacceptable when the request must never reach the model. This adds a per-guardrail run_in_parallel flag (default off). Guardrails that opt in are pulled out of the sequential loop and run concurrently via asyncio.gather after every sequential (payload-mutating) guardrail has run, so they observe the mutated payload and still form a hard barrier before the LLM call; the first to raise blocks the request. Their returned data is discarded since they are declared block-only. The flag is wired from LitellmParams onto the guardrail instance at the same generic choke point in initialize_guardrail that already sets skip_system_message_in_guardrail, so no per-provider initializer needs to change. * feat(guardrails): extend run_in_parallel opt-in to post_call guardrails post_call_success_hook ran guardrails sequentially for the same reason pre_call did: response-modifying guardrails thread the response forward. But block-only output scanners (which read the response and reject on violation without changing it) serialize for no benefit and add latency. This reuses the existing run_in_parallel flag for the post_call hook. Opted-in post_call guardrails are pulled out of the sequential loop and run concurrently via asyncio.gather after the sequential (response-modifying) guardrails and before the non-guardrail CustomLogger callbacks, so they inspect the final response and still block it from reaching the client if any raises. Their returned response is discarded since they are block-only. The apply_guardrail path sets data["guardrail_to_apply"] immediately before awaiting, and unified_guardrail pops it before its first suspension point, so concurrent guardrails never race on that key under asyncio's cooperative scheduling. * fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes Addresses review feedback on the run_in_parallel opt-in. asyncio.gather propagated the first exception without cancelling or awaiting the siblings, so a block at t=0 left the other guardrails running as unobserved background tasks (wasted external calls plus event-loop warnings), and a fast SensitiveDataRouteException/ModifyResponseException could return a reroute or passthrough before a slower block finished, letting crafted input bypass the block. Both the pre_call and post_call parallel batches now gather with return_exceptions=True so every guardrail runs to completion, then raise any blocking exception ahead of a flow-changing one. The registry choke point wrote bool(None)==False onto every instance when the config omitted run_in_parallel, silently disabling a constructor-set default; it now only writes when the config provides an explicit value. * fix(guardrails): record lifecycle logs for every concurrently-run guardrail The log_guardrail_information decorator skipped its auto-record when it saw that the count of standard_logging_guardrail_information entries in the shared request_data had grown during the wrapped call, taking that as proof the wrapped function had recorded its own richer entry. That heuristic breaks the moment guardrails run concurrently (parallel pre_call/post_call, during_call): a sibling guardrail's append inflates the shared count, so a guardrail that did not self-record wrongly concludes it already did and drops its own entry. The result is that enabling run_in_parallel silently loses per-guardrail lifecycle logs, so the Admin UI Request Lifecycle timeline and downstream loggers (Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent guardrails. Replace the shared-count heuristic with a ContextVar flag set when a guardrail records its own entry. asyncio copies the context into each gathered task, so the flag is isolated per concurrent guardrail while still catching the self-record-then-skip-auto-record case within a single invocation. * test(guardrails): declare run_in_parallel on post_call guardrail mocks The post_call partition reads run_in_parallel on every CustomGuardrail callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is set in __init__, not on the class) so the attribute access raised, and even a class-level default would return a truthy child mock that wrongly routes the double into the parallel batch. Declare the flag False on the shared mock factories so these pre-existing hook tests exercise the sequential path they assert on. * fix(guardrails): harden run_in_parallel reads and address review feedback Read run_in_parallel via getattr(..., False) in the pre_call and post_call partitions so a third-party CustomGuardrail subclass that overrides __init__ without chaining super().__init__() no longer raises AttributeError on a path that previously worked. Drop the redundant in-function GuardrailEventHooks import in _run_parallel_post_call_guardrails (already imported module-level). Remove the flaky wall-clock upper-bound assertions from the two concurrency tests; the all-start-before-any-end overlap assertion is the timing-independent signal that actually proves concurrency.
…ssages Router.acompletion() takes messages positionally, so splatting a body that omits it raised a TypeError that the generic handler mapped to a 500. Validate the required body param at the routing boundary and raise the existing 400 contract instead. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…p_e2e_poll test(e2e): poll MCP tools across multi-worker lag (BerriAI#35047)
Reverts BerriAI#32005. Team-scoped keys are governed by the team and team-member budgets only; the key owner personal max_budget no longer applies to them, restoring the hierarchy that existed before that PR. The skip_user_budget_on_team_key opt-out existed solely to turn the new behavior back off, so it is removed along with the behavior: the ConfigGeneralSettings field, the /config/list allowed_args entry that surfaced it as an Admin UI toggle, and the argument threaded through reserve_budget_for_request and _get_budget_counters. Regression tests cover both enforcement points in the restored direction: test_common_checks_personal_user_budget_skipped_for_team_key for the read-time check and test_should_not_reserve_user_budget_counter_for_team_key for the optimistic reservation path. (cherry picked from commit 6f1625d)
…rc_1_95_0 chore(release): backport BerriAI#35271 to rc/1.95.0
chore(release): sync rc/1.95.0 with the v1.95.0-rc.1 main SHA
…_landing fix(ui): land general login on the keys dashboard, send MCP consent to /ui/connect (cherry picked from commit ceaf556)
…-rc-1-95-0-f65d9e fix(ui): land general login on the keys dashboard, send MCP consent to /ui/connect (backport BerriAI#35523 to rc/1.95.0)
Automatic sync from upstream BerriAI/litellm tag v1.94.0 Strategy: Merge with history preservation (main syncs to stable tag)
Automatic sync from upstream BerriAI/litellm tag v1.95.0 Strategy: Merge with tree-level conflict resolution (accepted all upstream changes) Conflicts resolved: 151 files (11 rename/rename, 25 modify/delete, 118 rename/delete, 0 content)
|
No description provided. |
🤖 Conflict Resolution StartedStatus: ⏳ In progress... Claude Code (Opus 4.5) is resolving merge conflicts in this PR.
Note This may take 30-90 minutes for large PRs. Resolution commits will be pushed directly to this PR. 📋 Resolution Process (click to expand)
|
Conflicts resolved by Claude Code following CARTO priority rules. Resolution strategy: - Preserved CARTO customizations (workflows, docs, infrastructure) - Accepted upstream improvements (core litellm, tests, dependencies) - Manually merged mixed files (Dockerfile, Makefile) This is a MERGE COMMIT with both main and carto/main as parents, preserving full git history from upstream. Resolves: #128
✅ Conflict Resolution CompleteAll conflicts resolved and pushed to this PR.
Important Ready to merge! Use "Create a merge commit" — do NOT squash or rebase. CARTO Customization DecisionsSummary
Preserved CARTOFiles where CARTO implementation was kept (verbatim from carto/main):
Synced (Required)Files synced entirely from upstream (no CARTO customizations needed):
Fix Loop InterventionsFiles synced due to repeated conflicts:
Verification Steps Completed
Next Steps
🔧 Workflow Details (click to expand)Workflow Run: https://github.com/CartoDB/litellm/actions/runs/31387990410 |
|
Caution
|
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
|
| Decision | Count |
|---|---|
| Upstream Substitutes | 0 |
| Customized Upstream | 0 |
| Preserved CARTO | 11 |
| Incorrectly Dropped | 1 |
Overall Assessment: NEEDS_ATTENTION
🔧 Auto-fix enabled: The fix job will run next to restore dropped features.
📋 Full details in PR description above.
🔧 CARTO Feature Fix StartedRestoring 1 incorrectly dropped CARTO feature(s). |
…ay content flattening PR #112 added _content_to_text_string to flatten array-form message content to strings (required by Cortex error 390142), but the call sites were lost during upstream sync conflict resolution The function was defined but never called, meaning Cortex would reject array-form content like [{"type": "text", "text": "..."}] with error 390142 on multi-turn conversations where the OpenAI Agents SDK replays prior assistant turns in list-of-blocks form Added _flatten_messages_content helper that transforms all messages by flattening array-form content to strings, wired into _transform_request_openai before sending to the /chat/completions endpoint Verified: syntax check, import check, 16 existing Snowflake tests pass
🔧 CARTO Feature Fix CompleteSummaryRestored 1 CARTO feature (PR #112 - array content flattening) by adding call sites for the orphaned _content_to_text_string function Decisions MadePR #112: Snowflake Cortex Array Content FlatteningDecision: The original PR #112 modified a _transform_messages method that doesn't exist in the current architecture. The current file uses _transform_request_openai for the OpenAI/chat completions path. Created _flatten_messages_content helper that iterates over messages and flattens array-form content to strings using the existing _content_to_text_string function. This is called in _transform_request_openai before sending messages to Snowflake Cortex. Next Steps:
|
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
|
Closing in favor of a fresh sync straight to the latest upstream stable (v1.98.0). v1.95.0 is already several releases behind, so rather than merge it and wait for the next scheduled run to produce v1.98.0, we're skipping ahead. A new upstream-sync PR will be created by a manual run of the sync workflow. |
🔄 Upstream Sync: LiteLLM v1.95.0
Syncs CARTO's LiteLLM fork with upstream stable release v1.95.0.
1.92.0→v1.95.0Caution
Use "Create a merge commit" only. Squashing destroys upstream history and breaks future syncs.
🧪 Pre-Merge Checklist
pyproject.tomlversion matches upstream📊 Release Information (click to expand)
v1.95.01.92.0🔀 Branch Flow (click to expand)
BerriAI/litellm:mainmerged intoCartoDB/litellm:mainupstream-sync/v1.95.0upstream-sync/v1.95.0→carto/main📝 CARTO-Specific File Guidelines (click to expand)
When reviewing or resolving conflicts:
✅ Keep CARTO Versions (Ours)
.github/workflows/carto_*.yaml- CARTO workflows.github/workflows/carto-*.yml- CARTO workflowsCARTO_*.md,docs/CARTO_*.md- CARTO documentation🔄 Accept Upstream (Theirs)
pyproject.toml- Version fieldlitellm/- Core library codetests/- Upstream testsrequirements.txt- DependenciesDockerfile,docker/Dockerfile.non_root- CARTO customizationsMakefile- Check# CARTO:sections🔧 Conflict Resolution (click to expand)
If this PR has conflicts:
Option 1: Automated (Recommended)
The carto-upstream-sync-resolver workflow triggers automatically.
What it does:
carto/main→ ✏️ Resolves conflicts → 🧪 Runs tests → 📌 Pushes to this PRYou just need to: Wait for resolution commits, verify CARTO customizations, merge.
Option 2: Manual Resolution
📚 Documentation Links (click to expand)
🤖 This PR was automatically created by the carto-upstream-sync workflow.
🔧 CARTO Feature Fixes Applied
Status: ✅ Fixed
Features Restored: 1
PR #112: Snowflake Cortex Array Content Flattening
Fixed: 2026-08-10 13:06:28 UTC
Workflow Run: #41
CARTO Customizations Analysis
Overall Assessment: ✅ PASS
CARTO Feature Preservation Analysis
Summary
Overall Assessment: PASS
All 16 analyzed CARTO features have been correctly preserved during the v1.95.0 upstream sync. The conflict resolution maintained full wiring for all CARTO customizations, with no orphaned helpers or broken call sites detected.
Feature Details
Customized Upstream (4)
These features combined upstream infrastructure with CARTO-specific enhancements:
PR fix(rate-limiter-v3): constant hash tag to avoid CROSSSLOT on Redis Cluster #119 - Redis Cluster CROSSSLOT Fix - Upstream rate limiter v3 base preserved; CARTO added hash tag grouping (
REDIS_NODE_HASHTAG_NAME,_group_keys_by_hash_tag) for cluster compatibilityPR fix(docker): fetch arm64 prisma schema-engine for non_root image #118 - ARM64 Prisma Schema-Engine - Upstream Dockerfile structure preserved; CARTO wolfi-base image and arm64 schema-engine fetch integrated
PR fix(oci): Add tool calling support for OCI Gemini streaming #68 - OCI Gemini Tool Call UUIDs - Upstream modular OCI helpers; CARTO setdefault patterns for tool call ID generation integrated
PR feat(docker): multi-arch builds (AMD64 + ARM64) for non_root image #90/multi-arch Docker - Upstream Docker workflow structure; CARTO multi-arch build matrix integrated
Preserved CARTO (12)
These are CARTO-only features with no upstream equivalent:
Snowflake/Cortex (5 features):
_tool_index)_content_to_text_string,_flatten_messages_content) - recently fixed in commit ee89405_strip_openai_annotations)SnowflakeStreamingHandler)CARTO: skip path construction...comment)Databricks (2 features):
_strip_openai_annotations)_normalize_empty_tool_call_arguments)OCI (2 features):
_reorder_tool_results_to_match_tool_calls)Other Core (3 features):
_validate_and_repair_tool_arguments)_store_session_in_redis,_patch_store_session_in_redis)CI/CD Infrastructure (preserved, not counted above)
All CARTO-specific workflows are preserved:
Issues Found
None. All CARTO features are correctly preserved with verified wiring.
Wiring Verification Notes
The analysis verified not just string presence but actual call site wiring for critical features:
_content_to_text_string→ called via_flatten_messages_contentat line 367_strip_openai_annotations(Snowflake) → called at lines 308, 333_strip_openai_annotations(Databricks) → called at line 478_normalize_empty_tool_call_arguments→ called at line 479_validate_and_repair_tool_arguments→ called at lines 351, 394_reorder_tool_results_to_match_tool_calls→ called at line 237_store_session_in_redis→ called at line 901, chains to_patch_store_session_in_redisRecent Fix Applied
Commit ee89405 restored the
_content_to_text_stringcall sites that were lost during the initial sync conflict resolution. This was caught by the wiring-aware analysis approach documented in PR #126.Feature-by-Feature Breakdown
PR #127: fix(gh-workflows): wait long enough for the release image digest to resolve
PR #126: docs(ci): align sync resolver/fixer/analyzer prompts with wiring-aware reasoning
PR #119: fix(rate-limiter-v3): constant hash tag to avoid CROSSSLOT on Redis Cluster
PR #118: fix(docker): fetch arm64 prisma schema-engine for non_root image
PR #116: fix(snowflake): assign distinct index per streamed Cortex tool call
PR #112: fix(snowflake): flatten array-form message content for Cortex
PR #111: fix(snowflake): unblock Claude function-calling follow-up turns on Cortex
PR #110: fix: strip OpenAI 'annotations' field from outbound Databricks chat messages
PR #109: fix: stop Databricks streaming from rewriting tool_call arguments '{}' to ''
PR #108: fix(oci): restore inline-PEM key normalization dropped during upstream sync
PR #70: fix(azure): Strip operation suffixes from deployment URLs to prevent 404 errors
PR #68: fix(oci): Add tool calling support for OCI Gemini streaming
PR #54: fix: repair malformed JSON in streaming tool call arguments
PR #38: fix: Snowflake PAT auth and Claude streaming support
PR #5: Jatorre/fix/responses api redis session timing
PR #121: OCI Parallel Tool Result Reordering
Analyzed: 2026-08-10 13:21:55 UTC
Workflow Run: #42
Analysis Artifacts: Download JSON/MD
Method: Claude Code (Opus 4.5) post-resolution semantic analysis