Skip to content

🔄 Upstream Sync: LiteLLM v1.95.0 - #128

Closed
Cartofante wants to merge 3745 commits into
carto/mainfrom
upstream-sync/v1.95.0
Closed

🔄 Upstream Sync: LiteLLM v1.95.0#128
Cartofante wants to merge 3745 commits into
carto/mainfrom
upstream-sync/v1.95.0

Conversation

@Cartofante

@Cartofante Cartofante commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

🔄 Upstream Sync: LiteLLM v1.95.0

Syncs CARTO's LiteLLM fork with upstream stable release v1.95.0.

Metric Value
Version 1.92.0v1.95.0
Commits 3743
Files Changed 3924
Upstream Release v1.95.0

Caution

⚠️ DO NOT SQUASH MERGE THIS PR

Use "Create a merge commit" only. Squashing destroys upstream history and breaks future syncs.


🧪 Pre-Merge Checklist

  • CI checks pass (lint, tests, Docker build)
  • CARTO customizations preserved
  • pyproject.toml version matches upstream

📊 Release Information (click to expand)
🔀 Branch Flow (click to expand)
  1. BerriAI/litellm:main merged into CartoDB/litellm:main
  2. ✅ Created dedicated sync branch: upstream-sync/v1.95.0
  3. 📝 This PR: upstream-sync/v1.95.0carto/main

[!NOTE]
Why a dedicated branch? Allows pushing conflict resolution commits directly to this PR.

📝 CARTO-Specific File Guidelines (click to expand)

When reviewing or resolving conflicts:

✅ Keep CARTO Versions (Ours)

  • .github/workflows/carto_*.yaml - CARTO workflows
  • .github/workflows/carto-*.yml - CARTO workflows
  • CARTO_*.md, docs/CARTO_*.md - CARTO documentation

🔄 Accept Upstream (Theirs)

  • pyproject.toml - Version field
  • litellm/ - Core library code
  • tests/ - Upstream tests
  • requirements.txt - Dependencies

⚠️ Manual Review Required

  • Dockerfile, docker/Dockerfile.non_root - CARTO customizations
  • Makefile - Check # CARTO: sections
🔧 Conflict Resolution (click to expand)

If this PR has conflicts:

Option 1: Automated (Recommended)

The carto-upstream-sync-resolver workflow triggers automatically.

What it does:

  1. 🤖 Detects conflicts → 🔀 Merges carto/main → ✏️ Resolves conflicts → 🧪 Runs tests → 📌 Pushes to this PR

You just need to: Wait for resolution commits, verify CARTO customizations, merge.

[!TIP]
Single PR workflow! No separate resolution PR needed.

Option 2: Manual Resolution

git fetch origin
git checkout upstream-sync/v1.95.0
git merge origin/carto/main  # Creates conflicts
# ... resolve conflicts ...
make lint && make test-unit
git push origin upstream-sync/v1.95.0
📚 Documentation Links (click to expand)

🤖 This PR was automatically created by the carto-upstream-sync workflow.

🔧 CARTO Feature Fixes Applied

Status: ✅ Fixed
Features Restored: 1

PR #112: Snowflake Cortex Array Content Flattening

  • Files: litellm/llms/snowflake/chat/transformation.py
  • Decision: The original PR fix(snowflake): flatten array-form message content for Cortex #112 modified a _transform_messages method that doesn't exist in the current architecture. The current file uses _transform_request_openai for the OpenAI/chat completions path. Created _flatten_messages_content helper that iterates over messages and flattens array-form content to strings using the existing _content_to_text_string function. This is called in _transform_request_openai before sending messages to Snowflake Cortex.
  • Evidence: Syntax check passed, import check passed, all 16 existing Snowflake tests pass

Fixed: 2026-08-10 13:06:28 UTC
Workflow Run: #41

CARTO Customizations Analysis

Overall Assessment: ✅ PASS

Decision Count Description
Upstream Substitutes 0 Upstream provides equivalent functionality
Customized Upstream 4 Upstream enhanced with CARTO-specific behavior
Preserved CARTO 12 Full CARTO implementation kept
Incorrectly Dropped 0 CARTO feature lost (needs fixing!)
Total 16

CARTO Feature Preservation Analysis

Summary

Decision Count
Upstream Substitutes 0
Customized Upstream 4
Preserved CARTO 12
Incorrectly Dropped 0

Overall Assessment: PASS

All 16 analyzed CARTO features have been correctly preserved during the v1.95.0 upstream sync. The conflict resolution maintained full wiring for all CARTO customizations, with no orphaned helpers or broken call sites detected.

Feature Details

Customized Upstream (4)

These features combined upstream infrastructure with CARTO-specific enhancements:

  1. PR fix(rate-limiter-v3): constant hash tag to avoid CROSSSLOT on Redis Cluster #119 - Redis Cluster CROSSSLOT Fix - Upstream rate limiter v3 base preserved; CARTO added hash tag grouping (REDIS_NODE_HASHTAG_NAME, _group_keys_by_hash_tag) for cluster compatibility

  2. PR fix(docker): fetch arm64 prisma schema-engine for non_root image #118 - ARM64 Prisma Schema-Engine - Upstream Dockerfile structure preserved; CARTO wolfi-base image and arm64 schema-engine fetch integrated

  3. PR fix(oci): Add tool calling support for OCI Gemini streaming #68 - OCI Gemini Tool Call UUIDs - Upstream modular OCI helpers; CARTO setdefault patterns for tool call ID generation integrated

  4. PR feat(docker): multi-arch builds (AMD64 + ARM64) for non_root image #90/multi-arch Docker - Upstream Docker workflow structure; CARTO multi-arch build matrix integrated

Preserved CARTO (12)

These are CARTO-only features with no upstream equivalent:

Snowflake/Cortex (5 features):

Databricks (2 features):

OCI (2 features):

Other Core (3 features):

CI/CD Infrastructure (preserved, not counted above)

All CARTO-specific workflows are preserved:

Issues Found

None. All CARTO features are correctly preserved with verified wiring.

Wiring Verification Notes

The analysis verified not just string presence but actual call site wiring for critical features:

  • _content_to_text_string → called via _flatten_messages_content at line 367
  • _strip_openai_annotations (Snowflake) → called at lines 308, 333
  • _strip_openai_annotations (Databricks) → called at line 478
  • _normalize_empty_tool_call_arguments → called at line 479
  • _validate_and_repair_tool_arguments → called at lines 351, 394
  • _reorder_tool_results_to_match_tool_calls → called at line 237
  • _store_session_in_redis → called at line 901, chains to _patch_store_session_in_redis

Recent Fix Applied

Commit ee89405 restored the _content_to_text_string call sites that were lost during the initial sync conflict resolution. This was caught by the wiring-aware analysis approach documented in PR #126.


Feature-by-Feature Breakdown

PR #127: fix(gh-workflows): wait long enough for the release image digest to resolve

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: CARTO-only workflow file (.github/workflows/carto-n8n-notifier.yml) has no upstream equivalent. The timing fix for Docker image digest resolution is fully preserved
  • Files: .github/workflows/carto-n8n-notifier.yml
  • Recommendation: Correct decision - CARTO-specific CI/CD infrastructure

PR #126: docs(ci): align sync resolver/fixer/analyzer prompts with wiring-aware reasoning

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: CARTO-only workflow documentation and prompts. No upstream equivalent exists for these sync automation files
  • Files: .github/workflows/CARTO_UPSTREAM_SYNC.md, .github/workflows/carto-upstream-sync-ci-fixer.yml, .github/workflows/carto-upstream-sync-customizations-analyzer.yml, .github/workflows/carto-upstream-sync-resolver.yml
  • Recommendation: Correct decision - CARTO-specific CI/CD infrastructure

PR #119: fix(rate-limiter-v3): constant hash tag to avoid CROSSSLOT on Redis Cluster

  • Decision: 🔧 Customized Upstream (high confidence)
  • Reason: Upstream has rate limiter v3 infrastructure. CARTO added hash tag grouping logic (REDIS_NODE_HASHTAG_NAME, _group_keys_by_hash_tag) to avoid CROSSSLOT errors on Redis Cluster. This CARTO enhancement is preserved alongside upstream base code
  • Files: litellm/proxy/hooks/parallel_request_limiter_v3.py
  • Recommendation: Correct decision - CARTO Redis Cluster fix integrated with upstream limiter

PR #118: fix(docker): fetch arm64 prisma schema-engine for non_root image

  • Decision: 🔧 Customized Upstream (high confidence)
  • Reason: Docker non_root image uses CARTO wolfi-base pinned image with arm64 Prisma schema-engine fetch. Upstream base structure preserved with CARTO customizations for multi-arch support
  • Files: docker/Dockerfile.non_root
  • Recommendation: Correct decision - CARTO arm64 support integrated

PR #116: fix(snowflake): assign distinct index per streamed Cortex tool call

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: Snowflake streaming handler has _tool_index field that tracks distinct index per tool call. This is CARTO-specific functionality not present in upstream. Verified wiring: _tool_index is set to 0 in init and updated via chunk.get('index', 0)
  • Files: litellm/llms/snowflake/chat/transformation.py
  • Recommendation: Correct decision - CARTO streaming tool index preserved

PR #112: fix(snowflake): flatten array-form message content for Cortex

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: _content_to_text_string function flattens array-form content to strings (required by Cortex error 390142). Verified wiring: _flatten_messages_content calls _content_to_text_string and is called from _transform_request_openai. Recently fixed in commit ee89405 to restore call sites that were lost during sync
  • Files: litellm/llms/snowflake/chat/transformation.py
  • Recommendation: Correct decision - CARTO Cortex content flattening preserved and wired

PR #111: fix(snowflake): unblock Claude function-calling follow-up turns on Cortex

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: _strip_openai_annotations function strips OpenAI-only annotations that Cortex rejects. Verified wiring: called at lines 308 and 333 within _extract_system_and_messages
  • Files: litellm/llms/snowflake/chat/transformation.py
  • Recommendation: Correct decision - CARTO annotations stripping preserved and wired

PR #110: fix: strip OpenAI 'annotations' field from outbound Databricks chat messages

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: _strip_openai_annotations function strips OpenAI-only annotations from Databricks messages. Verified wiring: called at line 478 within transform_request
  • Files: litellm/llms/databricks/chat/transformation.py
  • Recommendation: Correct decision - CARTO Databricks annotations stripping preserved

PR #109: fix: stop Databricks streaming from rewriting tool_call arguments '{}' to ''

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: _normalize_empty_tool_call_arguments function restores empty tool_call arguments to valid JSON. Verified wiring: called at line 479 within transform_request
  • Files: litellm/llms/databricks/chat/transformation.py
  • Recommendation: Correct decision - CARTO tool call argument fix preserved

PR #108: fix(oci): restore inline-PEM key normalization dropped during upstream sync

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: OCI inline PEM key normalization is present in common_utils.py. Verified: 'oci_key must be a string containing the PEM private key content' error message present at line 310
  • Files: litellm/llms/oci/common_utils.py
  • Recommendation: Correct decision - CARTO OCI PEM normalization preserved

PR #70: fix(azure): Strip operation suffixes from deployment URLs to prevent 404 errors

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: Azure URL suffix stripping regex pattern preserved. Verified: pattern 'chat/completions|completions|embeddings|audio/speech' found at line 264 in common_utils.py
  • Files: litellm/llms/azure/common_utils.py
  • Recommendation: Correct decision - CARTO Azure URL fix preserved

PR #68: fix(oci): Add tool calling support for OCI Gemini streaming

  • Decision: 🔧 Customized Upstream (high confidence)
  • Reason: OCI generic handler uses setdefault for tool call ID generation. Verified: setdefault patterns at lines 400-402 and 459 in generic.py. CARTO's UUID generation merged with upstream modular helpers
  • Files: litellm/llms/oci/chat/generic.py
  • Recommendation: Correct decision - CARTO tool call support integrated with upstream

PR #54: fix: repair malformed JSON in streaming tool call arguments

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: _validate_and_repair_tool_arguments function repairs malformed JSON in streaming. Verified wiring: called at lines 351 and 394 in streaming_chunk_builder_utils.py
  • Files: litellm/litellm_core_utils/streaming_chunk_builder_utils.py
  • Recommendation: Correct decision - CARTO JSON repair preserved and wired

PR #38: fix: Snowflake PAT auth and Claude streaming support

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: SnowflakeStreamingHandler class preserved. Verified: class definition at line 575, _extract_system_and_messages at line 254
  • Files: litellm/llms/snowflake/chat/transformation.py
  • Recommendation: Correct decision - CARTO Snowflake streaming preserved

PR #5: Jatorre/fix/responses api redis session timing

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: Redis session storage functions preserved. Verified wiring: _store_session_in_redis at line 1106, _patch_store_session_in_redis at line 2135, call site at line 1135
  • Files: litellm/responses/litellm_completion_transformation/streaming_iterator.py, litellm/responses/litellm_completion_transformation/transformation.py
  • Recommendation: Correct decision - CARTO Redis session storage preserved and wired

PR #121: OCI Parallel Tool Result Reordering

  • Decision: 🔒 Preserved CARTO (high confidence)
  • Reason: _reorder_tool_results_to_match_tool_calls function preserved. Verified wiring: function definition at line 168, call site at line 237
  • Files: litellm/llms/oci/chat/generic.py
  • Recommendation: Correct decision - CARTO OCI tool reordering preserved

Analyzed: 2026-08-10 13:21:55 UTC
Workflow Run: #42
Analysis Artifacts: Download JSON/MD
Method: Claude Code (Opus 4.5) post-resolution semantic analysis

yuneng-berri and others added 30 commits July 23, 2026 23:18
…o shadcn

Replaces antd and Tremor with shadcn primitives across the 18 files these three
routes exclusively own. Markup only: no behaviour, data flow or copy changed, and
no shared or form-bearing component is touched, so the blast radius stops at
these pages.

The 12 tests covering these components are unchanged from the previous commit and
still pass, which is the evidence that the rewrite preserved behaviour. Also
prunes the six antd no-restricted-imports suppressions these files no longer
need.
A template with no LLM enrichment rendered every parameter field twice: the
shared list already covers them, because nonEnrichmentParams is the full
parameter list when there is no enrichment, and a second no-enrichment branch
mapped the same list again.

Predates the shadcn migration and was carried forward by it. The test now
asserts exactly one field per parameter, and fails if the duplicate branch
comes back.
…emory-9b2c09

refactor(ui): migrate memory page to shadcn
The suggested CIDR chip was a click-only span both before and after the shadcn
migration, so keyboard users could not reach or activate it. Render it as a
Button, which brings focus and Enter/Space activation with it, and cover the
keyboard path with a test that fails against the old span.
25 test functions across three files pass unchanged when every function
they execute is mutated; the owning file killed zero of their scored
mutants. Four zero-kill tests tied to the fix in BerriAI#31288 are kept for
rewrite instead of removal.
…itellm_/test-coverage-mutation-analysis-e42223
…in-3d4c69

refactor(ui): migrate budgets, skills, ui-theme to shadcn
…0e7ee6

refactor(ui): migrate access-groups, vector-stores, organizations to shadcn
…hcutt-14b37d

refactor(ui): migrate logging-and-alerts, caching, policies to shadcn
…andalone

feat(ui): standalone /connect route for MCP OAuth, decoupled from Chat UI flag
…lds-5a5be3

refactor(ui): migrate mcp-servers, tag-management, tool-policies to shadcn
Registers claude-opus-5 across the cost maps and provider lists so the model
prices, reports its real 1M/128K limits, and advertises its capabilities instead
of falling through the generalization patterns at zero cost.

Adds the first-party entry plus the Bedrock (base, global, us, eu, au, jp),
Vertex AI, and Azure AI variants. Pricing matches Opus 4.8 at $5/$25 per MTok
with the usual 1.1x regional premium on the cross-region inference profiles, and
fast mode is priced at 2x through provider_specific_entry on the first-party
entry only.

Two fields deliberately differ from Opus 4.8: prompt_cache_min_tokens drops to
512, and bedrock_output_config_effort_ceiling is omitted because Bedrock accepts
output_config.effort="max" for Opus 5.
…orks (BerriAI#34512)

The a2a completion-bridge tests registered the agent with only
custom_llm_provider and model, so the bridge's litellm.acompletion had no
api_key and relied on the gateway resolving ANTHROPIC_API_KEY from its ambient
env. When that env var is absent, POST /a2a/{id} returns 500 with
"Missing Anthropic API Key" and every message/send test (completion bridge,
pinned v0.3/v1.0 message shapes, semver serves) fails while the
register/discovery/rejection tests still pass.

Give A2ABridgeParams an optional api_key and register the bridge agent with
api_key="os.environ/ANTHROPIC_API_KEY", matching how the rest of the suite
wires anthropic-backed models (e.g. the ratelimit redis tests). The agent now
carries its provider key explicitly instead of depending on ambient gateway env.
…ss_response (BerriAI#34390) (BerriAI#34405)

* fix(guardrails/model_armor): handle None metadata in post_call _process_response

On batch routes data["metadata"] is normalized to None (present key, None
value), so request_data.get("metadata", {}) returned None and _process_response
raised 'NoneType' object has no attribute 'get', 500ing every /v1/batches create
with a post_call Model Armor guardrail (regression from v1.93.0 activating the
post_call hook). Coalesce a falsy metadata to {}

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Clean up test case documentation

Remove regression comment from test_process_response_with_none_metadata_does_not_crash.

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
chore(ci): promote internal staging to main
…async_failure_handler (BerriAI#34306)

The async streaming error paths fired the sync failure_handler in a thread
and the async_failure_handler via create_task at the same time, so both
mutated the shared logging object concurrently and could crash pydantic-core.
Route failure logging through a single guarded dispatch_failure_handlers, so
the sync handler only runs after the async one completes.
…event multi-value wedge (BerriAI#34320)

The accumulated-JSON fallback ran json.loads over the whole buffer after every fragment and, on failure, kept the buffer without resetting it. A buffer that ever held more than one concatenated JSON value could never parse (json raises on trailing data), so it returned None on every subsequent chunk while growing without bound - an unrecoverable per-request CPU spin. Parse one value at a time from the front with raw_decode and keep the remainder, draining trailing values on later calls and at end of stream.
…odex-test-27520b

test: replace deprecated gpt-5-codex with gpt-5.3-codex
…erriAI#34426)

The gateway image installed --extra proxy/proxy-runtime/extra_proxy/semantic-router
but not bedrock-realtime, so aws-sdk-bedrock-runtime was absent. Bedrock Nova Sonic
speech-to-speech uses InvokeModelWithBidirectionalStream (which boto3 cannot do) via
that package, so realtime requests failed at startup with 'Missing
aws_sdk_bedrock_runtime'. Add the extra to both uv sync stages; it is already in
uv.lock so --frozen resolves, and the package/marker (python>=3.12) matches the
python3.13 image.
litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.

A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.

SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.

python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.

Resolves LIT-4016
…ays stream completed responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…and post_call guardrails (BerriAI#33770)

* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails

Pre-call guardrails run sequentially because each may mutate the request
payload and later guardrails depend on earlier mutations. Deployments with
several slow block-only pre_call guardrails (external moderation, Bedrock,
LLM-judge) therefore pay the sum of their latencies. during_call guardrails
run concurrently but alongside the LLM call, so a violating payload has
already been sent, which is unacceptable when the request must never reach
the model.

This adds a per-guardrail run_in_parallel flag (default off). Guardrails that
opt in are pulled out of the sequential loop and run concurrently via
asyncio.gather after every sequential (payload-mutating) guardrail has run, so
they observe the mutated payload and still form a hard barrier before the LLM
call; the first to raise blocks the request. Their returned data is discarded
since they are declared block-only.

The flag is wired from LitellmParams onto the guardrail instance at the same
generic choke point in initialize_guardrail that already sets
skip_system_message_in_guardrail, so no per-provider initializer needs to
change.

* feat(guardrails): extend run_in_parallel opt-in to post_call guardrails

post_call_success_hook ran guardrails sequentially for the same reason
pre_call did: response-modifying guardrails thread the response forward. But
block-only output scanners (which read the response and reject on violation
without changing it) serialize for no benefit and add latency.

This reuses the existing run_in_parallel flag for the post_call hook. Opted-in
post_call guardrails are pulled out of the sequential loop and run concurrently
via asyncio.gather after the sequential (response-modifying) guardrails and
before the non-guardrail CustomLogger callbacks, so they inspect the final
response and still block it from reaching the client if any raises. Their
returned response is discarded since they are block-only.

The apply_guardrail path sets data["guardrail_to_apply"] immediately before
awaiting, and unified_guardrail pops it before its first suspension point, so
concurrent guardrails never race on that key under asyncio's cooperative
scheduling.

* fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes

Addresses review feedback on the run_in_parallel opt-in.

asyncio.gather propagated the first exception without cancelling or awaiting
the siblings, so a block at t=0 left the other guardrails running as
unobserved background tasks (wasted external calls plus event-loop warnings),
and a fast SensitiveDataRouteException/ModifyResponseException could return a
reroute or passthrough before a slower block finished, letting crafted input
bypass the block. Both the pre_call and post_call parallel batches now gather
with return_exceptions=True so every guardrail runs to completion, then raise
any blocking exception ahead of a flow-changing one.

The registry choke point wrote bool(None)==False onto every instance when the
config omitted run_in_parallel, silently disabling a constructor-set default;
it now only writes when the config provides an explicit value.

* fix(guardrails): record lifecycle logs for every concurrently-run guardrail

The log_guardrail_information decorator skipped its auto-record when it saw
that the count of standard_logging_guardrail_information entries in the shared
request_data had grown during the wrapped call, taking that as proof the
wrapped function had recorded its own richer entry. That heuristic breaks the
moment guardrails run concurrently (parallel pre_call/post_call, during_call):
a sibling guardrail's append inflates the shared count, so a guardrail that did
not self-record wrongly concludes it already did and drops its own entry. The
result is that enabling run_in_parallel silently loses per-guardrail lifecycle
logs, so the Admin UI Request Lifecycle timeline and downstream loggers
(Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent
guardrails.

Replace the shared-count heuristic with a ContextVar flag set when a guardrail
records its own entry. asyncio copies the context into each gathered task, so
the flag is isolated per concurrent guardrail while still catching the
self-record-then-skip-auto-record case within a single invocation.

* test(guardrails): declare run_in_parallel on post_call guardrail mocks

The post_call partition reads run_in_parallel on every CustomGuardrail
callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is
set in __init__, not on the class) so the attribute access raised, and even
a class-level default would return a truthy child mock that wrongly routes
the double into the parallel batch. Declare the flag False on the shared
mock factories so these pre-existing hook tests exercise the sequential
path they assert on.

* fix(guardrails): harden run_in_parallel reads and address review feedback

Read run_in_parallel via getattr(..., False) in the pre_call and post_call
partitions so a third-party CustomGuardrail subclass that overrides __init__
without chaining super().__init__() no longer raises AttributeError on a path
that previously worked. Drop the redundant in-function GuardrailEventHooks
import in _run_parallel_post_call_guardrails (already imported module-level).
Remove the flaky wall-clock upper-bound assertions from the two concurrency
tests; the all-start-before-any-end overlap assertion is the timing-independent
signal that actually proves concurrency.
…ssages

Router.acompletion() takes messages positionally, so splatting a body that omits it raised a TypeError that the generic handler mapped to a 500. Validate the required body param at the routing boundary and raise the existing 400 contract instead.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
yuneng-berri and others added 10 commits July 29, 2026 09:06
…p_e2e_poll

test(e2e): poll MCP tools across multi-worker lag (BerriAI#35047)
Reverts BerriAI#32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.

The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.

Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.

(cherry picked from commit 6f1625d)
chore(release): sync rc/1.95.0 with the v1.95.0-rc.1 main SHA
…_landing

fix(ui): land general login on the keys dashboard, send MCP consent to /ui/connect

(cherry picked from commit ceaf556)
…-rc-1-95-0-f65d9e

fix(ui): land general login on the keys dashboard, send MCP consent to /ui/connect (backport BerriAI#35523 to rc/1.95.0)
Automatic sync from upstream BerriAI/litellm tag v1.94.0

Strategy: Merge with history preservation (main syncs to stable tag)
Automatic sync from upstream BerriAI/litellm tag v1.95.0

Strategy: Merge with tree-level conflict resolution (accepted all upstream changes)
Conflicts resolved: 151 files (11 rename/rename, 25 modify/delete, 118 rename/delete, 0 content)
@github-actions

Copy link
Copy Markdown

No description provided.

@Cartofante

Copy link
Copy Markdown
Collaborator Author

🤖 Conflict Resolution Started

Status: ⏳ In progress...

Claude Code (Opus 4.5) is resolving merge conflicts in this PR.

Step Status
🔍 Analyze conflicts In progress
✏️ Resolve files Pending
🧪 Run tests Pending
📌 Push to PR Pending

Note

This may take 30-90 minutes for large PRs. Resolution commits will be pushed directly to this PR.

📋 Resolution Process (click to expand)
  1. 🔍 Analyzing conflicts in all files
  2. 📋 Reading CARTO customization guidelines
  3. ⚖️ Applying resolution priorities (CARTO first, then upstream)
  4. ✏️ Resolving conflicts file by file
  5. 🧪 Running tests (lint, mypy, unit tests)
  6. 📌 Pushing resolution directly to this PR

View workflow run →

Conflicts resolved by Claude Code following CARTO priority rules.

Resolution strategy:
- Preserved CARTO customizations (workflows, docs, infrastructure)
- Accepted upstream improvements (core litellm, tests, dependencies)
- Manually merged mixed files (Dockerfile, Makefile)

This is a MERGE COMMIT with both main and carto/main as parents,
preserving full git history from upstream.

Resolves: #128
@Cartofante

Copy link
Copy Markdown
Collaborator Author

✅ Conflict Resolution Complete

All conflicts resolved and pushed to this PR.

Step Status
🔍 Analyze conflicts ✅ Complete
✏️ Resolve files ✅ Complete
🧪 Verify resolution ✅ Complete
📌 Push to PR ✅ Complete

Important

Ready to merge! Use "Create a merge commit" — do NOT squash or rebase.


CARTO Customization Decisions

Summary

Decision Count
Upstream Substitutes 0
Preserved CARTO 10
Merged/Customized 0
Synced (required) 1168

Preserved CARTO

Files where CARTO implementation was kept (verbatim from carto/main):

  • litellm/llms/oci/chat/generic.py - OCI parallel tool reordering, tool call UUID setdefault, tool_call_indices param
  • litellm/llms/oci/common_utils.py - OCI inline PEM key normalization
  • litellm/llms/snowflake/chat/transformation.py - Snowflake streaming handler, annotations stripping, array content flattening, URL passthrough
  • litellm/llms/databricks/chat/transformation.py - Empty tool call arguments normalization, annotations stripping
  • litellm/litellm_core_utils/streaming_chunk_builder_utils.py - JSON repair for streaming tool calls
  • litellm/responses/litellm_completion_transformation/streaming_iterator.py - Redis session storage, litellm_completion_request wiring
  • litellm/responses/litellm_completion_transformation/transformation.py - Redis session handlers, tool_choice_value logic

Synced (Required)

Files synced entirely from upstream (no CARTO customizations needed):

  • schema.prisma - Database schema must match upstream
  • All litellm-rust/ files - Rust gateway components
  • All ui/ files - UI components (fully restored from upstream)
  • All tests/ files - Test suite
  • All helm/, gateway/, enterprise/ files - Infrastructure
  • All other litellm/ files not listed in Preserved CARTO

Fix Loop Interventions

Files synced due to repeated conflicts:

  • None

Verification Steps Completed

  1. All 18 CARTO feature patterns verified via grep
  2. No orphaned CARTO helpers (all defined helpers are called)
  3. Cross-file wiring verified for Redis session storage
  4. Docker build passes successfully
  5. No conflict markers remain in codebase

Next Steps

  1. Review the CARTO customization decisions above
  2. Merge using "Create a merge commit"
  3. 🎉 Upstream sync complete!
🔧 Workflow Details (click to expand)

Workflow Run: https://github.com/CartoDB/litellm/actions/runs/31387990410

@Cartofante

Copy link
Copy Markdown
Collaborator Author

Caution

⚠️ Merge Method Reminder

Use "Create a merge commit" — Click the dropdown arrow next to merge button.

❌ Do NOT use "Squash and merge" or "Rebase and merge"

Squashing destroys upstream history and breaks future syncs (see PR #26).

@Cartofante

Copy link
Copy Markdown
Collaborator Author

📊 CARTO Feature Analysis Started

Mode: Analysis + Auto-fix if issues found

Analyzing how each CARTO customization was handled during conflict resolution.
This will explain WHY each decision was made.

Step Status
Extract CARTO PRs ⏳ In progress
Compare code versions ⏳ Pending
Analyze decisions ⏳ Pending
Generate report ⏳ Pending

View workflow →

@Cartofante

Copy link
Copy Markdown
Collaborator Author

⚠️ CARTO Feature Analysis - Attention Needed

Decision Count
Upstream Substitutes 0
Customized Upstream 0
Preserved CARTO 11
Incorrectly Dropped 1

Overall Assessment: NEEDS_ATTENTION

🔧 Auto-fix enabled: The fix job will run next to restore dropped features.

📋 Full details in PR description above.


View workflow run → | Download analysis artifacts →

@Cartofante

Copy link
Copy Markdown
Collaborator Author

🔧 CARTO Feature Fix Started

Restoring 1 incorrectly dropped CARTO feature(s).

View workflow →

…ay content flattening

PR #112 added _content_to_text_string to flatten array-form message content
to strings (required by Cortex error 390142), but the call sites were lost
during upstream sync conflict resolution

The function was defined but never called, meaning Cortex would reject
array-form content like [{"type": "text", "text": "..."}] with error 390142
on multi-turn conversations where the OpenAI Agents SDK replays prior
assistant turns in list-of-blocks form

Added _flatten_messages_content helper that transforms all messages by
flattening array-form content to strings, wired into _transform_request_openai
before sending to the /chat/completions endpoint

Verified: syntax check, import check, 16 existing Snowflake tests pass
@Cartofante

Copy link
Copy Markdown
Collaborator Author

🔧 CARTO Feature Fix Complete

Summary

Restored 1 CARTO feature (PR #112 - array content flattening) by adding call sites for the orphaned _content_to_text_string function

Decisions Made

PR #112: Snowflake Cortex Array Content Flattening

Decision: The original PR #112 modified a _transform_messages method that doesn't exist in the current architecture. The current file uses _transform_request_openai for the OpenAI/chat completions path. Created _flatten_messages_content helper that iterates over messages and flattens array-form content to strings using the existing _content_to_text_string function. This is called in _transform_request_openai before sending messages to Snowflake Cortex.
Files Modified: litellm/llms/snowflake/chat/transformation.py
Verification: Syntax check passed, import check passed, all 16 existing Snowflake tests pass


Next Steps:

  1. Review the changes in the Files tab
  2. CI will re-run automatically
  3. Once CI passes, the sync-ready label will be added

View full workflow logs →

@Cartofante

Copy link
Copy Markdown
Collaborator Author

📊 CARTO Feature Analysis Started

Mode: Analysis + Auto-fix if issues found

Analyzing how each CARTO customization was handled during conflict resolution.
This will explain WHY each decision was made.

Step Status
Extract CARTO PRs ⏳ In progress
Compare code versions ⏳ Pending
Analyze decisions ⏳ Pending
Generate report ⏳ Pending

View workflow →

@Cartofante

Copy link
Copy Markdown
Collaborator Author

✅ CARTO Feature Analysis Complete

Decision Count
Upstream Substitutes 0
Customized Upstream 4
Preserved CARTO 12
Incorrectly Dropped 0

Overall Assessment: PASS

📋 Full details in PR description above.


View workflow run → | Download analysis artifacts →

@mateo-di

Copy link
Copy Markdown
Collaborator

Closing in favor of a fresh sync straight to the latest upstream stable (v1.98.0). v1.95.0 is already several releases behind, so rather than merge it and wait for the next scheduled run to produce v1.98.0, we're skipping ahead. A new upstream-sync PR will be created by a manual run of the sync workflow.

@mateo-di mateo-di closed this Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automated ignore-semantic-pull-request Exempt from Conventional PR Title check (e.g. auto-generated upstream-sync PRs) sync-ready upstream-sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.