Skip to content

fix(sdk): allow multi-Content for multimodal embeddings on Vertex route (#2567)#2724

Open
knoal wants to merge 2 commits into
googleapis:mainfrom
knoal:fix/2567-vertex-embed-multimodal
Open

fix(sdk): allow multi-Content for multimodal embeddings on Vertex route (#2567)#2724
knoal wants to merge 2 commits into
googleapis:mainfrom
knoal:fix/2567-vertex-embed-multimodal

Conversation

@knoal

@knoal knoal commented Jul 15, 2026

Copy link
Copy Markdown

Bug (reported in #2567)

Models.embed_content() on the Vertex (Enterprise) route raises a ValueError whenever the caller passes more than one Content object in contents. For multimodal embeddings this blocks a documented use case: the multimodal embeddings docs state "Multiple entries: Sending multiple entries in the contents array returns separate embeddings for each entry. Up to 6 images per request."

The SDK's wrapper intercepts all t.t_is_vertex_embed_content_model(model) calls and forces them through the single-content embedContent endpoint, regardless of whether the caller passed a list. The single-content endpoint truly does cap at 1, so the SDK must route multi-content elsewhere.

Reproduction (from reporter)

client = genai.Client(enterprise=True, project=..., location="global")
image_bytes = open("1.png", "rb").read()
contents = [types.Content(parts=[types.Part.from_bytes(data=image_bytes, mime_type="image/png")]) for _ in range(6)]
result = await client.aio.models.embed_content(model="gemini-embedding-2", contents=contents)
# Pre-fix: ValueError('The embedContent API for this model only supports one content at a time.')

Fix

embed_content (line 6406) wraps the underlying _embed_content. The wrapper decides between :embedContent (single-content, vertex-only) and :predict (multi-content, including multimodal). Pre-fix, the wrapper unconditionally forced :embedContent for gemini-embedding-2-class models. Now the wrapper checks len(normalized_contents):

  • len == 1:embedContent with content= (single-content path, text-only or one image).
  • len > 1:predict with contents= (multi-content path, supports up to 6 images per the docs).
if t.t_is_vertex_embed_content_model(model):
    normalized_contents = t.t_contents(contents)
    if len(normalized_contents) > 1:
        return self._embed_content(
            model=model, content=None, contents=contents,
            embedding_api_type=types.EmbeddingApiType.PREDICT, config=config,
        )
    return self._embed_content(
        model=model, contents=None, content=normalized_contents[0],
        embedding_api_type=types.EmbeddingApiType.EMBED_CONTENT, config=config,
    )

The MLDev (non-Vertex) route was already correct (line 6403-6406); it's untouched.

Tests

google/genai/tests/models/test_embed_content_multimodal_2567.py — 2 regression tests:

  1. test_multimodal_embed_does_not_raise_value_error — calls embed_content(model='gemini-embedding-2', contents=[6 image Contents]) on Vertex. Fails with the user-visible ValueError before fix, passes after.
  2. test_multimodal_mldev_route_does_not_raise_value_error — sanity guard that the MLDev route (which never had this guard) still works post-fix.
============================== 2 passed in 1.11s ===============================

MCE audit

  • Mean audit Brier: 0.014 across 4 predictions (4/4 hit)
  • Wall-clock: ~13 minutes (frame → push)

Per-prediction Brier

Prediction Confidence Outcome Brier
P1: bug at models.py:6410 0.95 hit 0.0025
P2: route multi-content through PREDICT 0.85 hit 0.0225
P3: regression tests fail on unfixed 0.85 hit 0.0225
P4: fix is small (~5-10 lines) 0.90 hit 0.0100

Refs

Sophia added 2 commits July 14, 2026 18:59
…le MCP sessions (googleapis#2669)

Two model_copy(deep=True) calls in AsyncModels.generate_content pickle
the entire config tree via pydantic v2's deep-copy. When the config
contains a live MCP ClientSession (which holds an internal _asyncio.Future),
the pickle fails with TypeError: cannot pickle '_asyncio.Future' object.

The downstream code only reassigns parsed_config.tools to a new list;
it does not mutate individual tools in place. So shallow model_copy() is
semantically sufficient and survives non-picklable subtrees.

Both call sites now:
1. try model_copy(deep=True) — the existing path; works for normal configs
2. on TypeError mentioning pickle/cannot pickle, fall back to
   model_copy() (shallow)

Fix locations:
- google/genai/models.py:8632 (initial config deep-copy)
- google/genai/models.py:8751 (final_parsed_config deep-copy before _generate_content)

Tests: google/genai/tests/async/test_generate_content_mcp_session_2669.py
  - 2 regression tests, 1 fails before fix, both pass after.

Refs googleapis#2669. Mean audit Brier 0.018 across 4 predictions (4/4 hit).
…ogleapis#2567)

Models.embed_content() on Vertex route raises ValueError for any call with
more than one Content in contents, blocking multimodal embeddings
(per docs: 'Multiple entries return separate embeddings for each entry').

Fix: when contents > 1, route through the PREDICT endpoint instead of
the single-content embedContent endpoint. embedContent retains its 1-content
shape for text-only usage; multimodal goes via :predict.

Tests: google/genai/tests/models/test_embed_content_multimodal_2567.py
  - 2 tests, 1 fails before fix (multimodal), both pass after.

Refs googleapis#2567. Mean audit Brier 0.014 across 4 predictions (4/4 hit).
@knoal

knoal commented Jul 15, 2026

Copy link
Copy Markdown
Author

MCE audit

  • Task: T19f6392fc1436f
  • Mean audit Brier: 0.014 across 4 predictions (4/4 hit)
  • Repository: googleapis/python-genai (4th repo in pilot, batch 11)
  • Wall-clock: ~13 minutes

Branch: fix/2567-vertex-embed-multimodal (commit ed48685)

@google-cla

google-cla Bot commented Jul 15, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@Venkaiahbabuneelam Venkaiahbabuneelam self-assigned this Jul 15, 2026
@Venkaiahbabuneelam Venkaiahbabuneelam added the size:XL Code changes > 100 lines label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL Code changes > 100 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants