Skip to content

feat: Gemini media-v1 wire plans, retry guards and safe diagnostics - #65

Draft
Brian Krabach (bkrabach) wants to merge 5 commits into
mainfrom
feat/media-projection-v1
Draft

Brian Krabach (bkrabach) wants to merge 5 commits into
mainfrom
feat/media-projection-v1

Conversation

@bkrabach

@bkrabach Brian Krabach (bkrabach) commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Implement frozen P1 Gemini media-v1 projection: exact documented model/position facts, original-position user inline_data and call-bound functionResponse.parts, private count/send plan identity, per-attempt media validation and pre-emit pixel-safe diagnostics. Runtime imports Core only: no Core changes or Foundation/app/peer-SDK helper imports. No host activation.

The initial two-model allowlist was incomplete. The current catalogue follows public vendor evidence audited 2026-11-14, including the configured gemini-3.8-flash default. Exact per-model URLs, modality quotations, tool scope and route/availability caveats are in contracts/gemini-media-evidence.v1.json. Unknown positions are not qualified for this route, not asserted vendor-incapable.

Exact source and dependency holds

Base: accepted main e040c7cee5561c0b5b2df66a3dc8ac8cd2a4e043, not an unrelated local provenance branch.

DRAFT / DO NOT MERGE: parent106 final exact-head Astra/Fable review and integration against accepted F1/L1/host dependencies remain required. Offline pairing uses immutable archived Foundation candidate 31766b42efe71920386975cf10d136b38b85fc14, not a moving dirty checkout and not an accepted-graph claim. No DTU launch, credentialed model catalogue, live inference qualification, merge, release or live adoption is performed here. Integration owns the later accepted graph/installed qualification.

Structural seam: get_media_request_support(*,model,request_options=None) returns v1 positions/mime_types/evidence and optional reason. cap.validate(exact_private_request,purpose='count'|'transport',request_options=effective_options) gates every real attempt. Any cap-present request needs its host-owned private association, including text-only/unavailable projections. No witness marker or hydration is added to canonical metadata/history/suspension.

Current support matrix — offline wire, not entitlement/live vision

PNG/JPEG only under the frozen producer scope. Exact enumerated IDs, never a runtime prefix guess or automatic local model switch:

Exact IDs User image Call-bound tool image
gemini-3.8-flash, gemini-3.7-flash, gemini-3.6-flash, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.1-flash-lite, gemini-3.1-pro-preview, gemini-3.1-pro-preview-customtools, gemini-3-flash-preview inline data nested functionResponse.parts, explicit vendor G3 text-model scope
gemini-4-hydrogen inline data nested functionResponse.parts, named vendor example
gemini-4-carbon, gemini-4-argon, gemini-robotics-er-2-preview inline data unconfirmed scope; warning at original position before count
gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite inline data unqualified placement; warning at original position before count
Other/unresolved latest or dated aliases, retired/conflicting IDs not catalogued for this route not catalogued for this route

This is 16 user-image / 10 tool-image IDs. get_info and list_models vision tags use the same exact catalogue. System/developer/assistant images remain unqualified. No synthetic user image turn is added. Specialized image-generation models can genuinely accept images but have different output/function-calling contracts; Live, TTS, embedding, managed-agent, Decisions and video-output APIs are distinct. Existing specialized image-operation code is unchanged. No blanket Gemini-family capability claim.

Vendor routing notices (3.7-flash to 3.8-flash, 3.5-flash to 3.6-flash) and 2.5 existing-user restrictions/future December 1 routing are disclosed, not implemented locally.

Limits versus native measurement

Robotics standard ER2's exact public input/output facts are 131,072 / 65,536 tokens. get_info().defaults.context_window now publishes 131,072 rather than inheriting the unknown-model 1,048,576 fallback. Publication-only exact facts are separate from native-counter qualification; the separate Live endpoint and guessed Robotics suffixes do not inherit them.

Native budget counting remains unavailable for the three Gemini 4 IDs and Robotics ER2. Gemini 4 unified-window semantics are not guessed or changed. Tests distinguish actual mocked HTTP count on the previously qualified 3.x/2.5 set from wire-projection parity only on 4/Robotics; no fake provider_count or borrowed measured-fit claim.

Identity, retry, metadata and privacy

A weak request-scoped vendor plan binds the exact candidate/options/selected connection; SDK calls receive copies. Native signatures, provenance and call pairing are retained. Finalization tool_choice=none stays NONE. Host/loop single-refit remains authoritative; no new refit loop, compact_context or public iterator stream method is fabricated. complete owns the existing internal streaming transport.

Qualified media requests disable SDK/aiohttp inner retries, AFC, redirects and unqualified HTTP auth/request hooks/retrying transports without mutating the shared client. Received transient failures revalidate before retry; unknown sends, timeouts and partial streams do not automatically replay. Completion/error/cancellation/count failure/session close release provider-owned cache state.

Diagnostic COPY projection occurs before hooks.emit. Known image transports are omitted with explicit not-retained markers, not fabricated resolvable refs or raw-base64 fallbacks. Legacy-image retry/error diagnostics are protected too. No diagnostic ingest/publish or new canonical retention authority is added.

Verification and review

Existing isolated dependencies: Core1.6.1, google-genai2.30.1, HTTPX0.28.1, Pillow12.3.0. No manifest/lock changes. Both provider keys unset; real socket connects refused before pytest/provider imports, plus maintained autouse offline fence.

  • Catalogue source at 66f92628... / final evidence wording 264b59ae...: 187 focused passes paired with immutable F1 candidate31766. Matrix: 64 valid PNG/JPEG user complete/SSE cases, 40 tool cases, 64 actual F1 canonical-to-private cases plus prior guards. Every literal declared model is covered by actual SDK MockTransport, decoded original bytes/SHA/order/MIME/options/privacy; warnings are frozen at the original call-bound block before count.
  • Last unchanged broader standalone campaign at catalogue source: 1,041 passed, 1 optional F1-module skipped, 1 live test deselected, 38 warnings. Reused, not rerun or relabeled as a fresh full pass for the narrow Robotics fact correction.
  • Current narrow Robotics published-limit/native-gate repair: 88 published-limit and native-count tests passed, 25 preexisting warnings; 8 Robotics SDK/F1 wire cases passed, 179 unrelated media cases deliberately deselected. Exact input/output publication, native count unavailability/no transport, unverified Robotics endpoint/suffix and unchanged Gemini 4 behavior covered.
  • Independent worker accepted the catalogue source and reproduced187/1041. Manager-commissioned critique230921 accepted exact catalogue delta through264b (vendor/offline scope only). The current narrow correction receives its own changed-risk review; parent final Astra/Fable remains separate.
  • Helper/changed test Ruff and git diff --check pass. Full init.py Ruff retains two proven baseline findings (unused DEFAULT_LIMITS, release_images lambda), not falsely green.

Retained failures / scope incident

The Robotics regression was first demonstrated failing (1048576 !=131072); failure log retained. Earlier catalogue run had one stale 3.8 no-vision expectation failure, corrected from exact documentation; prior valid-pixel assertions initially used standard rather than the SDK's URL-safe strict base64 alphabet, corrected while preserving exact original byte equality. Failed logs remain retained, not converted to passes.

An earlier initial focused command accidentally omitted the live-test exclusion and ran the existing real-API image test with an inherited credential; it failed, was disclosed, never repeated or used as qualification. Subsequent checks unset both keys, exclude live tests and enforce the pre-import socket fence. Initial wire-alias/Mock diagnostic-recursion failures and timeout/termination logs are retained. No later inference campaign is implied.

Evidence and cleanup

Task-owned untracked working-files retains exact result packets, sanitized actual-wire XML/JSON with decoded SHA/order, source/dependency joins, baseline/current/failure logs and immutable F1 archive. Source checkout and isolated .venv remain intentionally available for parent integration. No containers, DTUs or remote runtimes created; no unrelated teardown, automatic cleanup or live activation claimed.

@bkrabach

Copy link
Copy Markdown
Collaborator Author

Advertised-model offline wire coverage follow-up

Test-only follow-up closes the review gap for the already-advertised gemini-2.5-flash user-image path. Production provider code, model declarations and dependencies remain byte-identical to reviewed e7f959e.

Exact follow-up head: 6c1cff46ff3fa372da53e39023a4d39c4d42e42f.
Independent read-only review accepted this exact test delta and independently
reproduced 29 focused / 943 standalone passes under the pre-import socket fence.
Reviewed test hashes: 8ab502ab7b841751613657f50e2bfef76277dcc724f2edfc274a4e46d708dfa8
and 457f1a48a89487e717a3ab5a2539ebb01a66be896b27fb8bf40b83aef6246e43.

Literal model/position Valid PNG Valid JPEG Complete Real SSE stream
gemini-3.1-pro-preview user inline image pass pass pass pass
gemini-2.5-flash user inline image pass pass pass pass
gemini-2.5-flash tool result unavailable at original call-bound position before count unavailable at original call-bound position before count pass pass
gemini-3.1-pro-preview tool result existing fixtures retained no new broader tool-MIME claim existing fixture retained existing fixture retained

The 8-case model/MIME/transport matrix uses actual google-genai serialization into HTTPX MockTransport, Pillow-verified original images, exact selected URL/model, decoded original bytes/SHA, user text/image order, count/send options parity, unchanged owner metadata and pre-emit privacy. It does not merely assert a support declaration.

The 4 additional Flash fixtures use the real SessionMediaStore/capability/projector from immutable archived Foundation candidate 31766b42efe71920386975cf10d136b38b85fc14. They assert the tool unavailable warning at its exact message/block/call position before any HTTP count, then exercise the actual HTTP count and SDK send paths with the same private object/options, original user image bytes and no extra synthetic user turn. This remains candidate-graph evidence only.

Verification uses both provider key variables unset and a socket.connect/connect_ex refusal installed before pytest/provider imports, in addition to the maintained autouse fence. No inference, live credentials/config changes or DTU occurred.

  • Focused media + immutable Foundation candidate suite: 29 passed (17 retained + 12 added).
  • Full standalone offline suite: 943 passed, 1 skipped, 1 deselected, 38 warnings. The optional Foundation module is the skip; live marked test remains excluded.
  • Both changed tests pass Ruff; git diff --check passes.
  • First matrix run: 12 failed, 17 passed because the new assertions used standard-alphabet strict base64 decoding while locked google-genai2.30.1 _common.encode_unserializable_types uses urlsafe_b64encode. The corrected assertions strictly decode the SDK alphabet with altchars=b'-_' and still require exact original bytes. Original failure log/XML retained; not relabeled green. Privacy checks reject both alphabet representations.

Prior failed/inadvertent-live incident evidence remains retained and unchanged. No runtime fixes or blanket capability-tag changes were made for this coverage repair. Parent exact-head Astra/Fable review, accepted F1/L1 graph integration and integration-owned qualification remain holds; PR stays draft.

@bkrabach

Copy link
Copy Markdown
Collaborator Author

Full documented model catalogue correction (offline candidate)

Public Google documentation audited 2026-11-14 confirms the old two-ID allowlist was incomplete: it omitted even the provider's configured gemini-3.8-flash default. This follow-up expands exact documented input-model IDs, not a family-prefix guess. The checked-in contracts/gemini-media-evidence.v1.json records per-model URLs, modality quotes, narrower tool evidence, routing/availability notes, and specialized-route/unknown boundaries.

Final head 264b59aea12a1052d2f508facde070eb4c119ef9:
changed-risk source accepted by independent read-only reviewer at 66f92628...,
then one evidence-only wording correction clarifies that the Robotics Python
example uses an uploaded PNG URI while REST demonstrates inlineData.
Affected evidence assertion passes; reviewed production/tests unchanged.
Exact-head Python 3.11/3.12 CI and CLA pass:
run 43615458522.

Model/position matrix

  • 16 user-image IDs: gemini-4-carbon, gemini-4-argon, gemini-4-hydrogen, gemini-3.8-flash, gemini-3.7-flash, gemini-3.6-flash, gemini-3.5-flash, gemini-3.5-flash-lite, gemini-3.1-flash-lite, gemini-3.1-pro-preview, gemini-3.1-pro-preview-customtools, gemini-3-flash-preview, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite, gemini-robotics-er-2-preview.
  • 10 nested tool-image IDs: the 9 explicitly enumerated G3 text-model IDs above (vendor guide explicitly scopes multimodal function responses to G3 series), plus gemini-4-hydrogen (named GenerateContent nested-image example).
  • Unconfirmed/unqualified tool positions: Carbon/Argon, the three 2.5 endpoints, and standard Robotics ER2. They preserve an unavailable warning at the original call-bound position; this is not a claim of vendor image incapability.
  • Specialized image-generation models genuinely accept image inputs but have image/text output and no function calling. Live/TTS/embedding/managed-agent/Decisions/video-output routes have distinct contracts, not implied text-provider parity. Existing specialized image-operation code is unchanged. No blanket gemini-* vision tag.
  • Unresolved *-latest/dated aliases and retired/conflicting IDs remain unknown for this route, not silently revived/resolved or called image-incapable.

get_info/list_models vision tags and get_media_request_support user positions share the same exact catalogue. The structural API remains unchanged; its existing optional reason field now states route/availability caveats. Unknown-position diagnostics say not qualified for this route, not fictional vendor incapability.

Important limits retained

  • Google documents 3.7-flash -> 3.8-flash and 3.5-flash -> 3.6-flash routing at its service. The provider does not reroute locally. Existing-user restrictions on 2.5 and future December 1 routing changes remain disclosed.
  • Gemini 4's unified input/output window and Robotics ER2's smaller window are not qualified by the existing native-budget adapter. Native count remains unavailable on those four IDs; tests do not report a wire projection as successful HTTP provider_count. No unrelated limit/thinking-level/long-decode redesign is hidden in this media change.
  • PNG/JPEG only per frozen producer scope. System/developer/assistant images remain unqualified.
  • Runtime still imports Core only. Retry, expiry/revocation, signature/provenance, private association and pre-emit omission mechanics are unchanged by this catalogue repair.

Offline verification

Keys unset and socket.connect/connect_ex refused before pytest/provider imports, plus maintained autouse fence. Existing isolated deps: Core1.6.1, google-genai2.30.1, HTTPX0.28.1, Pillow12.3.0. No installs, credentialed API listing, inference, config changes, DTU or activation.

  • 187 focused passes against immutable archived F1 candidate 31766b42efe71920386975cf10d136b38b85fc14 (candidate-only, not accepted graph).
  • 1,041 standalone passes, one optional F1-module skip and one live-test deselection, 38 warnings.
  • Actual SDK complete/SSE matrix: 64 user cases (16 IDs x PNG/JPEG x complete/SSE), 40 tool cases (10 IDs x formats/transports), 64 real F1 canonical -> private -> count/send cases (16 IDs), plus prior retry/unknown-send/stream/privacy guards.
  • Literal test catalogue is independent of production declarations; tests require evidence JSON/vision tags/actual wire matrix to agree. Each case verifies valid original pixels/decoded byte SHA/order/MIME/options and diagnostic omission. Native count-unavailable cases are distinguished from mocked real HTTP count cases.
  • New helper/test lint and git diff --check pass. Full-file preexisting lint findings remain baseline, not relabeled green.
  • First standalone run after expansion: 1 maintained expectation failure (3.8-flash formerly had no vision tag), 1,040 passed. Updated that exact documented-model expectation; failed log retained. Earlier live-test incident/failure packet remains unchanged and is not qualification.

Parent exact-head Astra/Fable review, accepted dependency graph/host integration and integration-owned installed qualification remain holds. PR stays draft; no merge or release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant