Skip to content

fix(playground): preserve tool choice and strict setting for Anthropic and Bedrock - #15683

Open
adimalkar wants to merge 1 commit into
Arize-ai:mainfrom
adimalkar:fix/anthropic-bedrock-tool-contract
Open

fix(playground): preserve tool choice and strict setting for Anthropic and Bedrock#15683
adimalkar wants to merge 1 commit into
Arize-ai:mainfrom
adimalkar:fix/anthropic-bedrock-tool-contract

Conversation

@adimalkar

Copy link
Copy Markdown

Fixes #15644

The server provider clients drop parts of the tool contract before the request goes out. Because playground calls, evaluator previews, and online evaluator execution all share these clients, the same loss shows up in each.

1. Anthropic tool choice is overwritten

_anthropic_message_params builds tool_choice from tools.tool_choice, then immediately discards it:

                else:
                    assert_never(tc.type)
            if tools.disable_parallel_tool_calls:          # ← runs after, unconditionally
                params["tool_choice"] = ToolChoiceAutoParam(
                    type="auto", disable_parallel_tool_use=True
                )

Every branch of the block above already applies disable_parallel_tool_use, so this overwrite was only ever needed as the fallback for when no explicit choice was given. As written it fires whenever the flag is set, regardless:

tool_choice disable_parallel_tool_calls Sent before this PR
specific_function True {"type": "auto", ...} — named tool lost
one_or_more True {"type": "auto", ...} — "must call a tool" lost
none True {"type": "auto", ...}tool use enabled when it was turned off

The none case is worth calling out separately: it isn't in the issue description, but it inverts the user's intent rather than merely relaxing it — "do not call tools" becomes "call tools freely".

Fix: make the fallback an elif, so an explicit choice wins and the auto default still applies when there is none.

2. Anthropic and Bedrock drop the tool's strict setting

Both function-tool builders pass only name / schema / description, so PromptToolFunctionDefinition.strict never reaches the provider. The OpenAI builder already forwards it:

# OpenAI, existing
strict=f.strict if isinstance(f.strict, bool) else None,

Both target types accept the field, so this is a straightforward omission rather than an unsupported capability:

  • anthropic.types.ToolParamstrict
  • Bedrock ToolSpecificationTypeDefstrict: NotRequired[bool]

Fix: forward it when it is a bool, matching the adjacent description handling so an unset value is still omitted.

Acceptance criteria

  • Anthropic keeps a specific tool choice when parallel tool use is disabled
  • Anthropic sends the supported strict tool field from the stored prompt tool
  • Bedrock sends the supported strict tool field from the stored prompt tool
  • Request-shape tests cover the cases above

Tests

Seven tests added to tests/unit/server/api/helpers/test_playground_clients.py, split intentionally:

Four fail without the source change — the corrected behaviour:

  • test_specific_tool_choice_survives_disable_parallel_tool_calls
  • test_tool_choice_none_survives_disable_parallel_tool_calls
  • TestAnthropicStreamingClient::test_function_tool_strict_is_forwarded
  • TestBedrockClient::test_function_tool_strict_is_forwarded

Three pass with and without it — guards that this PR does not change behaviour it shouldn't:

  • test_disable_parallel_tool_calls_without_explicit_choice_falls_back_to_auto — the auto fallback still works when no explicit choice is given
  • test_function_tool_omits_strict_when_unset (Anthropic and Bedrock) — an unset strict is still omitted, not sent as None

Verified by stashing only the source change and re-running: 4 failed, 3 passed.

Verification

  • pytest tests/unit/server/api/helpers/505 passed, 113 skipped
  • ruff check and ruff format --check clean on both changed files (both confirmed clean at main first, so no unrelated reformatting is mixed in)
  • Source diff is +5 / −1

Note on the local environment

I could not reproduce the CI dependency set exactly, so CI is the real check on this:

  • requirements/unit-tests.txt did not resolve for me — litellm>=1.83.14 requires openai>=2.20.0,<3.0.0, while another pinned dependency requires openai>=3.1.0. requirements/type-check.txt hits a related openai conflict.
  • I ran the suite with ci.txt plus the test dependencies, excluding litellm and type-check.txt. Nothing in the changed code paths touches litellm.

Happy to adjust if there is a lockfile or resolver setting I missed.

…c and Bedrock

The server provider clients dropped parts of the tool contract before sending
the request, affecting playground calls, evaluator previews, and online
evaluator execution, which share these clients.

Anthropic tool choice: the `disable_parallel_tool_calls` branch overwrote
`params["tool_choice"]` unconditionally, discarding the choice built directly
above it. A `specific_function` choice silently became `auto`, and a `none`
choice became `auto` as well -- turning "do not call tools" into "call tools
freely". Each branch of that block already applies `disable_parallel_tool_use`,
so the overwrite only needed to be the fallback for when no explicit choice was
given; it is now an `elif`.

Anthropic and Bedrock strict: the function tool builders passed only
name/schema/description, dropping the stored prompt tool's `strict` setting.
Both `anthropic.types.ToolParam` and Bedrock's `ToolSpecificationTypeDef`
accept `strict`, and the OpenAI builder already forwards it.

Fixes Arize-ai#15644

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adimalkar
adimalkar requested a review from a team as a code owner August 27, 2026 05:43
Copilot AI lite review requested due to automatic review settings August 27, 2026 05:43
@github-project-automation github-project-automation Bot moved this to 📘 Todo in phoenix Aug 27, 2026
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Aug 27, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added the triage issues that need triage label Aug 27, 2026
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@adimalkar

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS This PR changes 0-9 lines, ignoring generated files. triage issues that need triage

Projects

Status: 📘 Todo

Development

Successfully merging this pull request may close these issues.

[server] preserve required and strict tool settings for Anthropic and Bedrock

2 participants