fix(playground): preserve tool choice and strict setting for Anthropic and Bedrock - #15683
Open
adimalkar wants to merge 1 commit into
Open
fix(playground): preserve tool choice and strict setting for Anthropic and Bedrock#15683adimalkar wants to merge 1 commit into
adimalkar wants to merge 1 commit into
Conversation
…c and Bedrock The server provider clients dropped parts of the tool contract before sending the request, affecting playground calls, evaluator previews, and online evaluator execution, which share these clients. Anthropic tool choice: the `disable_parallel_tool_calls` branch overwrote `params["tool_choice"]` unconditionally, discarding the choice built directly above it. A `specific_function` choice silently became `auto`, and a `none` choice became `auto` as well -- turning "do not call tools" into "call tools freely". Each branch of that block already applies `disable_parallel_tool_use`, so the overwrite only needed to be the fallback for when no explicit choice was given; it is now an `elif`. Anthropic and Bedrock strict: the function tool builders passed only name/schema/description, dropping the stored prompt tool's `strict` setting. Both `anthropic.types.ToolParam` and Bedrock's `ToolSpecificationTypeDef` accept `strict`, and the OpenAI builder already forwards it. Fixes Arize-ai#15644 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #15644
The server provider clients drop parts of the tool contract before the request goes out. Because playground calls, evaluator previews, and online evaluator execution all share these clients, the same loss shows up in each.
1. Anthropic tool choice is overwritten
_anthropic_message_paramsbuildstool_choicefromtools.tool_choice, then immediately discards it:Every branch of the block above already applies
disable_parallel_tool_use, so this overwrite was only ever needed as the fallback for when no explicit choice was given. As written it fires whenever the flag is set, regardless:tool_choicedisable_parallel_tool_callsspecific_functionTrue{"type": "auto", ...}— named tool lostone_or_moreTrue{"type": "auto", ...}— "must call a tool" lostnoneTrue{"type": "auto", ...}— tool use enabled when it was turned offThe
nonecase is worth calling out separately: it isn't in the issue description, but it inverts the user's intent rather than merely relaxing it — "do not call tools" becomes "call tools freely".Fix: make the fallback an
elif, so an explicit choice wins and theautodefault still applies when there is none.2. Anthropic and Bedrock drop the tool's
strictsettingBoth function-tool builders pass only name / schema / description, so
PromptToolFunctionDefinition.strictnever reaches the provider. The OpenAI builder already forwards it:Both target types accept the field, so this is a straightforward omission rather than an unsupported capability:
anthropic.types.ToolParam→strictToolSpecificationTypeDef→strict: NotRequired[bool]Fix: forward it when it is a bool, matching the adjacent
descriptionhandling so an unset value is still omitted.Acceptance criteria
Tests
Seven tests added to
tests/unit/server/api/helpers/test_playground_clients.py, split intentionally:Four fail without the source change — the corrected behaviour:
test_specific_tool_choice_survives_disable_parallel_tool_callstest_tool_choice_none_survives_disable_parallel_tool_callsTestAnthropicStreamingClient::test_function_tool_strict_is_forwardedTestBedrockClient::test_function_tool_strict_is_forwardedThree pass with and without it — guards that this PR does not change behaviour it shouldn't:
test_disable_parallel_tool_calls_without_explicit_choice_falls_back_to_auto— theautofallback still works when no explicit choice is giventest_function_tool_omits_strict_when_unset(Anthropic and Bedrock) — an unsetstrictis still omitted, not sent asNoneVerified by stashing only the source change and re-running: 4 failed, 3 passed.
Verification
pytest tests/unit/server/api/helpers/→ 505 passed, 113 skippedruff checkandruff format --checkclean on both changed files (both confirmed clean atmainfirst, so no unrelated reformatting is mixed in)Note on the local environment
I could not reproduce the CI dependency set exactly, so CI is the real check on this:
requirements/unit-tests.txtdid not resolve for me —litellm>=1.83.14requiresopenai>=2.20.0,<3.0.0, while another pinned dependency requiresopenai>=3.1.0.requirements/type-check.txthits a relatedopenaiconflict.ci.txtplus the test dependencies, excludinglitellmandtype-check.txt. Nothing in the changed code paths touches litellm.Happy to adjust if there is a lockfile or resolver setting I missed.