fix(ollama): make native think configurable via extra_body - #3344
Merged
Conversation
The native structured-output path hardcoded think=False, so gpt-oss models
failed fact extraction (they require a thinking level). Rather than a
model-name heuristic, merge the configured extra_body into the native
/api/chat payload: top-level native fields (think, keep_alive, ...) pass
through, and an "options" sub-dict merges into Ollama's generation options.
Set HINDSIGHT_API_LLM_EXTRA_BODY='{"think": "low"}' for gpt-oss.
Fixes #3246
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes Ollama native structured-output for gpt-oss models, which require a
thinking level (
thinkcannot beFalse) or fact extraction fails.The native
/api/chatpath previously hardcodedthink=False. Instead of amodel-name heuristic (
"gpt-oss" in model), this wires the existingextra_bodyconfig surface into the native payload, so operators can enablethinking — and tune any future native field — without a code change.
What changed
_call_ollama_nativenow merges the configuredextra_body(
HINDSIGHT_API_LLM_EXTRA_BODYand per-op variants) into the native payload,respecting Ollama's two-tier body shape:
think,keep_alive, …) pass through directly."options"sub-dict merges into Ollama's generation options(
seed,top_p,num_ctx, …), with user values winning over computed defaults.thinkstill defaults toFalse; enable it per model via config:This differs from the OpenAI-compatible endpoints, where the SDK flattens
everything to top-level — documented in
configuration.md.Testing
test_ollama_native_think.py: defaultthink=False,extra_bodytop-leveloverride (gpt-oss path), and
optionssub-dict merge (not leaked top-level)../scripts/hooks/lint.shpasses.Notes
Supersedes #3263, which hardcoded
think="low" if "gpt-oss" in model— brittle(guesses the level, no opt-out, bakes one model family into the transport layer).
Fixes #3246