Skip to content

fix(ollama): make native think configurable via extra_body - #3344

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/ollama-native-extra-body-3246
Aug 10, 2026
Merged

fix(ollama): make native think configurable via extra_body#3344
nicoloboschi merged 1 commit into
mainfrom
fix/ollama-native-extra-body-3246

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

Fixes Ollama native structured-output for gpt-oss models, which require a
thinking level (think cannot be False) or fact extraction fails.

The native /api/chat path previously hardcoded think=False. Instead of a
model-name heuristic ("gpt-oss" in model), this wires the existing
extra_body config surface into the native payload, so operators can enable
thinking — and tune any future native field — without a code change.

What changed

_call_ollama_native now merges the configured extra_body
(HINDSIGHT_API_LLM_EXTRA_BODY and per-op variants) into the native payload,
respecting Ollama's two-tier body shape:

  • Top-level native fields (think, keep_alive, …) pass through directly.
  • An "options" sub-dict merges into Ollama's generation options
    (seed, top_p, num_ctx, …), with user values winning over computed defaults.

think still defaults to False; enable it per model via config:

HINDSIGHT_API_LLM_EXTRA_BODY='{"think": "low"}'   # gpt-oss
HINDSIGHT_API_LLM_EXTRA_BODY='{"options": {"seed": 42, "top_p": 0.9}}'

This differs from the OpenAI-compatible endpoints, where the SDK flattens
everything to top-level — documented in configuration.md.

Testing

  • test_ollama_native_think.py: default think=False, extra_body top-level
    override (gpt-oss path), and options sub-dict merge (not leaked top-level).
  • ./scripts/hooks/lint.sh passes.

Notes

Supersedes #3263, which hardcoded think="low" if "gpt-oss" in model — brittle
(guesses the level, no opt-out, bakes one model family into the transport layer).

Fixes #3246

The native structured-output path hardcoded think=False, so gpt-oss models
failed fact extraction (they require a thinking level). Rather than a
model-name heuristic, merge the configured extra_body into the native
/api/chat payload: top-level native fields (think, keep_alive, ...) pass
through, and an "options" sub-dict merges into Ollama's generation options.

Set HINDSIGHT_API_LLM_EXTRA_BODY='{"think": "low"}' for gpt-oss.

Fixes #3246
@nicoloboschi
nicoloboschi merged commit 3a4b021 into main Aug 10, 2026
209 of 210 checks passed
@nicoloboschi
nicoloboschi deleted the fix/ollama-native-extra-body-3246 branch August 10, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

retain: every fact extraction fails on Ollama + gpt-oss — hardcoded think: False yields empty content under format

1 participant