fix(ollama): emit streamed tool calls instead of raw JSON content

[arthi-arumugam-git]

TLDR

Problem this solves:

• ollama/ silently drops every tool call when stream is true
• The raw {"name": ..., "arguments": ...} JSON is streamed to the user as content
• finish_reason comes back "stop", so callers never run the tool
• Non-streaming on the same route already returns the tool call correctly

How it solves it:

• Give the streaming iterator the tool-call detection transform_response already has
• Hold response text only while it can still be a JSON object, decide at done

Relevant issues

Fixes #19742

That issue reports the raw JSON blob rendered in the chat window with an ollama/ model configured to stream. It was closed by the stale bot rather than fixed, and it is reproducible on current litellm_internal_staging; the reason it read as unreproducible is that it only happens when stream is true, and the same request without streaming looks fine.

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy on localhost:4000 in front of a local ollama serving qwen2.5:3b. No mocks anywhere; the requests below hit the real model. Ollama is free to run, so this cost time rather than dollars, but it is a real provider call over HTTP end to end.

# config.yaml
model_list:
  - model_name: qwen-local
    litellm_params:
      model: ollama/qwen2.5:3b
      api_base: http://127.0.0.1:11434

python litellm/proxy/proxy_cli.py --config config.yaml --port 4000

The request, identical for both runs, at temperature: 0 and a fixed seed:

curl -sN http://localhost:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "qwen-local",
    "stream": true,
    "temperature": 0,
    "seed": 7,
    "messages": [
      {"role": "user", "content": "Cancel the subscription for maya.iyer@example.com and refund her last payment."}
    ],
    "tools": [
      {"type": "function", "function": {"name": "lookup_account",
        "description": "Look up a customer account by email address",
        "parameters": {"type": "object", "properties": {"email": {"type": "string"}}, "required": ["email"]}}}
    ]
  }'

Before, at 24123269cc (base of this branch). The tool call arrives as 21 content chunks and the turn ends with finish_reason: "stop". Trimmed to the first and last chunks:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"{\""}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"name"}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"\": \""}}]}
...
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"\"}}"}}]}
data: {"id":"chatcmpl-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

Reassembling that stream:

chunks: 21
finish_reason: stop
tool_calls: []
content: "{\"name\": \"lookup_account\", \"arguments\":{\"email\": \"maya.iyer@example.com\"}}"

The caller sees a plain text answer, the tool is never invoked, and that JSON is what the end user reads.

After, at db1cc736ad (this branch), same proxy, same config, same request:

data: {"id":"chatcmpl-52bde277-...","created":1785150640,"model":"qwen-local","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","role":"assistant","tool_calls":[{"id":"call_a90658ba-45c8-430b-ae47-d0ebaf69ce56","function":{"arguments":"{\"email\": \"maya.iyer@example.com\"}","name":"lookup_account"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-52bde277-...","object":"chat.completion.chunk","created":1785150640,"model":"qwen-local","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

chunks: 2
finish_reason: tool_calls
tool_calls: [{"id": "call_...", "function": {"arguments": "{\"email\": \"maya.iyer@example.com\"}", "name": "lookup_account"}, "type": "function", "index": 0}]
content: ""

That now matches what the same request has always returned with "stream": false, which is the behaviour this is being brought in line with:

{"id":"chatcmpl-2d0a168c-...","model":"qwen-local","object":"chat.completion","choices":[{"finish_reason":"tool_calls","index":0,"message":{"role":"assistant","tool_calls":[{"function":{"arguments":"{\"email\": \"maya.iyer@example.com\"}","name":"lookup_account"},"id":"call_e63adf28-...","type":"function"}],"content":null}}],"usage":{"completion_tokens":21,"prompt_tokens":146,"total_tokens":167}}

Plain streaming is unaffected. Same proxy at db1cc736ad, no tools, still token by token:

curl -sN http://localhost:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{"model":"qwen-local","stream":true,"temperature":0,"seed":7,
       "messages":[{"role":"user","content":"In one sentence, what is a refund?"}]}'

chunks: 33
finish_reason: stop
tool_calls: []
content: "A refund is the process of returning money to a customer who has paid for goods or services that were not delivered as agreed or have been found unsatisfactory."

JSON mode without tools is also untouched, still fragment by fragment:

curl -sN http://localhost:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{"model":"qwen-local","stream":true,"temperature":0,"seed":7,
       "response_format":{"type":"json_object"},
       "messages":[{"role":"user","content":"Give me a JSON object with keys city and temp_c for Chennai."}]}'

chunks: 20
finish_reason: stop
tool_calls: []
content: "{\n  \"city\": \"Chennai\",\n  \"temp_c\": 25\n}"

Type

🐛 Bug Fix

Changes

Tools are not a supported OpenAI param on the ollama/ route, so get_optional_params rewrites them into a JSON-only system prompt and sets format: "json" on the request. OllamaConfig.transform_response completes that round trip: it parses the finished body and, when the object carries name and arguments, turns it into a real tool_calls message with finish_reason: "tool_calls". OllamaTextCompletionResponseIterator.chunk_parser never had the second half, so on the streaming path the JSON object was emitted verbatim as content deltas and the turn closed with "stop".

chunk_parser only ever sees fragments, so it cannot parse mid-stream. _hold_back_or_release holds response text back while the accumulated text is still consistent with a JSON object, and releases everything held the moment it cannot be one. Prose fails that check on its first fragment, so plain text streaming keeps flowing exactly as before. At done, _build_tool_call validates the held text and either emits a chunk carrying tool_use and finish_reason: "tool_calls", or flushes the held text as content with finish_reason: "stop" so nothing is ever dropped.

Detection is not on by default, and it is keyed off the functions the request actually offered. By the time a request reaches the provider the tools survive only as prompt text, so get_optional_params passes on the schemas it rewrote as prompted_tool_calls; transform_request consumes that, so it never reaches ollama, and hands the function names through get_model_response_iterator to the iterator. A request that offered no tools never buffers a byte and cannot have its content or finish_reason changed. That includes response_format: {"type": "json_object"}, which streams fragment by fragment exactly as it does today. A body naming a function the request did not offer also stays content, so a model that invents a name cannot have it synthesised into a call.

Within a request that did offer tools, what counts as a tool call is deliberately stricter than the non-streaming test. _OllamaJsonModeToolCall requires exactly the object factory.function_call_prompt asks ollama to produce: a string name, an arguments object, and no other keys. transform_response only checks that name and arguments are present, so any body this converts is a body that path already converts today; the streaming path cannot reinterpret content the non-streaming path would have left alone. A JSON response that merely happens to carry a name, or one whose arguments is not an object, stays content and keeps finish_reason: "stop".

Within such a request, a JSON body that is not a tool call arrives in the final chunk rather than fragment by fragment, because a JSON object cannot be told apart from a tool call until it is complete. The content itself is unchanged.

ollama_chat/ is untouched. It gets tool_calls natively from /api/chat and needs none of this.

Ten tests in tests/test_litellm/llms/ollama/test_ollama_completion_transformation.py cover this. Two reproduce the bug: one drives litellm.completion(..., stream=True) with tools against a transport that fragments the body the way ollama does, and one exercises chunk_parser directly. The rest are guards: prose arriving one delta per fragment, a non-tool JSON body delivered in full as content, a JSON object with extra keys alongside name/arguments staying content, arguments that is not an object staying content, a tool-shaped body streaming untouched when the request offered no tools, a body naming a function the request never offered staying content, detection being armed only with the functions get_optional_params actually rewrote, and the marker never reaching ollama.

Reverting transformation.py to base with the tests kept fails the end-to-end one on exactly the reported symptom, while the guard that a tool-less request is left alone keeps passing:

$ pytest tests/test_litellm/llms/ollama/test_ollama_completion_transformation.py -q \
    -k "TestOllamaStreamingToolCallsEndToEnd"
>       assert streamed_content == ""
E       assert '{"name": "lo...xample.com"}}' == ''
E         + {"name": "lookup_account", "arguments": {"email": "maya.iyer@example.com"}}

FAILED ...::TestOllamaStreamingToolCallsEndToEnd::test_streamed_tool_call_is_not_delivered_as_content
3 failed, 1 passed, 18 deselected

The other two failures there are the gating tests, which cannot run at base because the marker and the parameter they assert on do not exist yet. test_detection_is_off_for_a_request_that_offered_no_tools passes at base, which is the point of it.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR adds request-scoped streamed tool-call detection for the Ollama completion adapter.

• Carries offered function names from optional-parameter preprocessing into the response iterator.
• Buffers JSON-object candidates until completion, emitting validated offered functions as tool calls and preserving other responses as content.
• Adds focused tests for tool calls, ordinary text, non-tool JSON, unoffered functions, and internal marker handling.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the current code scopes detection to requests with prompted tools and validates returned function names against the functions offered by that request.

Important Files Changed

Filename	Overview
litellm/llms/ollama/completion/transformation.py	Adds request-scoped buffering and validation that converts completed Ollama JSON responses into streamed tool calls only for functions offered by the request.
litellm/utils.py	Preserves the tool schemas rewritten into the Ollama prompt as internal request metadata for response validation.
tests/test_litellm/llms/ollama/test_ollama_completion_transformation.py	Adds regression and guard coverage for streamed tool conversion, ordinary content, JSON content, offered-name scoping, and request-marker removal.

_{Reviews (5): Last reviewed commit: "fix(ollama): emit streamed tool calls in..." | Re-trigger Greptile}

[veria-ai]

PR overview

This pull request updates Ollama completion transformation so streamed JSON tool-call responses are emitted as structured tool calls instead of raw content.

One issue has already been addressed, but streamed JSON output can still grow without a byte limit and incur quadratic copying. An API caller able to request a large response could cause excessive memory and CPU consumption in the proxy, creating a limited denial-of-service risk.

Open issues (1)

• Low: Unbounded streamed-output buffer — litellm/llms/ollama/completion/transformation.py:512

Fixed/addressed: 1 · PR risk: 5/10

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[arthi-arumugam-git]

Reworked to address the review. The conversion is now gated on the exact object litellm's own tool prompt asks ollama for: a string name, an arguments object, and no other keys. That is strictly narrower than the check transform_response already applies to the same body, so nothing the non-streaming path leaves as content can be reinterpreted as a tool call here.

Added two guards for the cases raised: a JSON body carrying extra keys alongside name/arguments, and one whose arguments is not an object. Both stay content with finish_reason: "stop".

@greptileai

[arthi-arumugam-git]

Pushed 439accfa55, which makes the conversion conditional rather than an unconditional shape heuristic.

transform_request now arms detection only when the request carries format: "json", the same signal transform_response keys off, and passes that into the iterator. A request that never went through the tool rewrite never buffers a byte, so its content and finish_reason cannot change.

Within JSON mode the accepted shape is exactly what factory.function_call_prompt asks ollama for: a string name, an arguments object, and no other keys. That is strictly narrower than the check transform_response already applies to the same body, so streaming cannot reinterpret anything the non-streaming path would leave as content.

New guards: a tool-shaped body streams untouched when the request never set format: "json"; a JSON body with extra keys alongside name/arguments stays content; arguments that is not an object stays content.

@greptileai

[veria-ai]

Low: Unbounded streamed-output buffer

An API caller can prompt Ollama to generate a large JSON object and select a large max_tokens; because the output continues to start with {, this concatenation retains the entire response and repeatedly copies the growing string, consuming unbounded memory and quadratic CPU in the proxy. Accumulate fragments without repeated copying and enforce a byte limit, releasing the buffered value as ordinary content once that limit is reached.

[arthi-arumugam-git]

Pushed 225c000715, which closes the tool-context gap directly.

By the time a request reaches the provider its tools survive only as prompt text, so nothing downstream could tell a tool request apart from any other JSON-mode request. get_optional_params now marks the requests it actually rewrote with prompted_tool_calls, transform_request consumes that marker so it never reaches ollama, and it is what arms the iterator.

A request that offered no tools never buffers a byte, response_format: {"type": "json_object"} included. Verified against a live proxy in front of a real ollama: tool call streaming returns a proper tool_calls delta with finish_reason: "tool_calls", prose still streams in 33 fragments, and JSON mode without tools still streams in 20 fragments rather than arriving at the end.

New guards: detection armed only for rewritten requests, the marker never appearing in the ollama request body, and a tool-shaped body streaming untouched when no tools were offered.

@greptileai

[arthi-arumugam-git]

Pushed db1cc736ad. Synthesised tool calls are now restricted to functions the request actually offered.

get_optional_params passes on the schemas it rewrote into the prompt, transform_request consumes them so they never reach ollama, and the function names are what arm the iterator. _build_tool_call rejects any name not in that set, so a model inventing a function cannot have it synthesised into a call, and a request that offered no tools never buffers a byte at all.

Verified against a live proxy in front of a real ollama: the tool call streams as a proper tool_calls delta with finish_reason: "tool_calls", prose still streams in 33 fragments, and json_object mode without tools still streams in 20 fragments.

@greptileai

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing arthi-arumugam-git:litellm_ollama_stream_tool_calls (db1cc73) with litellm_internal_staging (2412326)}