[mannuch]

This adds support for structured dspy.History inside the RLM module.

Currently, RLM only supports a REPLHistory that records the reasoning, code, and REPL output for each iteration. At the end of each call, the REPL history can be inspected by accessing the history field on the resulting Prediction. While this works well for debugging purposes, it doesn't support recording structured inputs and/or outputs and there is no way to pass history from prior calls into subsequent calls. This makes the RLM module incapable of continuing previous interactions.

Structured history support

RLM can adopt a similar approach to history as the ReActV2 module. As a refresher, the ReActV2 module contains a dspy.History object whose messages list contains one structured event per loop turn. The event contains the original signature's input fields (only at the start of a new turn), next_thought reasoning content, a tool_calls field of type dspy.ToolCalls containing structured tool calls and their results, and the original signature's output fields when the turn successfully calls the submit tool.

We take a very similar approach here for RLM. Specifically, each loop turn is structured the following way:

# query -> answer: int
{
    "query": "Double twenty",  # input fields, included only at the start of a new turn
    "variables_info": ..., # The variables that are available in the REPL
    "iteration": "1/5",
    "repl_entry": dspy.REPLEntry(
        reasoning: "First explore the data", 
        code: "x = 20\nprint(f'x = {x}')",  # code to execute
        output: "x = 20"  # the executed code's REPL output
    ), 
}
...
{
    "variables_info": ...,
    "iteration": "3/5",
    "repl_entry": dspy.REPLEntry(
        reasoning: "Now compute and return", 
        code: "y = x * 2\nSUBMIT(y)",
        output: "FINAL:  {'answer': 40}" 
    ),
    "answer": "40"  # output fields, included when the `SUBMIT` tool is successfully called
}

As seen above, each event contains a repl_entry field of type REPLEntry with reasoning, the code output by the LM to be executed, and the REPL output of the code. Each event also carries variables_info outlining the variables available in the REPL (the RLM signature's inputs are represented here) as well as the current iteration of the turn. The start of each new turn contains the signature's input fields (query), while the final message in a turn contains the signature's output fields (answer). Intermediate turns will only contain the repl_entry, variables_info, and iteration fields.

The extract fallback case

When the RLM has reached max_iters iterations, but hasn't yet SUBMITted a final answer successfully, the module falls back to extracting the answer from the REPL trajectory so far. In this case, the history would record an event that contains only the final outputs and repl_entry set to ExtractFallbackMarker:

# query -> answer: int, max_iters = 2
{
    "query": "Double ten, then add five",
    "variables_info": ...,
    "iteration": "1/2",
    "repl_entry": dspy.REPLEntry(
        reasoning: "First explore the data", 
        code: "x = 10\nprint(f'x = {x}')", 
        output: "x = 10"
    ), 
}
{
    "variables_info": ...,
    "iteration": "2/2", # <-- max iterations reached!
    "repl_entry": dspy.REPLEntry(
        reasoning: "Compute double 10", 
        code: "y = x * 2\nprint(f'y = {y}')", 
        output: "y = 20"
    ), 
}
{
    "repl_entry": ExtractFallbackMarker(), # Marker type that tells us this history event was from the extract fallback
    "answer": "25"  # output fields extracted
}

Note: repl_entry is always set to either REPLEntry or ExtractFallbackMarker. This key is how readers of the history messages know it's an RLM history message.

Formatting the conversation history for the LM

An event's REPLEntry is formatted into conversation history as follows:

• The reasoning and code are formatted into a "role": "assistant" message's content
• The output is formatted into a "role": "user" message's content

With the ChatAdapter, history is formatted like:

{
    "role": "user",
    "content": "[[ ## variables_info ## ]]\nVariable: `query` (access it in your code)\nType: str\nTotal length:  10 characters\nPreview:\n```\nDouble ten\n```\n\n[[ ## iteration ## ]]\n1/5\n\n[[ ## completed ## ]]\n"
}
{
    "role": "assistant",
    "content": "[[ ## reasoning ## ]]\nFirst explore the data\n\n[[ ## code ## ]]\nx = 10\nprint(f'x = {x}')\n\n[[ ## completed ## ]]\n"
}
{
    "role": "user",
    "content": "[[ ## repl_output ## ]]\nx = 10"
}
{
    "role": "assistant",
    "content": "[[ ## reasoning ## ]]\nNow compute and return\n\n[[ ## code ## ]]\ny = x * 2\nSUBMIT(y)\n\n[[ ## completed ## ]]\n"
}
{
    "role": "user",
    "content": "[[ ## repl_output ## ]]\nFINAL:  {'answer': 40}"
}

A replayed history where an extract fallback occurred would be like:

...
{
    "role": "user",
    "content": "[[ ## repl_output ## ]]\ny=20"  # Last REPL output before max iterations reached
}
{
    "role": "assistant",
    "content": "[[ ## answer ## ]]\n5" # Extracted final answer
}

This happens inside Adapter.format_conversation_history — it looks for history messages containing a repl_entry key containing REPLEntry | ExtractFallbackMarker and handles them accordingly.

Output Truncation

When formatting the REPLEntry's output, we truncate it according to a provided max_output_chars argument. The format as implemented in this PR as follows:

x = 10
... (100 characters omitted) ...
z = 20

This PR removes the output header (Output (500 chars)...) that specifies the total characters in the output. This was done because it felt redundant with the input field marker ([[ ## repl_output ## ]]) in the user message content inside the conversation history.

History processing hook

There is also a history_processor hook added to the module. This enables users to run e.g. compaction strategies. The hook is a Callable[[dspy.History], dspy.History | None]. A user can provide this during the instantiation of the RLM module passing RLM("query -> answer", max_iters=5, history_processor=my_processor) or when calling it my_rlm(interpreter, my_processor, ...). The per-call processor overrides any processor set at instantiation.

This design is intentionally very similar to what's proposed in #10217.
However, a key difference is that this implementation does not treat a return of None as a mistake/error. It allows for callbacks that simply mutate the passed in history in-place. It does degrade gracefully though on exceptions thrown from the processor hook, enabling the run to continue with the unprocessed history.

Passed-in History is Input Only

The RLM module treats the user's passed-in history as read only. It doesn't modify it. It makes its own copy of the passed-in history. The returned history on the resulting Prediction is the continuation of the user's passed-in history, but is a separate object.

This implies a kind of usage pattern where the caller manages history between RLM invocations, like so:

def process_history(history: dspy.History):
    # ...process / compact / etc.
    return history

rlm = RLM("query -> answer", max_iters=5, history_processor=process_history)

result_one = rlm(query="Double ten")
result_two = rlm(query="Double twenty, then add 5", history=result_one.history)
full_history = result_two.history # result_two's history is continuation of result_one's history

REPL History is no longer an input

Now that the prior interaction history and context is now properly replayed to the LM, this PR no longer has the REPL trajectory repl_history: REPLHistory as an input into the inner generate_action or extract Predict signatures.

REPL History is still returned to the caller

The REPL history is still returned on the resulting Predicition, but as repl_trajectory. It contains a dictionary representation of the REPLHistory.

rlm = RLM("query -> answer")
result = rlm(query="Double ten")
print(result.repl_trajectory) 
# [ {"reasoning": "First explore the data", "code": "x = 10\nprint(f'x = {x}')", "output":  "x = 10"} ... ]

Happy to discuss / modify any of the design decisions proposed here!

---

This PR addresses issue #10334

[greptile-apps]

Confidence Score: 5/5

[Medium impact] No outstanding blocking findings remain; the reviewed changes address both numbered findings.

Summary

Adds structured dspy.History support to RLM so callers can carry prior turns into later calls.

• RLM calls can continue from saved conversation history.
• Adapters replay RLM steps as conversation messages.
• Test support changes leave product behavior unchanged.

_{Reviews (13) · Last reviewed commit: "Reserve RLM runtime options and history-..." · Reviewed by Greptile}

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[isaacbmiller]

Hey Matt — thanks for raising this, and sorry for the slow response!

We’ve since merged #10643, which makes RLM use dspy.Reasoning: native reasoning when supported, otherwise a short textual plan. The old str field requested ordinary text; the fallback plan still replays through History.

The newer XMLAdapter is in #10402 — typed tags with literal leaves, on branch isaac/xml-adapter-literal-tags. It’s still open and im actively working on it, so rebasing onto main alone won’t include it.

You’re right that the native path currently omits prior reasoning from ordinary History replay, even when the provider exposes it. We’re tracking exposed-thinking replay and provider-native continuation separately (#10675), so you don’t need to solve those in this PR.

I tested 30 paired cases per benchmark/model. Before → PR compares newer main + fixed XMLAdapter against the same integration + your History PR, with JSON fallback:

Model	BrowseComp+ before → PR	OOLONG before → PR	Failures / 60 before → PR
Luna	67.07 → 75.69	82.60 → 85.04	0 → 0
Haiku 4.5	59.11 → 66.86	52.69 → 68.50	0 → 0
Kimi K2.6	69.82 → 54.57	73.24 → 64.92	9 → 16

The fixed XML + newer reasoning stack substantially helps: Kimi’s PR failures fell from 40/60 on the older stack to 16/60.

Luna BrowseComp cached-input share also improved 8.43% → 78.36%, with SDK-estimated cost falling $1.410 → $0.655 despite more input tokens.

Three remaining issues:

1. Kimi extraction: 13 PR failures return another action (reasoning/code) instead of the final answer/response, even after JSON retry. This is the wrong output schema, not merely malformed XML. Three other failures are unexplained HTTP 400s. From some testing I did, Across three replay cases, supplying history as reference text rather than assistant turns improved schema success 4/9 → 8/9.

2. Ownership: returning an external History from history_processor lets RLM mutate it. Editing the supplied private copy works correctly.

3. Saving/loading History loses the REPL transcript: restored entries become plain dictionaries, so Chat/JSON/XML replay omits their code and outputs:

   saved = history.model_dump_json()
   restored = dspy.History.model_validate_json(saved)

The repro bundle includes offline contract checks and five captured extraction fixtures, with controls, exact revisions, and setup instructions. After setup:

python check_history.py                              # offline contracts
python replay_extraction.py --live --case browsecomp-05 \
  --variant both --repeats 3 --out runs/before         # paid calls

rlm-pr-review.zip

[mannuch]

Hey Isaac, thanks for the feedback! Glad to see the positive results in the benchmarking.

I've addressed the issues you pointed out:

• dspy.History Ownership: now, a dspy.Historyreturned by a history_processor is deep-copied before the RLM mutates it internally.
• RLM history survives JSON / plain dict roundtrips: REPLEntry and ExtractFallbackMarker now serialize with a __dspy_repl__ tag.
• Extract Signature uses the current turn's repl_history only: The extraction signature now doesn't pass history as one of its inputs. Only the current turn's repl_history is passed, along with variables_info. I tried keeping both history and repl_history as inputs on the signature, but only got 6/9 valid answers. Using repl_history only gave 9/9 valid answers. It seems that having repl_history only prevents Kimi from sending back {"reasoning", "code"} instead of the actual answer.

Thanks for providing the repro bundle! It didn't initially work with the fix that removeshistory — so I used the following script to evaluate: replay_turn_extraction.py