Validate a submit_solution answer against its declared return type - #756
Open
datvo06 wants to merge 2 commits into
Open
Validate a submit_solution answer against its declared return type#756datvo06 wants to merge 2 commits into
datvo06 wants to merge 2 commits into
Conversation
A Template answered by the direct final message has its result validated through Encodable[return_type], so a return type carrying an AfterValidator (a compile gate, a probe) is enforced. A Template answered via submit_solution did not: it returned implementation(*args, **kwargs) unchecked, so the same validator was silently skipped depending only on how the model chose to reply. Run the declared return type's value-validators on the submitted answer. The answer is an already-built value, not the source the encoding's decode expects, so apply only the AfterValidators the Encodable annotation carries -- letting pydantic dispatch (value)/(value, info) and thread the decode context (the Template's lexical scope plus its arguments, as the direct path builds it). Return types without a value-validator are unaffected.
Extracting only AfterValidators was incomplete: it skipped types with no Annotated metadata entirely (int, BaseModel, list), and Before/Wrap validators and nested element validators never ran. The complete check is pydantic's own: validate the answer through the full evaluated encoding. The exception is a decode-style encoding, one carrying a PlainValidator anywhere (a synthesized callable's source, an image's data URL): its core validation is a decode from the wire form, so an already-built value cannot go through it (demonstrated: a built callable raises AttributeError on module_code, a built PIL image fails validation). For those, run only the top-level post-decode AfterValidators, which is the post-decode set by construction. str returns and missing annotations are skipped, matching the direct path's special case. Known parity gap, upstream and shared by both paths: TypeToPydanticType wraps an Annotated return's metadata in a tuple (Annotated[int, (AfterValidator(...),)]), so pydantic ignores user-authored constraints on the direct path and here alike; that is a separate fix.
datvo06
force-pushed
the
dn-validate-final-tool-return
branch
from
August 13, 2026 19:32
b2a793c to
50c6bdb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #694, a Template's return-type validator only runs when the model answers with a direct final message. When it answers via the
submit_solutionfinal tool, the answer is returned unvalidatedThis PR runs the declared return type's value-validators on a
submit_solutionanswer.