fix(2119): stop reporting empty --model formal-ai runs as successes - #2120
Merged
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #2119
Downloads the PR/issue metadata, conversation comments, review comments and full solution-draft logs of the three konard/test-hello-world-* reproductions cited in issue #2119 so the case study analysis is based on immutable local evidence.
`--model formal-ai` runs reported "0 input, 0 output" tokens even though the tools produced real usage. Formal AI emits pretty-printed JSON objects rather than NDJSON, and codex writes partial lines, so the per-line parsers silently dropped every event that carried the usage metadata. Adds src/json-stream.lib.mjs, a shared incremental framer that accepts both NDJSON and multi-line/pretty-printed JSON and buffers partial lines, and routes the agent and codex stream readers through it. Reproduction: konard/test-hello-world-019fb330-00e1-73b9-955e-f357a1600d5b#2
The credential sanitizer treated token telemetry (`"total_tokens":44511`, `tokens=44511`, `"maxTokens": 8192`) and JSON structural braces as secrets, so published logs and comments in the reproduction PRs showed redacted counters and mangled JSON, hiding the real usage numbers. Reproduction: konard/test-hello-world-019fb331-c107-78c7-8ff6-9f127a3c593c#2
The reproduction PRs attributed `--model formal-ai` to "OpenCode Zen" or "Anthropic" and billed it: the claude path published $0.252315 taken from Anthropic's `total_cost_usd`, and the public pricing estimate read "unknown". Formal AI is served by Link.Assistant and is free. Adds src/formal-ai-pricing.lib.mjs as the single source of the provider name and the zero-cost override, and applies it at the cost boundary of all six tools (claude, agent, opencode, codex, gemini, qwen) plus the agent commander and the cost-info renderer, which now prints "$0.00 (Free model)" instead of "unknown". Reproductions: - konard/test-hello-world-019fb330-fa49-7c9d-a664-b7ea33bb698a#2 - konard/test-hello-world-019fb330-00e1-73b9-955e-f357a1600d5b#2
Hive Mind had two independent auto-restart subsystems, each with its own counter reading the same --auto-restart-max-iterations flag. solve.mjs runs both in one process, so a limit of 5 permitted 10 AI sessions, and the reproduction PR shows the two incompatible labels the issue calls out: "Auto-restart triggered (iteration 1)" next to "Auto-restart 1/5 Log". Neither path failed at the limit, so a run that never resolved its blocker still exited 0 with the uncommitted work discarded along with the temporary clone. - src/auto-restart-budget.lib.mjs: one process-wide iteration counter both loops claim from, and one N/M label formatter (N alone when the limit is 0/unlimited). - src/auto-restart-exhaustion.lib.mjs: the single exhaustion path - log, auto- commit and push the uncommitted work via the existing critical-error recovery helper, post one "limit reached" comment, and record the failure. - solve.finalize.lib.mjs exits 1 when the budget was exhausted, so the result is actually visible instead of being reported as a success. Reproduction: konard/test-hello-world-019fb330-fa49-7c9d-a664-b7ea33bb698a#2
All three reproduction pull requests were titled `'Implement Hello World in
Scala'` with the single quotes as literal characters.
Root cause: command-stream shell-escapes every interpolated value, quoting it
whenever it contains a character the shell would split on. Wrapping the
placeholder in quotes - `--title "${updatedTitle}"` - therefore passes
`--title "'Implement Hello World in Scala'"` and the quotes land in the title.
Values without spaces were unaffected, which is why it went unnoticed.
Fixed at every such site in src/ and scripts/ (PR titles and body files, the
YouTrack issue sync, the claude runtime switch, reviewers-hive, the changeset
and release scripts). Where the quotes belonged to an inner language the
expression is now assembled in JS and passed as one argument: the fork-lookup
jq filters and the issue/PR-link GraphQL queries.
tests/test-shell-quoting-2119.mjs pins the command-stream behaviour that caused
this and scans src/ and scripts/ so the pattern cannot come back.
…to merge
Two of the three reproduction runs produced a pull request with an empty net
diff (the AI tool wrote nothing, so the branch held only the solver's own
scaffolding commit and a revert of it), yet both published
### Changes
- 1 file(s) modified
- 1 line(s) added
and the Kotlin run additionally posted "## ✅ Ready to merge ... - No pending
changes". Merging that would have closed the issue with nothing implemented.
src/pull-request-changes.lib.mjs is now the single place that answers "does
this pull request change anything", used by both the description writer and the
mergeability watcher. A diff that could not be read is reported as unmeasured
rather than as empty, so a transient API failure cannot trigger restarts.
Formal AI writes a .formal-ai/ plan directory into the workspace it runs in, so
every uncommitted-changes check saw
?? .formal-ai/
📝 Found uncommitted changes
🔄 AUTO-RESTART: Restarting Agent to handle uncommitted changes...
and restarting recreated the directory, so the blocker could never clear. The
same state reached 'git add -A' on the auto-commit paths, which would have
published a tool's private scratch files in the user's pull request.
This generalizes the .playwright-mcp/ special case from #1124: the paths are
written to .git/info/exclude, so git itself stops reporting them and all eight
copies of checkForUncommittedChanges agree without each needing its own filter.
The exclude file is local to the clone, so nothing leaks into the diff.
The Kotlin reproduction run posted
## Working session summary
The `pwd` command completed. Output:
/tmp/gh-issue-solver-1785421161275
on a pull request whose diff was empty. The AI tool returning nothing useful is
an upstream problem; what Hive Mind owns is that the comment read as a report of
completed work and published the solver's private workspace path.
The summary now carries an explicit notice when the pull request still contains
no changes, and solver workspace paths are replaced with <workspace>. An
unreadable diff produces no notice, so a transient API failure cannot turn into
a false "nothing was implemented" claim.
`formal-ai with <tool> --verbose` emits pretty-printed, multi-line JSON records. Commit 2119700 fixed the resulting "Token usage: 0 input, 0 output" for agent, opencode and codex by framing records on balanced JSON values instead of on newlines; gemini and qwen kept their own line-based parsers and so still had the identical defect. Both now share `takeJsonRecords` from json-stream.lib.mjs, which also handles records concatenated without a separator (issue #1250) and records split across process chunks. tests/test-formal-ai-uniform-tools-2119.mjs pins the uniformity down: same stream shapes, same token accounting and the same Link.Assistant $0.00 pricing across all six formal-ai tools.
The Rust reproduction PR contains exactly one file - hive-mind's own auto-generated .gitkeep placeholder - so the change stats reported "1 file changed" for a run that implemented nothing. pull-request-changes.lib.mjs now splits the diff per file and drops the sections whose content matches what solve.auto-pr.lib.mjs writes into .gitkeep / CLAUDE.md, exposes placeholderOnly, and offers buildEmptyPullRequestBlocker() so the restart reason names the placeholder. Matching is on the generated content, so a repository's own .gitkeep stays a real change.
Compiles every artifact of the three 2026-07-30 reproduction runs into docs/case-studies/issue-2119: the issue and its comments, all six test-hello-world repositories (three 2025 baselines for contrast), the three session logs, the PR/issue JSON, the net diffs and the URLs of all 13 published session logs. README.md reconstructs the timeline of each run, lists every requirement from the issue with where it is addressed, gives the root cause and fix of all 13 Hive Mind defects, records the two deliberate non-changes and why, reviews the existing libraries considered for each fix, and states where the data was insufficient and what instrumentation was added. upstream-formal-ai.md is the consolidated Formal AI report: five defects with reproductions, workarounds and code-level suggestions, framed as a request for generalization and self-healing rather than per-symptom patches.
This was referenced Jul 30, 2026
Files the consolidated Formal AI report as formal-ai#879 and, from the same logs, three Agent CLI defects as agent#285: unknown model ids fall through to the without-todo prompt, that prompt tells the model it is opencode and routes feedback to sst/opencode, and session records carry a hard-coded agent-cli-1.0.0 version. Section 9 is corrected too - the system prompt *is* in the archived log, in the request bodyPreview records, which is how the prompt-selection defect was found.
…batim The release workflow requires exactly one changeset per pull request and forbids manual package.json version edits, so the bump is declared here. The archived case-study data is raw evidence, so it is added to .prettierignore alongside every earlier case study's data directory rather than reformatted.
.gitkeep was committed to open this pull request; the solution replaced it, so it is not part of the change. This is the same placeholder D13 teaches the change counter to ignore.
konard
changed the title
--model formal-ai not working on simplest hello world--model formal-ai runs as successes
konard
marked this pull request as ready for review
July 30, 2026 17:56
Contributor
Author
Working session summaryPR 2120 is finished and green. Self-review of the full diff — completed the remaining portion (
The only pre-existing tests I modified are in CI. Every code-bearing job had been skipped on this branch, because the last two pushes were docs/chore-only and Final state. Working tree clean, 16 commits ahead, Upstream reports filed as the issue asked: formal-ai#879 and agent#285. One caveat worth stating plainly: the local This summary was automatically extracted from the AI working session output. |
Contributor
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $52.696379📊 Context and tokens usage:Claude Opus 5: (13 sub-sessions)
Total: (50.5K new + 1.2M cache writes + 56.7M cache reads) input tokens, 479.4K output tokens, $52.674294 cost Claude Haiku 4.5:
Total: 9.4K input tokens, 532 output tokens, $0.022085 cost 🤖 Models used:
📎 Log file uploaded as Gist (24200KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Contributor
Author
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
Contributor
Author
|
Both upstream issues are closed and new versions releases, double check that this issue is fully implemented and prepare pull request to be merged. |
konard
marked this pull request as draft
August 1, 2026 06:58
Contributor
Author
|
🤖 AI Work Session Started Starting automated work session at 2026-08-01T06:58:39.683Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback. |
Contributor
Author
|
Double-check complete:
The PR description has been updated with the released upstream fixes and final verification. |
konard
marked this pull request as ready for review
August 1, 2026 08:04
Contributor
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
📊 Context and tokens usage:
Total: (409.6K + 21.8M cached) input tokens, 35.1K output tokens, $27.460299 cost 🤖 Models used:
📎 Log file uploaded as Gist (5398KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Contributor
Author
🎉 Auto-mergedThis pull request has been automatically merged by hive-mind.
Auto-merged by hive-mind with --auto-merge flag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2119.
What was wrong
Three
hive-mind solve --model formal-airuns on 2026-07-30, one per--tool, produced zero lines of source code between them — and Hive Mind reported all three as successes. The AI side of that is Formal AI's problem and is reported upstream. This pull request is about our side: every signal Hive Mind published about those runs was wrong.019fb330-00e1-…agentProvider: OpenCode Zen,Token usage: 0 input, 0 output, 11 AI sessions under a limit of 5, exit 0019fb330-fa49-…claudeCalculated by Anthropic: $0.252315for a free model, then## ✅ Ready to mergeon an empty diff019fb331-c107-…codex.gitkeep, described as1 file(s) modifiedFor contrast, the same generator in 2025 produced two-file, 45–81 line pull requests with a conventional model. The pipeline works when the model answers; what it cannot do is notice when the model does not.
What this changes
Thirteen defects, each with a regression test. Every fix is applied across all six tools (
claude,agent,opencode,codex,qwen,gemini), not only where the symptom was observed. The Docker images also pin Formal AI 0.317.0, which contains the upstream generalized recovery fix.Attribution and cost —
src/formal-ai-pricing.lib.mjsis now the single source of the provider name and the zero-cost override, consulted at the cost boundary of all six tools plusagent-commanderand the cost renderer.Provider: OpenCode Zen/Calculated by Anthropic→Link.Assistant, because the tool that ran says nothing about who served the tokens$0.252315for a free model →$0.00Public pricing estimate: unknown→$0.00 (Free model)Telemetry —
src/json-stream.lib.mjsframes records by balanced JSON values instead of by newlines, so pretty-printed, concatenated and chunk-split records all parse.Token usage: 0 input, 0 outputfor a session that used 44 511 tokens → real counts, for every tool"tokens": [REDACTED]and JSON braces → counters and structure survive, secrets still do notThe restart loop —
src/auto-restart-budget.lib.mjsreplaces two independent loops that shared one flag.Auto-restart 1/5…5/5and thenAuto-restart triggered (iteration 1)…(iteration 5)— 11 AI sessions under--auto-restart-max-iterations 5→ one budget, oneN/Mlabel?? .formal-ai/counted as user changes, re-created by every restart so the blocker never cleared →src/ai-tool-scratch.lib.mjsexcludes AI scratch state via.git/info/excludeNot calling nothing a success
- 1 file(s) modifiedfor an empty net diff → measured from the diff, not from per-commit sums## ✅ Ready to mergeon a pull request that changes nothing → an empty diff is a blocker, not a green light.gitkeep/CLAUDE.mdplaceholder counted as the AI's work → matched by generated content (a repository's own.gitkeepstill counts) and reported asplaceholderOnlypwdoutput, leaking/tmp/gh-issue-solver-…→ summaries carry facts or say they have none'Implement Hello World in Scala', quotes included → command-stream escapes interpolations itself; the redundant quotes are gone everywhere they appearedTests
Nine new suites, all in the default suite (
npm test, currently 368 files):test-formal-ai-pricing-2119·test-agent-stream-json-2119·test-token-telemetry-sanitization-2119·test-auto-restart-budget-2119·test-empty-pull-request-2119·test-shell-quoting-2119·test-working-session-summary-2119·test-ai-tool-scratch-2119·test-formal-ai-uniform-tools-2119The last one is the anti-regression net for the whole class: it asserts that all five stream parsers import the shared framer, all six tools reach the formal-ai pricing path, and all four Docker image definitions pin Formal AI 0.317.0, so a tool or image variant cannot silently retain a divergent implementation.
Final verification after merging current
main: all 368 default test files passed under Node 24, together with ESLint, Prettier, MJS syntax checks, file-line limits, duplication analysis, changeset validation, andgit diff --check.Case study
docs/case-studies/issue-2119/holds the reconstruction the issue asked for: minute-by-minute timelines of the three runs, an inventory of all sixkonard/test-hello-world-*repositories (2025 baseline vs 2026), the requirement-by-requirement table, root cause and fix for each of the thirteen defects, the two deliberate non-changes, where the data was insufficient and what instrumentation was added, and the libraries considered for the JSON framing (jsonhilo, concatjson, clarinet, json-stream-es, stream-json) with the reason we did not use them. Raw evidence — logs, issue/PR JSON, diffs, gist URLs — is archived underdata/.Upstream resolved and shipped