Repository navigation
Publish the 500-task shopping selection and historical agent scores - #68
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
1 issue found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="consumer_bench/grok-historical-50.json">
<violation number="1" location="consumer_bench/grok-historical-50.json:10">
P3: `outcomeCounts` and `successRate` are hardcoded copies of data derived from `outcomes`. Add assertions in the scoring test that each `outcomeCounts` category matches the `outcomes` tally, so later edits cannot silently desynchronize them.</violation>
</file>
Requires human review: Auto-approval blocked because this review re-detected 3 unresolved issues already reported by Cubic.
View guided diff | Turn on auto-fix | Re-trigger cubic
| "runDate": "Oct 12–13, 2026", | ||
| "scope": "Search or category, then open a product or result. No cart requirement.", | ||
| "note": "Different tasks and CAPTCHA rules. Amazon used an earlier smoke prompt. This is not a matched 500-task score.", | ||
| "outcomeCounts": { |
There was a problem hiding this comment.
P3: outcomeCounts and successRate are hardcoded copies of data derived from outcomes. Add assertions in the scoring test that each outcomeCounts category matches the outcomes tally, so later edits cannot silently desynchronize them.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At consumer_bench/grok-historical-50.json, line 10:
<comment>`outcomeCounts` and `successRate` are hardcoded copies of data derived from `outcomes`. Add assertions in the scoring test that each `outcomeCounts` category matches the `outcomes` tally, so later edits cannot silently desynchronize them.</comment>
<file context>
@@ -0,0 +1,320 @@
+ "runDate": "Oct 12–13, 2026",
+ "scope": "Search or category, then open a product or result. No cart requirement.",
+ "note": "Different tasks and CAPTCHA rules. Amazon used an earlier smoke prompt. This is not a matched 500-task score.",
+ "outcomeCounts": {
+ "success": 36,
+ "uncleared challenge": 7,
</file context>
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
View guided diff | Turn on auto-fix | Re-trigger cubic
The current shopping comparison needs one versioned 500-task selection. V10 removes Ozon and Coupang equally, recalculates all scores, and preserves V9 plus every original outcome.
Browser Use Cloud shows 459/500, Dot shows 363/500, and Muse shows 306/500. The owner approves Dot as the display name for the historical OpenAI runner. Its original runner ID remains intact. Instinct remains partial. A later native message reports 292 provisional successes from 382 finished attempts on its original 513-task set. Updated task rows remain missing. Grok retains its separate 36/50 browsing study.
Per-task CAPTCHA report fields support the published counts. Original Muse failure categories distinguish website security blocks from browser errors. Later exclusions and the selected best Cloud run limit generalization. These are historical projections, not fresh V10 runs or infrastructure-only block rates.
Validation: six scoring tests pass. All 500 task IDs and domains are unique. Original outcome values remain unchanged. Public website downloads match the repository files.
The latest native Instinct report gives 292/513 provisional successes and a daily browser-time pause. It includes known partial results. The earlier archive snapshot remains separate.