Skip to content

Commit a17676f

Browse files
Alezander9bcode
andauthored
feat(browser_execute): auto-attach Page.captureScreenshot results as image attachments (#47)
Every successful Page.captureScreenshot made during a browser_execute call is now collected from the CDP transport and surfaced as a FilePart on the tool result. The opencode runner appends those attachments to the next assistant turn as image parts, so the model sees the screenshot natively as vision input. No more decode-write-read dance from inside the snippet. Same channel that read.ts and webfetch.ts already use when they surface images; we're adding browser_execute as a third producer. Mechanism (Level 1, zero upstream diff): - cdp/session.ts: new generic onCallResult(fn) listener API, symmetric with existing onEvent. Fires after every successful _call resolve. Keeps the Session agnostic of any one method's semantics. - browser-execute.ts (Level 1): subscribes for the duration of each execute() call, filters to Page.captureScreenshot, accumulates results into a per-call collector returned alongside output/result. When BCODE_SCREENSHOT_DIR is set, the same tap also writes each screenshot to disk (eval-judge consumption — second consumer of the same hook). - tool/browser-execute.ts (Level 2): maps the collector to attachments[] on the ExecuteResult. BROWSER.md and interaction-skills/screenshots.md updated to tell the agent the auto-attach behavior. Two new smoke tests (gated on BCODE_SMOKE_CHROME) verify screenshots round-trip + the env-var disk dump. Co-authored-by: bcode <bcode@agents.local>
1 parent 4884129 commit a17676f

6 files changed

Lines changed: 179 additions & 6 deletions

File tree

‎packages/bcode-browser/skills/BROWSER.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,11 @@ await session.Input.dispatchMouseEvent({ type: "mouseReleased", x, y, button: "l
133133
await session.Input.insertText({ text: "hello" })
134134

135135
// Screenshot.
136-
const { data } = await session.Page.captureScreenshot({ format: "png" })
137-
// data is base64; write with the `write` tool or process in JS.
136+
await session.Page.captureScreenshot({ format: "png" })
137+
// You see the image inline on the next turn — `browser_execute` automatically
138+
// attaches every `Page.captureScreenshot` result. No need to decode, save, or
139+
// `read` the bytes back. The base64 is still in `data` (via the return value)
140+
// for the rare case you want to process it programmatically.
138141
```
139142

140143
For the full menu of UI mechanics — dropdowns, dialogs, iframes, shadow DOM, uploads, scrolling, screenshots-with-highlights — list `{{SKILLS_DIR}}/interaction-skills/` to see all available topics, then read the relevant one.

‎packages/bcode-browser/skills/interaction-skills/screenshots.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,19 @@
22

33
`session.Page.captureScreenshot` is your default discovery and verification tool.
44

5+
**Auto-attached.** Every successful `Page.captureScreenshot` made during a `browser_execute` call is automatically surfaced to you as an inline image attachment on the next turn — same channel the `read` tool uses for image files. You don't need to decode the base64, save it, or `read` it back to see the image.
6+
7+
The `data` field on the return value still carries the base64 string for the rare case where you want to process the image programmatically (OCR, diff against a previous shot, dimension extraction).
8+
59
## Core calls
610

711
```js
812
// Viewport only (default) — fastest, matches what the user sees
13+
await session.Page.captureScreenshot({ format: 'png' })
14+
// You'll see the image inline on the next turn. No write/read step needed.
15+
16+
// If you do want the bytes (e.g. to write to disk yourself):
917
const { data } = await session.Page.captureScreenshot({ format: 'png' })
10-
// Cross-platform temp dir: /tmp on Linux, /var/folders/… on macOS, %TEMP% on Windows
1118
const { tmpdir } = await import('node:os')
1219
await Bun.write(`${tmpdir()}/shot.png`, Buffer.from(data, 'base64'))
1320

‎packages/bcode-browser/src/browser-execute.ts‎

Lines changed: 67 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
// Level-2 hook in packages/opencode is a thin adapter.
4242

4343
import fs from "fs/promises"
44+
import path from "path"
4445
import { Effect, Schema } from "effect"
4546
import { SessionStore } from "./session-store"
4647
import { Skills } from "./skills"
@@ -78,14 +79,44 @@ export interface ExecuteContext {
7879
readonly onChunk?: (output: string) => Effect.Effect<void>
7980
}
8081

82+
// One screenshot collected during an execute() call. Drained into the
83+
// Level-2 wrapper's `attachments[]` so the agent sees the image inline on the
84+
// next assistant turn — no decode/write/read dance from inside the snippet.
85+
export interface CollectedScreenshot {
86+
readonly mime: "image/png" | "image/jpeg" | "image/webp"
87+
readonly base64: string
88+
}
89+
8190
export interface ExecuteResult {
8291
readonly output: string
8392
// The snippet's `return` value, JSON-serialized when possible. `undefined`
8493
// serializes as `null` (JSON has no undefined). Non-serializable values
8594
// fall back to `String(v)`.
8695
readonly result: string
96+
// Every successful `Page.captureScreenshot` made by the snippet, in the
97+
// order the CDP responses came back. Empty when the snippet didn't take
98+
// any screenshots.
99+
readonly screenshots: readonly CollectedScreenshot[]
100+
}
101+
102+
const SCREENSHOT_FORMAT_TO_MIME: Record<string, CollectedScreenshot["mime"]> = {
103+
png: "image/png",
104+
jpeg: "image/jpeg",
105+
webp: "image/webp",
106+
}
107+
108+
const SCREENSHOT_FORMAT_TO_EXT: Record<string, string> = {
109+
png: "png",
110+
jpeg: "jpg",
111+
webp: "webp",
87112
}
88113

114+
const screenshotMime = (format: unknown): CollectedScreenshot["mime"] =>
115+
SCREENSHOT_FORMAT_TO_MIME[typeof format === "string" ? format : "png"] ?? "image/png"
116+
117+
const screenshotExt = (format: unknown): string =>
118+
SCREENSHOT_FORMAT_TO_EXT[typeof format === "string" ? format : "png"] ?? "png"
119+
89120
// AsyncFunction is not a global — pull it off an async arrow's constructor.
90121
const AsyncFunction = (async () => {}).constructor as new (
91122
...args: string[]
@@ -145,12 +176,46 @@ export const make = Effect.fn("BrowserExecute.make")(function* (dataDir: string)
145176
debug: tee,
146177
})
147178

179+
// Screenshot tap. Subscribes to the Session's call-result stream for
180+
// the duration of this execute() call; every successful
181+
// `Page.captureScreenshot` is collected (drained into `attachments[]`
182+
// by the Level-2 wrapper so the agent sees the image inline) and,
183+
// when `BCODE_SCREENSHOT_DIR` is set, also written to disk for
184+
// eval-judge consumption. Two consumers of one tap.
185+
//
186+
// Concurrency note: parallel execute() calls against the same Session
187+
// (rare but possible — different sessionIDs share no Session, but a
188+
// single sessionID with two in-flight tool calls would) each subscribe
189+
// independently and would each see all screenshots produced during
190+
// their lifetime. Acceptable for v1; opencode tool calls within one
191+
// assistant message are serialized anyway.
192+
const screenshots: CollectedScreenshot[] = []
193+
const dumpDir = process.env.BCODE_SCREENSHOT_DIR
194+
const startedAt = Date.now()
195+
let seq = 0
196+
const unsubscribe = session.onCallResult((method, params, result) => {
197+
if (method !== "Page.captureScreenshot") return
198+
const r = result as { data?: unknown }
199+
if (typeof r?.data !== "string") return
200+
const p = (params ?? {}) as { format?: unknown }
201+
const mime = screenshotMime(p.format)
202+
const ext = screenshotExt(p.format)
203+
const idx = seq++
204+
screenshots.push({ mime, base64: r.data })
205+
if (dumpDir) {
206+
const filename = `${ctx.sessionID}-${startedAt}-${String(idx).padStart(3, "0")}.${ext}`
207+
fs.mkdir(dumpDir, { recursive: true })
208+
.then(() => fs.writeFile(path.join(dumpDir, filename), Buffer.from(r.data as string, "base64")))
209+
.catch(() => { /* eval-side dump is best-effort */ })
210+
}
211+
})
212+
148213
const ran = yield* Effect.tryPromise({
149214
try: () => wrapped(session, snippetConsole),
150215
catch: (err) => new Error(`browser_execute snippet threw: ${err instanceof Error ? err.stack ?? err.message : String(err)}`),
151-
})
216+
}).pipe(Effect.ensuring(Effect.sync(() => unsubscribe())))
152217

153-
return { output, result: serialize(ran) } satisfies ExecuteResult
218+
return { output, result: serialize(ran), screenshots } satisfies ExecuteResult
154219
}).pipe(
155220
Effect.scoped,
156221
Effect.timeoutOrElse({

‎packages/bcode-browser/src/cdp/session.ts‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ export class Session implements Transport {
4747
private pending = new Map<number, Pending>();
4848
private activeSessionId: string | undefined;
4949
private eventListeners: Array<(method: string, params: unknown, sessionId?: string) => void> = [];
50+
private callResultListeners: Array<(method: string, params: unknown, result: unknown) => void> = [];
5051

5152
// Generated bindings — one per CDP domain.
5253
// Initialized lazily after construction so `_call` is available.
@@ -170,6 +171,23 @@ export class Session implements Transport {
170171
};
171172
}
172173

174+
/**
175+
* Subscribe to all successful CDP method results. Returns an unsubscribe fn.
176+
* Fires after `_call` resolves; listener errors are swallowed.
177+
*
178+
* Used by `browser-execute` to collect `Page.captureScreenshot` outputs
179+
* from inside an execute() call (drained into `attachments[]` so the agent
180+
* sees the image inline; optionally also written to `BCODE_SCREENSHOT_DIR`
181+
* for eval-judge consumption). Generic by design — keeps `Session`
182+
* agnostic of any one method's semantics.
183+
*/
184+
onCallResult(fn: (method: string, params: unknown, result: unknown) => void): () => void {
185+
this.callResultListeners.push(fn);
186+
return () => {
187+
this.callResultListeners = this.callResultListeners.filter(x => x !== fn);
188+
};
189+
}
190+
173191
/** Wait for the next event matching `method` (and optional predicate). */
174192
waitFor<T = unknown>(method: string, predicate?: (params: T) => boolean, timeoutMs = 30_000): Promise<T> {
175193
return new Promise((resolve, reject) => {
@@ -198,7 +216,15 @@ export class Session implements Transport {
198216
msg.sessionId = this.activeSessionId;
199217
}
200218
return new Promise((resolve, reject) => {
201-
this.pending.set(id, { resolve, reject });
219+
this.pending.set(id, {
220+
resolve: (v) => {
221+
for (const fn of this.callResultListeners) {
222+
try { fn(method, params, v); } catch { /* ignore */ }
223+
}
224+
resolve(v);
225+
},
226+
reject,
227+
});
202228
this.ws!.send(JSON.stringify(msg));
203229
});
204230
}

‎packages/bcode-browser/test/browser-execute.test.ts‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,63 @@ test.skipIf(!enabled)("workspace import inside a snippet", async () => {
113113
expect(JSON.parse(result.result)).toBe("bcode-be")
114114
})
115115

116+
test.skipIf(!enabled)("Page.captureScreenshot is collected into result.screenshots", async () => {
117+
const result = await Effect.runPromise(
118+
Effect.scoped(
119+
Effect.gen(function* () {
120+
const impl = yield* BrowserExecute.make(dataDir)
121+
return yield* impl.execute(
122+
{
123+
code: `await session.Page.enable();
124+
await session.Page.navigate({ url: "data:text/html,<title>shot</title><body>hi" });
125+
await session.waitFor("Page.loadEventFired", undefined, 5000);
126+
const a = await session.Page.captureScreenshot({ format: "png" });
127+
const b = await session.Page.captureScreenshot({ format: "jpeg", quality: 50 });
128+
return { aLen: a.data.length, bLen: b.data.length };`,
129+
},
130+
{ sessionID, workspaceDir },
131+
)
132+
}),
133+
),
134+
)
135+
expect(result.screenshots).toHaveLength(2)
136+
expect(result.screenshots[0]!.mime).toBe("image/png")
137+
expect(result.screenshots[1]!.mime).toBe("image/jpeg")
138+
// base64 must round-trip back to non-empty bytes for both shots.
139+
expect(Buffer.from(result.screenshots[0]!.base64, "base64").length).toBeGreaterThan(0)
140+
expect(Buffer.from(result.screenshots[1]!.base64, "base64").length).toBeGreaterThan(0)
141+
})
142+
143+
test.skipIf(!enabled)("BCODE_SCREENSHOT_DIR dumps screenshots to disk", async () => {
144+
const dump = await fs.mkdtemp(path.join(os.tmpdir(), "bcode-shotdump-"))
145+
const prev = process.env.BCODE_SCREENSHOT_DIR
146+
process.env.BCODE_SCREENSHOT_DIR = dump
147+
try {
148+
await Effect.runPromise(
149+
Effect.scoped(
150+
Effect.gen(function* () {
151+
const impl = yield* BrowserExecute.make(dataDir)
152+
return yield* impl.execute(
153+
{
154+
code: `await session.Page.captureScreenshot({ format: "png" });`,
155+
},
156+
{ sessionID, workspaceDir },
157+
)
158+
}),
159+
),
160+
)
161+
// Disk dump is fire-and-forget; give it a tick to land.
162+
await new Promise((r) => setTimeout(r, 150))
163+
const files = await fs.readdir(dump)
164+
expect(files.length).toBeGreaterThan(0)
165+
expect(files.every((f) => f.endsWith(".png"))).toBe(true)
166+
} finally {
167+
if (prev === undefined) delete process.env.BCODE_SCREENSHOT_DIR
168+
else process.env.BCODE_SCREENSHOT_DIR = prev
169+
await fs.rm(dump, { recursive: true, force: true })
170+
}
171+
})
172+
116173
test.skipIf(!enabled)("syntax error in snippet surfaces a clean failure", async () => {
117174
await expect(
118175
Effect.runPromise(

‎packages/opencode/src/tool/browser-execute.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,15 +55,30 @@ export const BrowserExecuteTool = Tool.define(
5555
metadata: { output: preview(output) },
5656
}),
5757
})
58+
// Drain every `Page.captureScreenshot` made during this snippet
59+
// into `attachments[]`. Opencode appends FilePart attachments to
60+
// the next assistant turn as image parts, so the model receives
61+
// the screenshot natively as vision input — no decode/write/read
62+
// dance from inside the snippet. Same channel `read` and
63+
// `webfetch` use when they surface images.
64+
const attachments = result.screenshots.map((s) => ({
65+
type: "file" as const,
66+
mime: s.mime,
67+
url: `data:${s.mime};base64,${s.base64}`,
68+
}))
5869
return {
5970
title: "browser_execute",
6071
output: [
6172
result.output.trimEnd(),
6273
result.result === "null" ? "" : `=> ${result.result}`,
74+
attachments.length > 0
75+
? `(${attachments.length} screenshot${attachments.length === 1 ? "" : "s"} attached)`
76+
: "",
6377
]
6478
.filter(Boolean)
6579
.join("\n\n"),
6680
metadata: { result: result.result, output: preview(result.output) },
81+
attachments,
6782
}
6883
}).pipe(Effect.orDie),
6984
}

0 commit comments

Comments
 (0)