Every observation glass offers is a point in time or a comparison of two points in time. Nothing
reports what happened between them.
glass_screenshot / glass_a11y_snapshot — the state at one instant.
glass_diff — one frame against a saved baseline.
glass_wait_stable — detects that change has stopped, not what the change was.
glass_wait_for_region / glass_wait_for_element — a condition became true, not the path taken.
So a class of behaviour is unobservable through the tool surface today:
- an animation or transition — whether it ran, how it ended, whether it ended in the right place;
- drag-follow — whether the dragged thing tracked the pointer, or jumped, or was dropped early
(glass_drag reports the endpoints it was given, not what the UI did in between);
- a transient state that appears and disappears inside a single wait — a spinner, a toast, a
validation message — which wait_stable will deliberately wait past.
For an agent building UI, several of these are the thing it most needs to check, and the current
answer is that it cannot.
The honest counter-argument
This may be a deliberate boundary rather than a gap. glass's shape is cheap text-first observation
— changed_pct and a bbox instead of a vision round-trip — and a video is the opposite of that: it
cannot be read without a vision model, which is the cost the whole design exists to avoid. A
recording an agent cannot cheaply interpret may be worth little.
Two cheaper shapes might fit the existing grain better than recording:
- Sampled diff over a window — sample the frame at an interval for a bounded duration and return
the changed_pct/bbox series as text. Answers "did it animate, for how long, and where" with no
image tokens, and reuses the glass_diff machinery.
- Peak/settle summary — over a wait, report the maximum change observed and where, so a
transient that came and went leaves a trace instead of vanishing.
What this issue is asking for
A decision, and if the answer is yes, the shape. Filing it so the capability boundary is written
down rather than implicit: today an agent cannot verify an animation, a drag-follow, or a transient
state, and nothing in the tool descriptions says so.
Every observation glass offers is a point in time or a comparison of two points in time. Nothing
reports what happened between them.
glass_screenshot/glass_a11y_snapshot— the state at one instant.glass_diff— one frame against a saved baseline.glass_wait_stable— detects that change has stopped, not what the change was.glass_wait_for_region/glass_wait_for_element— a condition became true, not the path taken.So a class of behaviour is unobservable through the tool surface today:
(
glass_dragreports the endpoints it was given, not what the UI did in between);validation message — which
wait_stablewill deliberately wait past.For an agent building UI, several of these are the thing it most needs to check, and the current
answer is that it cannot.
The honest counter-argument
This may be a deliberate boundary rather than a gap. glass's shape is cheap text-first observation
—
changed_pctand a bbox instead of a vision round-trip — and a video is the opposite of that: itcannot be read without a vision model, which is the cost the whole design exists to avoid. A
recording an agent cannot cheaply interpret may be worth little.
Two cheaper shapes might fit the existing grain better than recording:
the
changed_pct/bbox series as text. Answers "did it animate, for how long, and where" with noimage tokens, and reuses the
glass_diffmachinery.transient that came and went leaves a trace instead of vanishing.
What this issue is asking for
A decision, and if the answer is yes, the shape. Filing it so the capability boundary is written
down rather than implicit: today an agent cannot verify an animation, a drag-follow, or a transient
state, and nothing in the tool descriptions says so.