Skip to content

no tool observes what happens between observations — animations, drag-follow and transient states are unverifiable #508

Description

@xxx

Every observation glass offers is a point in time or a comparison of two points in time. Nothing
reports what happened between them.

  • glass_screenshot / glass_a11y_snapshot — the state at one instant.
  • glass_diff — one frame against a saved baseline.
  • glass_wait_stable — detects that change has stopped, not what the change was.
  • glass_wait_for_region / glass_wait_for_element — a condition became true, not the path taken.

So a class of behaviour is unobservable through the tool surface today:

  • an animation or transition — whether it ran, how it ended, whether it ended in the right place;
  • drag-follow — whether the dragged thing tracked the pointer, or jumped, or was dropped early
    (glass_drag reports the endpoints it was given, not what the UI did in between);
  • a transient state that appears and disappears inside a single wait — a spinner, a toast, a
    validation message — which wait_stable will deliberately wait past.

For an agent building UI, several of these are the thing it most needs to check, and the current
answer is that it cannot.

The honest counter-argument

This may be a deliberate boundary rather than a gap. glass's shape is cheap text-first observation
changed_pct and a bbox instead of a vision round-trip — and a video is the opposite of that: it
cannot be read without a vision model, which is the cost the whole design exists to avoid. A
recording an agent cannot cheaply interpret may be worth little.

Two cheaper shapes might fit the existing grain better than recording:

  • Sampled diff over a window — sample the frame at an interval for a bounded duration and return
    the changed_pct/bbox series as text. Answers "did it animate, for how long, and where" with no
    image tokens, and reuses the glass_diff machinery.
  • Peak/settle summary — over a wait, report the maximum change observed and where, so a
    transient that came and went leaves a trace instead of vanishing.

What this issue is asking for

A decision, and if the answer is yes, the shape. Filing it so the capability boundary is written
down rather than implicit: today an agent cannot verify an animation, a drag-follow, or a transient
state, and nothing in the tool descriptions says so.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions