crates/glass-android/src/axmap.rs:46 maps ("WebView", AxRole::Group). A WebView therefore
arrives in the tree as an ordinary group node, and the web document inside it — which on a hybrid
app is most or all of the UI — is not published as elements.
That much is a capability limit. The defect is that nothing tells the agent it happened.
It falls between our two disclosure mechanisms
glass has two ways to say "what you're looking at is not the whole picture", and a WebView screen
triggers neither:
AxTree::empty_guidance (crates/glass-core/src/accessibility.rs:747) fires only on
self.root.children.is_empty() — a root-only tree. A WebView screen has a Group child, so this
returns None.
Truncation::notice (accessibility.rs:449) fires when the walk hits MAX_NODES / MAX_DEPTH /
MAX_SIBLINGS. Nothing was truncated here — the walk completed and the group genuinely has no
children to report.
So the agent receives a completed, untruncated tree containing an empty Group, and cannot
distinguish that from a group that is genuinely empty. Both disclosure paths end by naming the
pixel fallback (glass_screenshot, then glass_click at x,y); this case needs the same steer and
does not get it.
Impact
An agent driving a hybrid app sees a screen it reads as nearly empty, has no reason to fall back to
pixels, and concludes the UI failed to render — when the content is there and simply isn't
published through uiautomator.
Status
Theoretical. Read from the mapping table and the two disclosure paths; not reproduced against a
hybrid app on a device. What would confirm it: snapshot an app whose content is a WebView and check
whether any part of the result indicates the group is an unexpanded web document.
Fix direction
Two separable steps, smallest first:
- Disclose. When a mapped node is a WebView (or any container known to host content the reader
cannot enter), say so in the rendered outline and name the pixel path — the shape
empty_guidance and Truncation::notice already use. This is the part that converts a silent
blind spot into an actionable one.
- Expose, separately and later: whether the web document's own nodes can be read at all on this
backend is an open question and much larger than (1).
Other backends may not share this: crates/glass-a11y-linux/src/events.rs:261 notes that a walk
there crosses process boundaries and an embedded view publishes its own subtree. Windows, macOS and
iOS have not been checked.
crates/glass-android/src/axmap.rs:46maps("WebView", AxRole::Group). A WebView thereforearrives in the tree as an ordinary group node, and the web document inside it — which on a hybrid
app is most or all of the UI — is not published as elements.
That much is a capability limit. The defect is that nothing tells the agent it happened.
It falls between our two disclosure mechanisms
glass has two ways to say "what you're looking at is not the whole picture", and a WebView screen
triggers neither:
AxTree::empty_guidance(crates/glass-core/src/accessibility.rs:747) fires only onself.root.children.is_empty()— a root-only tree. A WebView screen has aGroupchild, so thisreturns
None.Truncation::notice(accessibility.rs:449) fires when the walk hitsMAX_NODES/MAX_DEPTH/MAX_SIBLINGS. Nothing was truncated here — the walk completed and the group genuinely has nochildren to report.
So the agent receives a completed, untruncated tree containing an empty
Group, and cannotdistinguish that from a group that is genuinely empty. Both disclosure paths end by naming the
pixel fallback (
glass_screenshot, thenglass_click at x,y); this case needs the same steer anddoes not get it.
Impact
An agent driving a hybrid app sees a screen it reads as nearly empty, has no reason to fall back to
pixels, and concludes the UI failed to render — when the content is there and simply isn't
published through
uiautomator.Status
Theoretical. Read from the mapping table and the two disclosure paths; not reproduced against a
hybrid app on a device. What would confirm it: snapshot an app whose content is a WebView and check
whether any part of the result indicates the group is an unexpanded web document.
Fix direction
Two separable steps, smallest first:
cannot enter), say so in the rendered outline and name the pixel path — the shape
empty_guidanceandTruncation::noticealready use. This is the part that converts a silentblind spot into an actionable one.
backend is an open question and much larger than (1).
Other backends may not share this:
crates/glass-a11y-linux/src/events.rs:261notes that a walkthere crosses process boundaries and an embedded view publishes its own subtree. Windows, macOS and
iOS have not been checked.