Skip to content

android a11y: a WebView maps to an opaque Group, so an agent cannot tell an empty group from unexposed web content #506

Description

@xxx

crates/glass-android/src/axmap.rs:46 maps ("WebView", AxRole::Group). A WebView therefore
arrives in the tree as an ordinary group node, and the web document inside it — which on a hybrid
app is most or all of the UI — is not published as elements.

That much is a capability limit. The defect is that nothing tells the agent it happened.

It falls between our two disclosure mechanisms

glass has two ways to say "what you're looking at is not the whole picture", and a WebView screen
triggers neither:

  • AxTree::empty_guidance (crates/glass-core/src/accessibility.rs:747) fires only on
    self.root.children.is_empty() — a root-only tree. A WebView screen has a Group child, so this
    returns None.
  • Truncation::notice (accessibility.rs:449) fires when the walk hits MAX_NODES / MAX_DEPTH /
    MAX_SIBLINGS. Nothing was truncated here — the walk completed and the group genuinely has no
    children to report.

So the agent receives a completed, untruncated tree containing an empty Group, and cannot
distinguish that from a group that is genuinely empty. Both disclosure paths end by naming the
pixel fallback (glass_screenshot, then glass_click at x,y); this case needs the same steer and
does not get it.

Impact

An agent driving a hybrid app sees a screen it reads as nearly empty, has no reason to fall back to
pixels, and concludes the UI failed to render — when the content is there and simply isn't
published through uiautomator.

Status

Theoretical. Read from the mapping table and the two disclosure paths; not reproduced against a
hybrid app on a device. What would confirm it: snapshot an app whose content is a WebView and check
whether any part of the result indicates the group is an unexpanded web document.

Fix direction

Two separable steps, smallest first:

  1. Disclose. When a mapped node is a WebView (or any container known to host content the reader
    cannot enter), say so in the rendered outline and name the pixel path — the shape
    empty_guidance and Truncation::notice already use. This is the part that converts a silent
    blind spot into an actionable one.
  2. Expose, separately and later: whether the web document's own nodes can be read at all on this
    backend is an open question and much larger than (1).

Other backends may not share this: crates/glass-a11y-linux/src/events.rs:261 notes that a walk
there crosses process boundaries and an embedded view publishes its own subtree. Windows, macOS and
iOS have not been checked.

Metadata

Metadata

Assignees

No one assigned

    Labels

    theoreticalA bug theorized as possible by reading the code, but never reported or seen in actual use.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions