Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ internal refactors, CI, or test-only changes.
- A web view whose content the platform has not published is disclosed in the snapshot instead of arriving as an indistinguishable empty group: a childless `Document` is named by id and bounds and steers to a re-snapshot before pixels, while a placeholder the app published for content it withheld steers straight to pixels.
- On Linux, `glass_set_value` now confirms a write actually landed instead of trusting the AT-SPI toolkit's own acknowledgment of it, the way the Windows and macOS readers already did: it reads the element back (polling briefly, since the toolkit applies the write on a later main-loop pass), and an element that acknowledges a write without applying it — as web content can — now reports that instead of a silent false success.
- On Linux, the container of an `<iframe>` inside a browser page no longer reads as a `Window`: AT-SPI's `internal frame`, which the web engines publish for an embedded frame and no desktop toolkit uses, now maps to `Group`, so a `role:"Window"` selector matches the app's real windows again instead of frames inside a page. The nested `Document` inside such a frame is unchanged.
- On Android, a WebView's page no longer reads as two nested `Document`s: the host view and the page root the engine publishes inside it both report `android.webkit.WebView`, so a host that holds the page root now reads as the `Group` around it and `role:"Document"` matches once per web view instead of twice. A WebView that has published no page root yet still reads as a `Document`, and is still disclosed as one with no readable content.

## [1.5.0] - 2026-08-22

Expand Down
29 changes: 28 additions & 1 deletion crates/glass-android/src/a11y_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use glass_core::accessibility::{
use glass_core::platform::WindowGeometry;
use glass_core::{GlassError, Result, read_back_failed, write_took_no_effect};

use crate::axmap::{LabelInputs, class_to_role, labels};
use crate::axmap::{LabelInputs, class_to_role, demote_web_hosts, labels};
use crate::conn::{CallFailure, Conn};

/// One walk of a device `tree` reply: the bounds it runs under, and the device `ref` of every
Expand Down Expand Up @@ -179,6 +179,7 @@ pub(crate) fn tree_from_json(
) -> Result<RefTree> {
let mut walk = Walk::new(limits);
let mut root = json_to_node(tree, win, 0, &mut walk)?;
demote_web_hosts(&mut root);
// The device answers with the root of the ACTIVE WINDOW, so this node is the window
// whatever layout class it carries. Both Android readers have to agree about the root, or
// a `role:` selector written against one misses on the other.
Expand Down Expand Up @@ -1647,6 +1648,32 @@ mod tests {
assert_eq!(tree.root.children[1].role, AxRole::Button);
}

#[test]
fn the_host_view_of_a_published_page_is_a_group_here_too() {
// Both readers have to agree about a web page's shape, or one `role:` selector cannot
// address it on both (glass#521).
let page = json!({
"class": "android.widget.FrameLayout",
"bounds": {"x": 0, "y": 0, "w": 1080, "h": 2400},
"children": [{
"class": "android.webkit.WebView", "desc": "the web view",
"bounds": {"x": 0, "y": 0, "w": 1080, "h": 2148},
"children": [{
"class": "android.webkit.WebView",
"bounds": {"x": 0, "y": 0, "w": 1080, "h": 2148},
"children": [{
"class": "android.view.View", "text": "Role fixture",
"bounds": {"x": 0, "y": 0, "w": 1080, "h": 120}, "children": []
}]
}]
}]
});
let tree = read_json(&page, WalkLimits::DEFAULT).expect("builds").tree;
let host = &tree.root.children[0];
assert_eq!(host.role, AxRole::Group, "the host view");
assert_eq!(host.children[0].role, AxRole::Document, "the page root");
}

#[test]
fn reads_checkable_and_checked_from_json() {
// The companion now carries isCheckable/isChecked; surface them on the node's states.
Expand Down
74 changes: 73 additions & 1 deletion crates/glass-android/src/axmap.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,32 @@ pub fn class_to_role(class: &str) -> AxRole {
AxRole::Other
}

/// Re-role a web view's host to [`AxRole::Group`] where the engine has published the page root
/// inside it, leaving one `Document` per web view instead of two nested ones (glass#521).
///
/// A web engine gives the page root the host's own class, so both nodes report
/// `android.webkit.WebView`. Both readers call this, or a `role:` selector written against one
/// misses on the other. No node is dropped and no id moves.
///
/// Not bounds: on the reading this came from the host was 2062px tall and the page root inside it
/// 2064, so a same-rect test would never have fired.
///
/// A host that has published no page root stays a `Document` — the node
/// [`glass_core::accessibility::AxTree::document_guidance`] names to say the page is not in the
/// tree yet.
pub(crate) fn demote_web_hosts(node: &mut AxNode) {
if node.role == AxRole::Document
&& let [child] = node.children.as_slice()
&& child.role == AxRole::Document
&& child.raw_role == node.raw_role
{
node.role = AxRole::Group;
}
for child in &mut node.children {
demote_web_hosts(child);
}
}

/// `com.example:id/search_src_text` → `search_src_text`. The package-qualified form is noise in an
/// outline; the bare leaf is still not unique within an app's tree — see [`labels`]'s doc for why
/// it is only a label of last resort. The leaf can be blank where the whole id was not
Expand Down Expand Up @@ -120,7 +146,7 @@ pub fn build_tree(xml: &str, window: &WindowGeometry, limits: WalkLimits) -> Res
}
children.push(map_node(n, window, 0, &mut budget));
}
let root = AxNode {
let mut root = AxNode {
id: AxNodeId(0),
role: AxRole::Window,
raw_role: "hierarchy".into(),
Expand All @@ -136,6 +162,7 @@ pub fn build_tree(xml: &str, window: &WindowGeometry, limits: WalkLimits) -> Res
}),
children,
};
demote_web_hosts(&mut root);
let mut tree = AxTree::new(root);
tree.truncated = budget.truncation();
Ok(tree)
Expand Down Expand Up @@ -636,6 +663,51 @@ mod tests {
assert_eq!(class_to_role("android.webkit.WebView"), AxRole::Document);
}

/// A published web page as `uiautomator` dumps it: the host view, the page root inside it
/// reporting the same class, and one page node.
const PUBLISHED_WEB_PAGE_XML: &str = concat!(
"<?xml version='1.0'?><hierarchy rotation=\"0\">",
"<node index=\"0\" text=\"\" class=\"android.webkit.WebView\" package=\"com.x\" ",
"content-desc=\"the web view\" enabled=\"true\" focusable=\"true\" focused=\"true\" ",
"bounds=\"[0,0][1080,2148]\">",
"<node index=\"0\" text=\"\" class=\"android.webkit.WebView\" package=\"com.x\" ",
"content-desc=\"\" enabled=\"true\" focusable=\"false\" bounds=\"[0,0][1080,2148]\">",
"<node index=\"0\" text=\"Role fixture\" class=\"android.view.View\" package=\"com.x\" ",
"enabled=\"true\" focusable=\"false\" bounds=\"[0,0][1080,120]\" />",
"</node></node></hierarchy>",
);

#[test]
fn the_host_view_of_a_published_page_is_a_group_not_a_second_document() {
// glass#521, read on an API 34 emulator: the host view and the page root the engine
// publishes inside it both report `android.webkit.WebView`, so every web view arrived as
// two nested `Document`s and `role:"Document"` matched twice.
let tree = build_tree(PUBLISHED_WEB_PAGE_XML, &win(), WalkLimits::DEFAULT).unwrap();
let host = &tree.root.children[0];
assert_eq!(host.role, AxRole::Group, "the host view");
assert_eq!(host.children[0].role, AxRole::Document, "the page root");
}

#[test]
fn a_web_view_that_has_published_no_page_root_stays_a_document() {
// Demoting a host before its page root arrives would silence the childless-`Document`
// notice — on this fixture the first read after a launch is that shape.
let xml = concat!(
"<?xml version='1.0'?><hierarchy rotation=\"0\">",
"<node index=\"0\" text=\"\" class=\"android.webkit.WebView\" package=\"com.x\" ",
"content-desc=\"the web view\" enabled=\"true\" focusable=\"true\" ",
"bounds=\"[0,0][1080,2148]\" />",
"</hierarchy>",
);
let mut tree = build_tree(xml, &win(), WalkLimits::DEFAULT).unwrap();
tree.assign_ids();
assert_eq!(tree.root.children[0].role, AxRole::Document);
assert!(
tree.document_guidance().is_some(),
"the notice must still name it"
);
}

#[test]
fn class_tokens_have_no_duplicates() {
for (i, (leaf, _)) in CLASS_TOKENS.iter().enumerate() {
Expand Down
2 changes: 1 addition & 1 deletion docs/explanation/web-content.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ engine — a later version, or the same engine under a different embedder, can d
| Linux (AT-SPI) | Brave 151 (Chromium) | Publishes a null placeholder child while renderer accessibility is off — read as withheld content, not an empty page. |
| Windows (UIA) | Edge 151, Brave 151, Firefox 154 | Publish at baseline (0.4–3.3s). A web page's `Document` and a text editor's edit surface report the same UIA control type; they're told apart per element by the `FrameworkId` each reports (`Chrome`/`Gecko` vs. `Win32`). Clicks and `set_value` land. |
| macOS (AX) | Safari 26.5 (WebKit) | Publishes `AXWebArea` in the very first snapshot — reading the tree is itself what materializes the lazily built web area. `AXPress` clicks land; `set_value` on a web input is refused honestly (`AxValueNotApplied`) — type into it instead. |
| Android | System WebView (version not read), on an API 34 emulator | Read through both readers — `uiautomator` and the on-device companion (v0.6.0). A WebView's page arrives as `Document` twice — the host view and the page root. The first snapshot right after launch can show a childless `Document`; the next one holds the page. |
| Android | System WebView (version not read), on an API 34 emulator | Read through both readers — `uiautomator` and the on-device companion (v0.6.0). The host view and the page root the engine publishes inside it both report `android.webkit.WebView`, so a host holding a page root is read as the `Group` around the page's `Document` rather than as a second one. The first snapshot right after launch can show a childless `Document`; the next one holds the page. |
| iOS (idb) | WKWebView | No element at all — not the view, not the page, not an empty web area. See below. |

## What stays open: iOS
Expand Down