Skip to content

ares-do: send key events and take screenshots - #15

Open
mariotaku wants to merge 7 commits into
mainfrom
feat/ares-do
Open

ares-do: send key events and take screenshots#15
mariotaku wants to merge 7 commits into
mainfrom
feat/ares-do

Conversation

@mariotaku

@mariotaku mariotaku commented Aug 23, 2026

Copy link
Copy Markdown
Member

Adds ares-do, a tenth tool: it presses buttons and captures the screen, so a UI flow can be scripted and replayed instead of walked by hand. The work that motivated it (webosbrew/samples#1) meant running luna-send once per keypress and diffing PNGs by hand.

ares-do -d tv key TAB TAB ENTER      # a web view starts with nothing focused
ares-do -d tv text demo
ares-do -d tv screenshot > shot.png
ares-do -d tv luna luna://com.webos.applicationManager/dev/running | jq
ares-do -d tv --json run - --out ./frames <<'EOF'
launch com.webos.app.livetv
wait 2s
shot 01-livetv
EOF

Everything is host-side over SSH. Nothing is installed on the device and nothing is written to its filesystem beyond one staging file per screenshot, which is deleted again.

Two undocumented services

test/sendKeyCode on com.webos.service.networkinput, and executeOneShot on com.webos.service.capture falling back to com.webos.service.tv.capture — a 49LK5900 only has the latter, so the fallback is not theoretical. Both are on the private bus, so a uid probe runs before anything is sent rather than after three keys have gone nowhere.

sendKeyCode reaches UInputWriter::sendKeyPress, which writes to /dev/uinput. That settles two things: the codes are ordinary evdev and stop at KEY_MAX, and there is no held modifier, so capitals cannot be typed at all. text refuses them up front naming every character, rather than a --shift flag that would quietly do nothing.

Which code each button sends

LG's choice, and often not the mainline name. /usr/share/X11/xkb/keycodes/lg in the firmware is the authority, and the surprises are aliases with notes in ares-do keys. The remote's Back is 412, not KEY_BACK (158) — confirmed on hardware, where 158 moved 970 pixels and 412 dismissed the launcher and moved 1.4M. Likewise the remote's Home is HOMEPAGE (172), not KEY_HOME.

The flow language is the CLI

A line is tokenized and handed to the same clap Command the shell uses, so the two cannot drift and the flags are identical in both. The whole file parses before anything runs, and every error is reported at once with line numbers.

Ergonomics

Follows adb and xdotool rather than inventing conventions: stdout is data and stderr is for people, so screenshot > shot.png works; --json emits NDJSON ending with an error object whose code is the exit code; exit codes are a documented table; and run - replays a flow from stdin over one SSH handshake instead of one per key. keys and --dry-run need no device at all.

Also fixes the shared Luna client

Luna::call read luna-send's stdout and stderr and then threw both away, collapsing "no such service", "not allowed on this bus" and "the channel died" into one value — and it read to EOF with no bound, so a method that subscribes instead of replying hung forever.

ares-connection-lib gains a public Exec::exec_timeout that drains both streams against a deadline, and Luna gains call_with/subscribe_with taking a LunaOptions carrying the bus and the timeout. call(uri, payload, public) stays as a defaulted delegate, so ares-launch and ares-install did not change a line and are bounded anyway — verified against a device.

Worth review: the 30s default is a behaviour change for those two, which previously waited forever. subscribe_with is deliberately left unbounded, since a subscription is meant to stay open. LunaError gained variants and is now #[non_exhaustive], which protects future ones but not this change — a downstream exhaustive match (dev-manager-desktop) would need a wildcard.

Not included

Pointer/tap is parked on backup/ares-do-tap. It works and lands on the pixel, but the TV runs dropbear, which has no direct-streamlocal, so a unix socket cannot be forwarded and it needs a helper on the far end. That is a bigger departure than this tool should make.

Verified

On a 49LK5900 (webOS 4.4.3): key injection, both capture services, PNG to stdout, a multi-step flow from stdin, remote temp cleanup, and every documented exit code. 67 unit tests, clippy clean at workspace pedantic, fmt --check clean, and the committed keycode table matches its generator.

The second commit is unrelated rustfmt churn from a newer nightly, split out so the ares-do diff stays about ares-do.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a

Mariotaku and others added 7 commits August 23, 2026 01:58
Adds the tenth tool. It presses buttons and captures the screen, so a UI
flow can be scripted and replayed instead of walked by hand — the work
that motivated it (webosbrew/samples#1) meant running luna-send once per
keypress and diffing PNGs by hand.

Two undocumented private-bus services do the work:
test/sendKeyCode on com.webos.service.networkinput, and executeOneShot
on com.webos.service.capture, falling back to com.webos.service.tv.capture
(a 49LK5900 only has the latter). Both need root, so a uid probe runs
before anything is sent rather than after three keys have gone nowhere.

sendKeyCode reaches UInputWriter::sendKeyPress, which writes to
/dev/uinput. That settles two things: the codes are ordinary evdev and
stop at KEY_MAX, and there is no held modifier, so capitals cannot be
typed at all. `text` refuses them up front, naming every character, in
preference to a --shift flag that would quietly do nothing.

Which code each remote button sends is LG's choice and often not the
mainline name. /usr/share/X11/xkb/keycodes/lg in the firmware is the
authority, and the surprises are now aliases with notes: the remote's
Back is 412, not KEY_BACK (158). Confirmed on hardware — 158 moved 970
pixels, 412 dismissed the launcher and moved 1.4M.

The flow language is not invented: a line is tokenized and handed to the
same clap Command the shell uses, so the two cannot drift. The whole file
parses before anything runs, and every error is reported at once.

Ergonomics follow adb and xdotool rather than inventing conventions:
stdout is data and stderr is for people, so `screenshot > shot.png`
works; --json emits NDJSON; exit codes are a documented table; and
`run -` replays a flow from stdin over one SSH handshake instead of one
per key. `keys` and --dry-run need no device at all.

Also fixes LunaError in ares-connection-lib, which read luna-send's
stdout and stderr and then threw both away, collapsing "no such
service", "not allowed on this bus" and "the channel died" into one
value. New non_exhaustive Command variant carries them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
Pre-existing code that a newer rustfmt lays out differently. No
behaviour change; split out of the ares-do commit so that diff stays
about ares-do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
…at was never made

Both found by running --method VIDEO against a TV with no tuner signal.

com.webos.service.tv.capture answers that with
`"errorCode":"CAPTURE_ERROR_09"` — a string. LunaReply typed errorCode as
i32, so deserializing the reply failed and a perfectly clear error came
out as "the reply was not JSON", pointing at the transport instead of at
the TV. errorCode is now either shape, rendered rather than typed. The
message is now what the service actually said: "Could not capture in no
signal state (CAPTURE_ERROR_09)".

RemoteTemp also warned that it could not delete the staging file after a
capture the service had refused — there was no file, because the capture
never happened. It is now armed only once the service accepts, so the
guard still covers every path that can leave a file, and stays quiet
about the ones that cannot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
… everything

Three gaps, all of which the internals already had the machinery for.

`launch`/`close` now take -p, in the two shapes ares-launch accepts —
key=value and a whole {json}, repeatable and merged — so the same input
works with either tool. Unlike ares-launch, a param that is neither shape
is an error: a launch missing half its parameters is worse than one that
did not happen.

`luna` makes one call and puts the reply on stdout so it pipes into jq.
Private bus by default like the rest of ares-do, --public to switch. A
returnValue:false reply exits 7, which makes a `luna` line usable as an
assertion inside a flow, with --allow-false when probing for a service.

`exec` runs one command and exits with its status, the way ares-shell
does. That is the one place ares-do departs from its own exit-code table,
so --json also carries exitCode.

And the reason all three needed work rather than a wrapper: nothing sent
to the device was bounded. Luna::call reads to EOF, so a command that
never exits, or a luna method that subscribes instead of replying, hung
forever — the one failure an unattended run cannot recover from. Every
device command now goes through one exec path built on
Channel::read_timeout, draining stdout and stderr together so a full
stderr window cannot deadlock a wait on stdout. --timeout bounds each
command (30s default, 0 waits forever), verified: `exec 'sleep 300'`
--timeout 3s gives up after 3.3s with exit 9.

A side effect worth noting: ares-do no longer calls Luna::call at all, so
LunaError::LunaUnavailable is gone from its error enum. It builds the
luna-send line itself with the same snailquote escaping, and can now tell
"luna-send exited non-zero, and here is what it said" from "the reply was
not a reply".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
… around it

The previous commit bounded ares-do's device calls by building the
luna-send line itself. That fixed one tool and left the same hang in
every other one, which is backwards — the unbounded read was never
ares-do's problem, it was Luna::call's.

So the bound moves into the library. A new `exec` module offers a public
`Exec::exec_timeout` that drains stdout and stderr together against a
deadline, and `Luna` gains `call_with`/`subscribe_with` taking a
`LunaOptions` that carries the bus and the timeout. `call(uri, payload,
public)` and `subscribe` stay, now as defaulted delegates, so
ares-launch and ares-install did not change a line and are bounded
anyway — verified against a device.

Two things this makes possible that the workaround did not. Transfer's
own exec is still private and unbounded, but there is now something
public to point it at. And LunaError can finally say which of the four
things went wrong: NotAvailable is back to meaning only "it failed and
said nothing", with Command carrying luna-send's output, Reply carrying
what came back when it was not a reply, and Timeout carrying how long we
waited.

ares-do drops its own exec.rs and its hand-built command line, and goes
back to being a caller. Its three luna error variants collapse into one
that wraps LunaError, so the mapping to an exit code is in one place —
a timeout is 9 rather than 6, because a caller retries those differently.

Default is 30s, `--timeout 0` waits forever. Verified: `exec 'sleep 300'`
with --timeout 3s still gives up after 3.3s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
The aliases for buttons whose evdev code is not the obvious one were
hand-picked - GOBACK, GUIDE, SOURCE, VOICE - which made them my names
rather than LG's, and covered whatever happened to come up.

They are now generated from /usr/share/X11/xkb/keycodes/lg, which is LG's
own answer to "which code does this button send", under a REMOTE_ prefix.
That is not what LG calls them internally, where the word is RCU
(com.webos.service.mrcu, "LGE M-RCU", "Smart Remote RCU Input", 83 files),
but nobody outside the platform knows what an RCU is and REMOTE_BACK needs
no glossary.

Each button appears under LG's name for it and, where that differs from
what LG says it means, under both: the Input button is REMOTE_TV_VIDEO and
REMOTE_INPUT_SOURCE, and 139 is REMOTE_MENU and REMOTE_SETTINGS. The
meanings come from the Qt::Key_webOS_* names in the same file. 44 names for
27 buttons.

The generator drops anything above KEY_MAX, because sendKeyCode reaches
UInputWriter::sendKeyPress and that writes to /dev/uinput - so 149 of LG's
176 entries cannot be delivered however they are spelled, the remote's own
Home button (773) among them.

The prefix also settles the collision that started this. 158 is a real key -
KEY_BACK, XF86Back in xkb - which is exactly why it looks like the Back
button and is not; it is swallowed before any window sees it. Now BACK is
158 and REMOTE_BACK is 412, and neither has to pretend to be the other.
Re-measured on a 49LK5900: BACK moved 31k pixels, REMOTE_BACK moved 489k
and dismissed the launcher.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
Bare BACK resolved to 158, the kernel's KEY_BACK and XF86Back in xkb. That is
a real key, and the platform swallows it before any window sees it, so the tool
reported success and nothing happened - which cost another agent an afternoon.
The remote's Back button is 412.

BACK now refuses and says which spellings mean what. Both readings stay
reachable, just not by the ambiguous one: REMOTE_BACK for the button,
KEY_BACK or XF86BACK for 158. The XF86 prefix is accepted for any key, since
that is the name xkb uses and it says "the kernel's key, and I mean it" as
clearly as KEY_ does.

The rule is checked rather than remembered: a test walks every REMOTE_X whose
bare X is a different code and fails unless it is declared ambiguous. Today
BACK is the only one - EXIT, MENU, TV, SEARCH, INFO and the transport keys all
agree between the two tables.

Note this does not catch every trap, only the ones where both tables name the
same button. HOME is still 102 while the shell's home is HOMEPAGE (172), and
whether 102 reaches anything is untested.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gde2tcn5SbUHKhesuTNB7a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant