Skip to content

feat(ui): open a phone in a window of its own - #128

Merged
DIYgod merged 2 commits into
mainfrom
claude/blissful-ptolemy-ba1123
Nov 14, 2026
Merged

DIYgod merged 2 commits into
mainfrom
claude/blissful-ptolemy-ba1123

Conversation

@DIYgod

@DIYgod DIYgod commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Closes #122.

Open in a window in a phone's ⋯ menu under its mirror, and in its right-click menu (sidebar row, All phones tile), opens a window with only that phone's live screen, for iPhones, iPads and Android phones. Opening it again brings the existing window to the front.

What the window does

  • Layout. The screen keeps the phone's aspect ratio. Under it is a 44-point bar with:
    • the keys (Home; Back and Recents on Android);
    • while the person is in control, a short form of the control banner: the hand icon, the bold words when they fit (the full text is in the tooltip and the accessible label), and Hand back;
    • a Keep on top pin, off at first, remembered for each phone while Pinky runs.
  • Screen overlays. The lock pill and the agent's overlay appear on the screen, as on the page. Nothing covers the screen or moves it under the finger.
  • Input. Input goes through the existing mirrorInput. mirrorView now takes a surface (onPage, onTile, onWindow), and the mirror keeps the screen's box for each surface, so a click maps to device pixels at the size of the view it came from. This works even when the page and the window show the mirror at the same time at different sizes. A touch takes over and pauses the agent exactly as on the page.
  • One stream. The window shows the phone's own mirror from mirrorFor, so it uses the same gateway stream, decoder and bitmap as the main window. inWindow keeps that mirror out of sweepMirrors and closeMirrors, so it stays open while the main window shows another page or is hidden or minimized. tileStill keeps an iPhone's tile live while its window is open. When the window closes or the phone stops being live, the window lets the mirror go.
  • States.
    • Paused, connecting, offline, locked over Wi-Fi or needing setup: a box shows the state, with Show in Pinky.
    • Phone removed: its window closes.
    • ⌘W closes the window.
  • Shape. MyGo has no aspect-ratio lock, so:
    • The window opens with a screen up to 640 high.
    • StateKey phone-<id> remembers where it was placed.
    • 400 ms after a resize, it snaps to the screen's shape, keeping the side the person changed more.
    • When the phone turns, it reshapes and keeps the same screen area.
  • Refactors shared with the page:
    • controlFor: the banner's text and whether the person is in control.
    • phoneKeys: the keys.
    • The right-click menu now always exists, since it always has Open in a window.

Tests

ui/phonewindow_test.go:

  • TestPhoneWindowInput:
    • opening from the ⋯ menu;
    • one mirror shared with the page;
    • clicks in the window and on the page, at different sizes, reach the same device pixels;
    • takeover, then Hand back from the window's bar;
    • reopening keeps a single window.
  • TestPhoneWindowKeepsTheStream: hiding the main window or switching pages keeps the mirror; closing the window releases it.
  • TestPhoneWindowStates:
    • the sidebar row and tile right-click menus;
    • offline state and Show in Pinky;
    • an iPhone has only Home;
    • Keep on top turns on and off;
    • a removed phone closes its window.
  • TestPhoneWindowShape: sizes for opening, resizing and turning.
  • TestPhoneWindowScreenshots: needs PINKY_SCREENSHOTS; renders fake phones only.

make check passes.

I also checked it in the UI demo (-scenario three, test-pattern streams, driven through MyGo's automation socket):

  • the window opens with the live picture;
  • a click takes over, and both windows show Hand back;
  • Hand back from the window works;
  • Keep on top toggles;
  • ⌘W closes the window;
  • resizing to 360 wide settles at 360 × 848.

Not tried on a real phone.

Docs: README.md and README.zh-CN.md (new bullet before "Removing a phone"), plus a section in docs/notes/client.md. Messages are added in en, zh-Hans and ja. The new icons are MingCute external_link_line, pin_line and pin_fill.

🤖 Generated with Claude Code

DIYgod and others added 2 commits November 14, 2026 22:17
Add "Open in a window" to the phone's ⋯ menu and right-click menu. The
window shows only the phone's live screen at its aspect, with its keys,
a small control banner with Hand back, and an optional Keep on top pin.
It shares the phone's mirror and its one stream with the main window,
maps input at its own size through mirrorInput, follows rotation, shows
the offline, paused or connecting state, and closes when the phone is
removed.

Closes #122

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DIYgod
DIYgod merged commit 5c0e5ef into main Nov 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

可不可以把 UI 设计成代替 iPhone Mirroring 的功能?

1 participant