Pet — AI Desktop Companion · AI 桌面宠物

English | 中文

A Live2D desktop pet that lives on your screen and is backed by an LLM agent. It can chat, see your screen, run shell commands, read and drive your macOS apps, talk to you over Telegram, and act on its own via a scheduled heartbeat — all while keeping a persistent long‑term memory of you.

One Rust engine crate, two interfaces: a Tauri 2 + React desktop GUI, and a ratatui terminal UI (pet-cli) that shares the same config, sessions and memory — see docs/architecture.md.

Features

• Live2D pet — an animated character that floats on your desktop, auto‑collapses into a floating ball, and can be pinned on top.
• Chat in two windows — the pet itself and a panel window share one conversation; talk to whichever is in front.
• Multiple agents & group chat — configure several agents (each with its own model, persona/memory, MCP set), switch who answers, or put them in one room where they react to you and to each other concurrently.
• Terminal UI — pet-cli runs the same engine in your terminal: markdown rendering, collapsible tool calls, a / command palette with interactive pickers, and the group room. See docs/cli.md.
• Any OpenAI‑compatible model — point api_base at OpenAI, a local server, or a proxy (e.g. litellm). Streaming responses with a tool‑calling agent loop.
• Vision — paste images into the chat, or let the pet take a screenshot (the whole screen, or a single app's window) and look at it.
• Tools — bash, file read/write/edit, screenshot, web_search, and spawn_subagent for parallel background work; long commands run in the background and notify on completion.
• Web search — web_search looks things up on the live internet via Tavily. Set search_api_key (Settings → Web Search) to enable it; without a key the tool isn't offered to the model.
• macOS app control & reading — via osascript the pet can drive scriptable apps (AppleScript) and GUI‑script the rest (System Events) — e.g. open Terminal and run a command, or read text straight from an app's UI. See docs/macos-automation.md.
• Telegram bot — chat with your pet from your phone, including sending and receiving images. See docs/telegram.md.
• Scheduled heartbeat — the pet wakes up in the background on an interval to run timed tasks or reach out proactively.
• Persistent memory — SOUL.md (persona), USER.md (about you), and MEMORY.md (its long-term memory) survive across conversations.
• Agent Skills — drop a folder containing a SKILL.md into your skills directory to teach the pet a procedure. Only each skill's name and description sit in the prompt; the body is read on demand, so skills you don't use cost almost nothing. Invoke one by name with /skill:<name> in either interface. See docs/skills.md.
• MCP support — connect Model Context Protocol servers (stdio / SSE / HTTP) to extend the toolset.
• Gallery mode — swap the pet for a slideshow of a folder's images/videos.
• Bilingual UI — 中文 / English.

Tech stack

Cargo workspace: crates/pet-core (engine) · src-tauri (GUI) · crates/pet-cli (TUI). Tauri 2 · React 19 · TypeScript · Vite · TailwindCSS 4 · PIXI.js 7 + pixi-live2d-display · Rust (ratatui, teloxide, reqwest, rmcp, pulldown-cmark).

Prerequisites

• Node.js + pnpm
• Rust toolchain and the Tauri 2 system dependencies
• A Live2D model + Cubism SDK of your own — or let the app fetch a sample. These are copyrighted and not bundled: either drop them into public/models/ and public/lib/ (both gitignored) and point live_2d_model_path at your model's .model3.json, or click Use sample model in Settings → Live2D to download Live2D's official sample (© Live2D Inc.) automatically.

Quick start

pnpm install
pnpm tauri dev          # run the desktop GUI in development
pnpm tauri build        # build a release bundle

cargo run -p pet-cli    # terminal UI (shares the GUI's config & sessions)

On first launch the pet window appears. Open the panel, go to Settings, and fill in your API base, key, and model. Configuration is written to config.yaml — see docs/configuration.md for every field and where the data lives.

Documentation

• Architecture — one engine crate, two interfaces
• CLI — pet-cli commands, keys, and group mode
• Configuration reference — config.yaml fields and file locations
• Agent Skills — teach the pet a procedure with a SKILL.md folder
• macOS automation — reading/controlling apps + permissions
• Telegram bot — setup and image support
• Evals — behavioral cases run against the real engine in a sandbox
• Release — tag-driven GitHub Actions build of the app and CLI

---

中文

English | 中文

一个住在你屏幕上的 Live2D 桌面宠物，背后是一个 LLM 智能体。它能聊天、看你的屏幕、 执行 shell 命令、读取并操作你的 macOS 应用、通过 Telegram 和你对话，还能靠定时心跳 自己主动行动 —— 同时对你保有一份跨对话的长期记忆。

一套 Rust 引擎，两套界面：Tauri 2 + React 桌面 GUI，以及共享同一份配置/会话/记忆的 ratatui 终端界面（pet-cli）—— 见 docs/architecture.md。

功能

• Live2D 宠物 —— 浮在桌面上的动画角色，会自动收起为悬浮球，可一键置顶。
• 双窗口聊天 —— 宠物本体和面板窗口共享同一段对话，哪个在前就跟哪个说。
• 多 Agent 与群聊 —— 可配置多个 Agent（各自的模型、人设/记忆、MCP 工具集）， 随时切换应答者，或拉进同一个群里，让它们并发地回应你和彼此。
• 终端界面 —— pet-cli 在终端里跑同一套引擎：Markdown 渲染、可折叠的工具调用、 / 命令面板 + 交互式选择浮层、群聊模式。见 docs/cli.md。
• 任意 OpenAI 兼容模型 —— api_base 可指向 OpenAI、本地服务或代理（如 litellm）。 流式输出 + 工具调用 agent 循环。
• 视觉 —— 可往聊天里粘贴图片，或让宠物截图（整屏，或某个 App 的单个窗口）来看。
• 工具 —— bash、文件读/写/改、screenshot、web_search、spawn_subagent （并行后台任务）；长命令转入后台，完成后自动通知。
• 联网搜索 —— web_search 通过 Tavily 搜索实时互联网。 在「设置 → 联网搜索」填入 search_api_key 即启用；不填则不向模型提供该工具。
• macOS 应用操作与读取 —— 通过 osascript，宠物能驱动可脚本化应用（AppleScript）， 对其余应用做 GUI 自动化（System Events）—— 比如打开 Terminal 跑命令、或直接读取 App 界面上的文字。详见 docs/macos-automation.md。
• Telegram 机器人 —— 在手机上和宠物聊天，支持收发图片。 详见 docs/telegram.md。
• 定时心跳 —— 宠物按设定间隔在后台醒来，执行定时任务或主动找你。
• 长期记忆 —— SOUL.md（人设）、USER.md（关于你）、MEMORY.md（它的长期记忆） 跨对话保留。
• 技能（Skills） —— 往技能目录里放一个带 SKILL.md 的文件夹，就教会宠物一套 做事流程。系统提示里只放名称和用途，正文按需读取，用不上的技能几乎不占上下文。 两套界面里都能用 /skill:<名称> 直接点名调用。详见 docs/skills.md。
• MCP 支持 —— 接入 Model Context Protocol 服务（stdio / SSE / HTTP）扩展工具集。
• 画廊模式 —— 把宠物换成某个文件夹的图片/视频幻灯片。
• 中英双语界面。

技术栈

Cargo workspace：crates/pet-core（引擎）· src-tauri（GUI）· crates/pet-cli（TUI）。Tauri 2 · React 19 · TypeScript · Vite · TailwindCSS 4 · PIXI.js 7 + pixi-live2d-display · Rust（ratatui、teloxide、reqwest、rmcp、 pulldown-cmark）。

前置条件

• Node.js + pnpm
• Rust 工具链 及 Tauri 2 系统依赖
• 你自己的 Live2D 模型 + Cubism SDK，或者用 app 内置的示例获取按钮。它们受版权 保护、不随仓库分发 —— 既可以放进 public/models/ 和 public/lib/（均已 gitignore），再把 live_2d_model_path 指向你模型的 .model3.json；也可以在 「设置 → Live2D」点「使用示例模型」，自动下载 Live2D 官方示例（版权归 Live2D 所有）。

快速开始

pnpm install
pnpm tauri dev          # 开发运行桌面 GUI
pnpm tauri build        # 打包发布版

cargo run -p pet-cli    # 终端界面（与 GUI 共享配置和会话）

首次启动会出现宠物窗口。打开面板进入 设置，填入你的 API base、key 和模型。 配置写入 config.yaml —— 每个字段和数据存放位置见 docs/configuration.md。

文档

• 架构 —— 一套核心引擎，两套界面
• 命令行界面 —— pet-cli 命令、按键与群聊
• 配置参考 —— config.yaml 字段与文件位置
• 技能 —— 用 SKILL.md 目录教宠物一套做事流程
• macOS 自动化 —— 读取/操作应用与所需权限
• Telegram 机器人 —— 配置与图片支持
• 评测 —— 在沙箱里拿真引擎跑行为用例
• 发布 —— 打 tag 触发 GitHub Actions 构建 app 与 CLI