Description
Getting an AI agent to click through software usually means screenshots and guessing button positions from pixels, which breaks with a new resolution or theme. agent-desktop takes another route: it reads the OS accessibility tree and gives every actionable element a stable ref, so the agent clicks by ref and re-reads the UI to verify, with no pixel matching. It is a Rust CLI and C library from Vercel Labs for Windows, macOS and Linux, installed with one npm command.
It is a tool, not an agent: the observe, act, verify loop lives in the calling agent or script, and there is no model inside. Actions run in the background by default without taking your pointer or focus, and --headed enables physical input. Snapshots, clipboard reads and screenshots can contain private desktop content, so keep them out of public logs.
Snapshots and refs: JSON output with a ref per actionable element, bound to that snapshot; a changed UI returns STALE_REF and asks for a fresh snapshot.
Background actions: runs through accessibility APIs without moving the pointer or stealing focus.
Layered reads for dense apps: map the app with a skeleton view, then drill into one region.
Cross platform: macOS needs Accessibility permission, Windows needs no grant at the same integrity level, Linux needs at-spi2-core.
Built in guide: agent-desktop skills get desktop prints the usage guide matched to your version.
C library: embed it from other programs, not only the CLI.
It is a tool, not an agent: the observe, act, verify loop lives in the calling agent or script, and there is no model inside. Actions run in the background by default without taking your pointer or focus, and --headed enables physical input. Snapshots, clipboard reads and screenshots can contain private desktop content, so keep them out of public logs.
Features
Snapshots and refs: JSON output with a ref per actionable element, bound to that snapshot; a changed UI returns STALE_REF and asks for a fresh snapshot.
Background actions: runs through accessibility APIs without moving the pointer or stealing focus.
Layered reads for dense apps: map the app with a skeleton view, then drill into one region.
Cross platform: macOS needs Accessibility permission, Windows needs no grant at the same integrity level, Linux needs at-spi2-core.
Built in guide: agent-desktop skills get desktop prints the usage guide matched to your version.
C library: embed it from other programs, not only the CLI.
