vercel-labs/agent-desktop: Computer use CLI for AI agents

Overview

agent-desktop is a command-line interface and C library designed to give AI agents native computer-use capabilities. Instead of relying on brittle screenshots or pixel-matching techniques, it leverages official OS accessibility APIs to read desktop applications directly, presenting them as structured JSON trees.

How It Works

The tool inspects the target application and assigns a unique reference (ref) to every interactive UI element. The calling agent queries a snapshot, performs actions like clicking or typing using the ref, and checks the updated UI state in a continuous loop. By default, it operates headless, interacting with UI elements without needing to move the mouse pointer or steal active focus. Ecosystem and Integration agent-desktop natively supports macOS, Windows, and Linux by wrapping platform-specific accessibility APIs like AX API, UI Automation, and AT-SPI. It also includes an MCP (Model Context Protocol) server mode out of the box, allowing developers to easily expose all desktop commands as tools to any compatible AI assistant.