agent-desktop by Vercel Labs: Claude Code uses your Mac’s apps in the background while you keep working

Yes: with agent-desktop, a free, open source project from Vercel Labs, Claude Code uses your Mac’s apps in the background. It opens Notes, types in TextEdit, reads Control Center, and you keep working on the computer the whole time. agent-desktop reads windows through macOS accessibility, with no screenshots, and it never takes over your mouse and keyboard.

Here is what I tested, in order:

1. How to install agent-desktop with a single prompt
2. How Claude Code writes a note in Notes
3. Why you should write "with agent-desktop" in the prompt
4. Control Center and system apps since version 1.1
5. What agent-desktop can’t do well yet
6. How many tokens it uses
7. What changes for privacy
8. How it differs from Claude Code’s computer use
9. Where to find it

What is agent-desktop?

agent-desktop is a command-line tool from Vercel Labs that lets an AI agent see and operate desktop apps through the operating system’s accessibility tree. It’s the same structure macOS uses for screen readers: every button, text field and menu has a name and a role. The agent reads that structure, picks the element to act on, acts, and then reads the interface again to check the result. In Claude Code, agent-desktop comes as a skill.

According to the official repository, agent-desktop actions are headless by default: the pointer doesn’t move and the focused window doesn’t change. That’s why Claude Code can work in one app while you use your Mac. The project is licensed under Apache 2.0 and supports macOS, Windows and Linux.

How do you install agent-desktop in Claude Code?

Installing agent-desktop takes one prompt. I asked Claude Code to install agent-desktop from Vercel Labs, with the link to the GitHub repository and a request to install its skill for Claude Code. Claude Code did the rest on its own: it installed the skill and checked the macOS permissions.

On my Mac, the permission check came back like this:

• Accessibility: already granted. Claude Code needs it to launch apps and control the Mac.
• Screen Recording: denied. Without it, reading windows (snapshot), clicking and typing still work. If you want to enable it, Claude Code tells you which command to run. Otherwise you can skip it.

If the Accessibility permission is missing, give Claude Code a second prompt: "check whether agent-desktop has the Accessibility permission and, if it’s missing, open the guide for me". Then follow the steps. The agent-desktop README also documents a manual install with npm install -g agent-desktop and the permissions guide with agent-desktop permissions --guide.

After installing, quit Claude Code and open a new chat. That way the skill gets reloaded.

How does Claude Code write a note in Notes?

I ran the first test with the Notes app. The prompt was: "with agent-desktop, write a new note in Notes". Claude Code loaded the skill, opened Notes and found an empty note already selected. That’s why the new note command was disabled, so Claude Code typed straight into that note the text I had asked for: "today’s video: record, edit, publish".

While Claude Code was writing in Notes, I could have kept working in Claude Code or in another app. Some apps, though, don’t stay in the background and sometimes jump to the front. That depends on how the app you’re using is built, and agent-desktop has no control over it. Even then, you can keep working while Claude Code acts.

Why write "with agent-desktop" in the prompt?

Start the prompt with "with agent-desktop", then add the task. If you have other app-control tools in Claude Code, without that cue Claude Code can pick a different one. Naming the skill in the prompt makes Claude Code use agent-desktop specifically.

In the next test I forgot to write it, and Claude Code loaded agent-desktop on its own. It works that way too, but writing it removes any doubt.

Can Claude Code use the Mac’s Control Center?

Yes, since agent-desktop version 1.1. In the second test I asked Claude Code to open Control Center and check whether Bluetooth was on. Claude Code opened the panel on the right side of the screen and read the system settings.

Up to the previous version, Control Center, system apps and the Dock, the bar at the bottom with your apps, were out of reach, so computer use was limited. The v1.1.0 release notes, published on November 3, 2026, list the fix that lets agent-desktop reach the Dock, menu bar status items and system panels.

What can’t agent-desktop do well yet?

agent-desktop has just come out, and I noticed a few limits. The most obvious one: when it works in the background, agent-desktop can’t save the files it edits in TextEdit, the Mac’s text editor. I opened a new document and saving didn’t go through.

With TextEdit the problem barely matters, because TextEdit autosaves. The case still tells you something useful: some apps don’t work well with agent-desktop yet. Test the apps you need before you hand them to Claude Code.

How many tokens does agent-desktop use with Claude Code?

A simple task with agent-desktop uses tens of thousands of tokens. I checked the context of my two test chats:

• Notes chat: 79,000 tokens.
• Chat with the other tests: 69,000 tokens, 7% of the context.

In total, two simple tasks cost me about 150,000-160,000 tokens. If you pay per use, or you’re on the $20 plan, keep that in mind. The $100 plan leaves room for a few more tasks.

What changes for privacy with agent-desktop?

On privacy, agent-desktop changes nothing compared to the usual setup. The program is open source, you install it yourself and it runs entirely on your Mac. Claude Code, however, runs on Anthropic’s servers: the data Claude Code reads from your apps ends up on Anthropic’s servers, just as it does when you use Claude Code for vibe coding or any other work.

How does it differ from Claude Code’s computer use?

Claude Code already knows how to use the computer through computer use. At the time I recorded the video, though, Claude Code’s computer use doesn’t run in the background. Every task takes over the Mac, and you can’t keep working while Claude Code acts. The same goes for Codex. I’ve already written about models that use the computer in my piece on GPT-6 Astra with Excel, Power BI and KiCad.

Both Anthropic and OpenAI have introduced background work, but only in the Claude and Codex desktop apps, not in Claude Code, and for now it’s in beta. agent-desktop, on the other hand, runs in the background by default.

agent-desktop isn’t perfect either. It does have one extra advantage: being open source, it’s interchangeable and doesn’t tie you to a single provider. If you want the same approach on the web, I tested Browser Use with 10 AI browsers in parallel.

Where can you find agent-desktop?

agent-desktop is free and lives on GitHub, in the vercel-labs/agent-desktop repository. To get started you only need the install prompt and a new Claude Code chat.

The best way to see whether you need it is to give it a small task in an app you use every day. If you try it, tell me what you use it for: I’d like to know which job you handed to Claude Code while you were doing something else.