Anthropic has just rolled out a significant update to Claude Code: the Computer Use feature.
In short, Claude can now directly launch macOS applications, click interface buttons, take screenshots, and manipulate UI elements directly from the command line, feeding visual results right back into your terminal session.
Write code, compile, launch applications, click-test, catch bugs, patch them, and verify the fix: all streamlined within a single terminal workflow.
01 | Let's See It in Action
The official demo video walks through a complete, real-world scenario.
A user opens a pixel art editor project named SPM3000 inside Claude Code and types a simple prompt:
The feedback channel says the Gen button throws an ERR 19 mid-render. Open SPM3000, reproduce the issue, fix it, and ship a new build for me.
Claude first runs git status, maps out the Xcode project structure, and then directly launches the SPM3000 application.
02 | Taking Manual Control to Click
Next, Claude does something unexpected: it announces via the terminal, I'm enabling Computer Use so I can click around for you. By running /mcp, it activates the Computer Use capability, gaining screen-clicking, text input, and screenshot permissions.
It launches SPM3000, clicks the GEN button at coordinates (412, 580). A pixel-art creature appears on screen but crashes halfway through rendering. The bottom panel flashes: ERR 19, ROW 16/32, BUFFER OVERRUN.
Claude captures a screenshot, saves it to /tmp/gen-before.png, and confirms the issue: rendering aborts precisely at row 16.
03 | Tracking Down the Bug
With the bug visually confirmed, Claude pivots back to its native domain: source code.
It uses grep to search for ERR 19|rowIndex, pinpointing the relevant lines at 47 and 52 in Sources/GenViewModel.swift. After inspecting the renderSprite() function, it identifies the culprit: the palette logic immediately returns .err(19) if the palette is empty, halting the renderer.
The fix is straightforward: it replaces the strict guard let palette else return .err(19) condition with a fallback to lastGoodPalette, preventing a complete render crash just because of a missing color palette.
04 | One-Pass Verification
After patching the code, Claude triggers xcodebuild -scheme SPM3000 -configuration Debug build. Build Succeeded.
It relaunches SPM3000 and clicks the GEN button again. This time, all 32 rows render flawlessly. The pixel-art creature appears fully intact, wearing sunglasses in a looping animation.
Claude responds: Full render complete, all 32 rows passed, sunglasses animation looping successfully. The fix is verified. Want me to push a build? The user replies with a single word: Good.
From initial bug report to verified fix: roughly two minutes.
05 | Availability and Setup
Computer Use is currently available as a Research Preview, limited to macOS, and requires a Claude Pro or Max subscription.
How to enable it:
Run /mcp in your Claude Code session.
Select computer-use, then choose Enable.
On first use, macOS will prompt for two permissions: Accessibility (to allow clicks, typing, and scrolling) and Screen Recording (to let Claude view your display).
Once authorized, simply tell Claude what to operate. For example: Compile MenuBarStats, launch it, open the preferences window, and verify the interval slider updates the label correctly. Claude will handle compiling, launching, interacting, and screenshotting autonomously.
06 | It Is Not a Universal Tool
Claude follows a strict priority hierarchy when selecting execution methods:
If an MCP server is configured, it uses that first.
If a Shell or Bash command can solve it, it defaults to the CLI.
If it is a browser task and Claude in Chrome is installed, it operates there.
Computer Use is the explicit last resort.
This architecture makes perfect sense. Visual UI interaction is inherently slower and more resource-heavy. Direct command execution or API calls are always faster. Computer Use is strictly reserved for legacy, proprietary, or purely GUI-driven software lacking a CLI or automation API.
07 | Practical Use Cases
Based on documentation and the demo, here are the primary workflows where Computer Use shines:
Native App Verification: After tweaking SwiftUI code, let Claude compile, launch, click through the UI, and screenshot the result. No more manually opening the app to inspect changes.
End-to-End UI Testing: Point it at a local Electron app and say test the registration flow. Claude will launch it, walk through sign-up, and capture each step. No Playwright setup required.
Visual Regression Debugging: Tell it the modal is clipping on small windows. It will resize the viewport, capture the bug, patch the CSS, and verify the fix.
Controlling Pure GUI Tools: Design software, hardware control panels, iOS Simulators, and other command-line-hostile apps are now fully operable through Claude Code.
08 | Built-In Security Mechanisms
Anthropic has baked in robust safety guardrails:
Every application requires explicit, session-level approval. If Claude wants to access Finder, it prompts you: This app can read or write any file. Want to control the terminal? Warning: Equivalent to shell access. These permissions are rightly granular.
Control levels vary by app type: browsers and trading platforms are view-only. Terminals and IDEs allow clicking but block direct text input. Only whitelisted general apps get full interaction rights.
During operation, Claude automatically hides non-approved windows and only interacts with your authorized target. The terminal itself remains visible but is deliberately excluded from screen captures to prevent prompt injection from its own output. At any point, pressing Esc instantly kills the Computer Use session and returns control to you.
09 | Differences from Claude Desktop
Claude Desktop already supports similar Computer Use capabilities. The CLI version differs mainly in its activation method (/mcp versus a GUI settings panel). It currently lacks a custom blocked-apps list, and its restore-hidden-windows behavior is hard-coded as enabled. Under the hood, they share the exact same control engine.
10 | The Bigger Picture
THE CORE BREAKTHROUGH HERE IS THE EXPANSION OF THE AI PROGRAMMING ASSISTANT'S PERCEPTION BOUNDARY.
Previously, Claude Code could read, write, and execute code, but it was fundamentally blind to runtime output. Change a CSS rule, and you would have to open a browser, inspect the render, and textually describe it back to the AI. Now, it can see and interact natively.
THE DEVELOPER AND THE QA TESTER ARE NOW EFFECTIVELY THE SAME ENTITY.
This marks a paradigm shift for frontend, mobile, and desktop development. The tedious loop of coding one line and manually verifying it can finally be automated.
Current Limitations: Still a Research Preview, macOS-exclusive, requires Claude Code v2.1.85 or higher, does not support the -p non-interactive flag, and excludes Team and Enterprise plans at this stage.
I have already deployed it in my own workflow:
Relevant Links: