Hey guys, you’ve probably seen the huge Anthropic announcement introducing Computer Use – I’ve been playing with the demos and getting setup on my computer, and I’ve learned some limitations of the model, as well as certain prompts you can use to get it to perform better.
Introduction to Computer Use
Claude’s Computer Use is a next-level tool that lets you run AI tasks on your computer just like a human would, but with the power of Claude 3.5 Sonnet. You can have it open apps, navigate through websites, and even execute commands. But here’s the kicker: it’s not flawless yet. Despite the excitement, there are some quirks, limitations, and things to keep in mind when using it. I’ll walk you through how to set it up locally and share what I’ve learned while experimenting with it.
Getting Started: The Cloud Setup (Repl.it)
Let’s see what it’s like running Claude’s Computer Use in a cloud environment. Repl.it (now called Replit) is a great place to get started quickly. Here’s how to set up a basic environment:
- Sign up for Replit: If you don’t have a Replit account, head over to Replit.com and sign up for a free account. Once you’re in, you’ll land on your dashboard.
- Create a New Repl: From the dashboard, click on the “Create” button and search for “Computer Use”. Select it, and Replit will auto-generate a demo environment for you. Pretty neat, right?
- Connect Your API Key: Before you can run the demo, you’ll need to input your Anthropic API key. To get that, head over to the Anthropic Console and generate a new key. Once you’ve got it, paste it into the API key section in Replit. Boom, you’re good to go!
Once that’s set up, you should see a bunch of code running on the left side of your screen. The coolest part is the agent loop that manages Claude’s actions. It’s like the AI is doing its own thing in the background, figuring out how to click, type, and interact with your machine.
Quick Replit Demo (5-10 min)
If you’re just getting started, running the Replit demo is a great way to see Claude’s Computer Use in action. Follow these steps to get it running in just a few minutes
- Export API Key from Console:
JAVASCRIPT
ddocker run \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v $HOME/.anthropic:/home/computeruse/.anthropic \
-p 5900:5900 \
-p 8501:8501 \
-p 6080:6080 \
-p 8080:8080 \
-it ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest- Wait for It to Finish Downloading: Give it a few minutes to finish setting up the environment.
- Run the Demo: You’re now ready to start interacting with Claude via the demo environment!
Start the Day with 0 Cost
Exploring Claude’s Features: Let’s Play Around
So, once you’re set up, let’s play around and see what Claude can do! The first thing you’ll notice is how it takes screenshots and interacts with your environment. I had it search for something on Wikipedia and take screenshots of what it was doing in real-time.
Claude doesn’t just type text or make mouse clicks—it actually mimics human actions like scrolling, selecting buttons, and even typing in search bars. For example, I asked Claude to search for “neural networks” on Wikipedia. It pulled up the page, took screenshots, and generated a summary for me. It was almost like I had an assistant doing the research for me.
Thoughts: What’s Impressive & What Needs Work
Before diving deeper into the setup, here are some key thoughts I’ve gathered from using it:
Things That Impressed Me:
- Navigating Apps Like a Pro: Claude can open applications like Excel and run basic tasks without much help. I had it perform a simple search on Wikipedia, and it worked seamlessly.
- Screenshot Handling: Even though it can be a bit overzealous at times, I was surprised at how it used screenshots to “see” what’s happening on my screen. This is how Claude can perform tasks in web browsers and apps.
Some Key Limitations:
- Gets Stuck on Firefox: While Claude can interact with browsers, it does get stuck navigating through certain pages, like those with pop-ups or requiring login steps. I asked it to search for something, and while it could successfully navigate to the Wikipedia page, it struggled when trying to skip certain steps in the process.
- Excessive Screenshots: It tends to capture screenshots even when unnecessary, which can be a bit annoying and a waste of resources.
How They Trained the Model
Claude’s Computer Use is powered by Anthropic’s model that was trained to mimic how a human interacts with a computer. It can perform automated tasks by interacting with graphical user interfaces (GUIs), but this doesn’t mean it can perform all tasks flawlessly. It’s still a work in progress, which is why you see occasional mistakes or delays when completing actions.
For example, I had it scrape data from a website, and while Claude got the URL and basic info, it wasn’t able to properly “scrape” the content of the page like a more advanced tool would. It also had trouble scrolling to load more content, and sometimes it just froze altogether.
Limitations: What You Need to Know
Claude’s Computer Use is impressive in many ways, but it’s still limited in a few major areas. Let’s break it down:
- It Can’t Login to Your Accounts: One of the key limitations I encountered was Claude’s inability to log into websites for you. I tried to get it to sign into a Google account, but it couldn’t do that. This makes it less useful for tasks that involve personalized access, like checking email or social media accounts.
- Can’t Create Accounts: I also tried asking Claude to create a new Google Drive account and save a CSV file there. It couldn’t do this either, which is a major limitation for any workflow that requires handling personal data.
- Distracted by Screen Activity: At one point, it seemed like Claude stopped coding altogether, likely because it was distracted by my on-screen activity. This raises the question: was it trained on my screen activity? I think that’s possible, given the way it interacts with my screen.
- Base64 Encoded Images: One thing I noticed is that Claude only uses base64 encoded images for interactions, which limits its ability to scrape or gather content from dynamic pages. This means it’s heavily reliant on visual cues rather than understanding the content like a human would.
Cost Breakdown: How Much Does It Cost?
Running Claude comes with a cost, especially if you’re using a lot of actions or screenshots. Here’s an approximate breakdown:
- Per Screenshot: 19 screenshots = 13 cents
- Cost per Action: $0.0068 per action
- Estimated Cost for 100 Actions: $0.68
- Estimated Cost for 1,000 Actions: $6.80
If you’re doing repetitive tasks, it can quickly add up. Make sure you optimize the number of actions and be mindful of your usage!
Practical Tips for Prompting Computer Use
Now, let’s get into some practical tips to make the most of Computer Use:
- Specify Websites/Tools: Claude works best when you specify the exact website or tool you’re interacting with. It’s not great at doing everything in one go, so be specific in your prompts.
- Explain High-Level Steps: To save time and money (as actions can cost a bit), give Claude high-level steps. Instead of saying “find a document,” you could say, “open a file in Google Docs and edit it.”
- Focus on Simple Tasks: Claude is still learning. Complex tasks might result in failures or excessive resource consumption.
Running Locally
To run the full demo on your machine, follow these steps:
- Install Docker: You’ll need Docker to run the virtual environment. It’s free to download from Docker’s website.
- Clone the Demo Repo: Clone the repo from GitHub and pull down the demo files.
- Export Your API Key: Once you’ve set up your environment, use the API key to authenticate your session.
- Run the Docker Command: Execute the Docker command to run the local instance.
Conclusion
Claude’s Computer Use is an exciting new feature, but it’s still in its early stages. There are some things it does really well, like basic navigation and taking screenshots. However, it has limitations, especially when it comes to logging in to accounts or performing complex tasks. If you want to dive deeper into AI or explore how to build apps like this, I’ve got more resources in the description, and I’d be happy to help you build your first AI app in less than a week.
🎥 Watch the Full Demo on YouTube!
Want to see the whole setup and demo in action? Check out my full YouTube video where I walk you through everything, step by step: Claude Computer Use – How to Setup Locally
Let me know if you have any questions or want to dive deeper into this tech.