Drive a browser with Claude Computer Use

Jun RyuUpdated Apr 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

Claude sees the screen as an image and returns concrete actions at pixel coordinates: left_click [640, 412], type "claude 4.7 opus", scroll down 3. Something has to execute those actions against a real browser and send the next screenshot back. That "something" is the agent loop in index.ts, and the browser is a Steel session. The Claude Computer Use integration covers install and setup.

The loop

The whole thing fits in one while block inside Agent.executeTask. Each iteration sends the growing message history plus the computer tool definition to Claude:

const response = await this.client.beta.messages.create({
  model: this.model,
  max_tokens: 4096,
  messages: this.messages,
  tools: this.tools,
  betas: ["computer-use-2025-11-24"],
});

The tool definition declares computer_20251124 with the viewport's display_width_px and display_height_px. Keep it consistent with the Steel session's dimensions (1280x768 here) or clicks land in the wrong place.

executeComputerAction is the translation layer. Claude emits computer-use actions (left_click, type, key, scroll, screenshot, ...); Steel's Input API speaks a parallel vocabulary (click_mouse, type_text, press_key, scroll, take_screenshot):

case "left_click":
case "right_click":
case "middle_click":
case "double_click":
case "triple_click": {
  body = {
    action: "click_mouse",
    button: buttonMap[action],
    coordinates: coords,
    screenshot: true,
  };
  break;
}

Every action sets screenshot: true, so Steel returns a fresh base64 PNG after each interaction. That PNG becomes the content of a tool_result block in the next user message.

A few translation details:

Keys get normalized. normalizeKey maps synonyms (CTRL to Control, CMD to Meta, ENTER to Enter) before sending to Steel.
Scroll is delta-based. Claude says scroll_direction: "down", scroll_amount: 3; Steel expects delta_x/delta_y in pixels. The code multiplies by 100 per step.
Drags default from center. left_click_drag only gives an end coordinate, so the start is the viewport center.

Stop conditions

No tool calls. Claude wrote only text. Task is complete.
Repetition. detectRepetition compares the last assistant message against the previous three by word overlap (>80%).
Iteration cap. 50 iterations by default.

The finally block in main always calls agent.cleanup(), which releases the Steel session.

Run it

cd examples/claude-computer-use-ts
cp .env.example .env          # set STEEL_API_KEY and ANTHROPIC_API_KEY
npm install
npm start

Get keys from app.steel.dev and console.anthropic.com. Override the task inline:

TASK="Find the current weather in New York City" npm start

Your output varies. Structure looks like this:

Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to Steel.dev and look for the latest news.
computer({"action":"screenshot"})
computer({"action":"left_click","coordinate":[640,48]})
computer({"action":"type","text":"https://steel.dev"})
computer({"action":"key","text":"Return"})
...
Task complete - no further actions requested

TASK EXECUTION COMPLETED
Duration: 84.3 seconds
Result: Steel's latest news includes ...

Expect ~60-120 seconds and 15-40 iterations for a simple browsing task.

Make it yours

Change the viewport. viewportWidth and viewportHeight in the Agent constructor set both the Steel session dimensions and the tool definition's display_width_px/display_height_px. Keep them in sync.
Tune the system prompt. BROWSER_SYSTEM_PROMPT is where the browsing conventions live: date injection, screenshot-after-submit rule, black-screen recovery.
Raise the ceiling. Long tasks bump against the 50-iteration default in executeTask.
Hand off auth. Pair this recipe with Steel's credentials or auth contexts to start the session authenticated.

Computer use docs · Python version · Mobile variant

examples/claude-computer-use-py

Hussien Hussien, Jun RyuUpdated Apr 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

Computer use is Anthropic's primitive for giving Claude direct control of a screen. You declare a computer tool with a viewport size; Claude replies with actions like left_click at (x, y), type with text, scroll, key. You execute each one and hand back a screenshot.

Steel supplies the screen. A Steel session is a headful Chromium in a VM reachable over HTTPS, and the Input API (sessions.computer) executes mouse and keyboard actions and returns a PNG in the same call.

The loop

Everything in main.py hangs off a single loop in Agent.execute_task. Seed the conversation with a system prompt and the task, then on each turn:

response = self.client.beta.messages.create(
    model=self.model,
    max_tokens=4096,
    messages=self.messages,
    tools=self.tools,
    betas=["computer-use-2025-11-24"],
)

text, has_actions = self.process_response(response)

if not has_actions:
    break

tools declares the computer tool Claude is allowed to call:

self.tools = [
    {
        "type": "computer_20251124",
        "name": "computer",
        "display_width_px": self.viewport_width,
        "display_height_px": self.viewport_height,
        "display_number": 1,
    }
]

The viewport (1280x768) has to match what Steel renders or clicks land in the wrong place.

tool_use blocks go to execute_computer_action, which maps each Anthropic action name onto a Steel Input API call:

elif action in ("left_click", "right_click", "middle_click",
                "double_click", "triple_click"):
    body = {
        "action": "click_mouse",
        "button": button_map[action],
        "coordinates": [coords[0], coords[1]],
        "screenshot": True,
    }

screenshot: True tells Steel to attach a base64 PNG to the response, so a click and the screenshot that proves it landed are one round-trip. The PNG goes back into messages as a tool_result with the matching tool_use_id.

Two normalization details: key / hold_key run names like CTRL+A through normalize_key (CTRL to Control, ESC to Escape, UP to ArrowUp), and scroll_amount is multiplied by 100 pixels per step.

Two things end the loop: Claude responds with only text (task done), or the last two assistant messages overlap 80%+ on word content (detect_repetition). A hard cap of 50 iterations catches anything that slips past both.

Run it

cd examples/claude-computer-use-py
cp .env.example .env          # set STEEL_API_KEY and ANTHROPIC_API_KEY
uv run main.py

Get keys from app.steel.dev and console.anthropic.com. Default task lives in .env as TASK; you can override per-run:

TASK="Find the current weather in New York City" python main.py

Your output varies. Structure looks like this:

Starting Steel session...
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34…

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to Steel.dev and look for the latest news.
computer({"action": "key", "text": "ctrl+l"})
computer({"action": "type", "text": "https://steel.dev"})
computer({"action": "key", "text": "Return"})
computer({"action": "screenshot"})
…
Task complete - no further actions requested

TASK EXECUTION COMPLETED
Duration: 74.3 seconds
Result: Steel just shipped …

Releasing Steel session...

A run typically takes 60-180 seconds and 10-30 loop iterations.

Make it yours

Change the task. Edit TASK in .env or pass it per-run.
Tune the viewport. viewport_width / viewport_height in Agent.__init__.
Rework the system prompt. BROWSER_SYSTEM_PROMPT is where site-specific knowledge lives.
Persist a login. Pass session_context to sessions.create to resume with cookies and local storage. See credentials.
Raise the ceiling. max_iterations=50 in execute_task is the safety net.

Anthropic computer use docs · TypeScript version

examples/claude-computer-use-rs

Jun RyuUpdated Jun 23, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

There is no first-party Anthropic SDK for Rust, so the Messages API here is exactly what it is on the wire: one POST https://api.anthropic.com/v1/messages with reqwest, three headers, and a JSON body you assemble yourself. That turns out to be an advantage for computer use. The request body is dynamic (a growing transcript of text, tool_use, and screenshot tool_result blocks), so you build it with serde_json::json!; the response shape is fixed, so you decode it into a typed enum. The half that benefits from types gets them, the half that does not stays loose.

The other half of the loop is the browser. A Steel session is a headful Chromium in a VM, and client.sessions().computer(&id, action) runs one mouse or keyboard action server-side and returns a base64 PNG in the same call. The steel crate models the action set as a SessionComputerParams enum, so the actions you send Steel are fully typed even though the actions you receive from Claude arrive as untyped JSON.

Two type boundaries

This recipe straddles two APIs with opposite typing stories, and main.rs leans into both.

Claude's reply decodes into an internally tagged enum on the block's type field:

#[derive(Debug, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
enum ContentBlock {
    Text { text: String },
    ToolUse { id: String, name: String, input: Value },
    #[serde(other)]
    Other,
}

input stays a serde_json::Value on purpose: it is the computer tool's arguments (action, coordinate, text, ...), and those vary per action. The #[serde(other)] arm means a new block type in a future API version deserializes instead of panicking.

Going the other direction, execute_computer_action reads that loose input and constructs a typed Steel action. Claude's vocabulary (left_click, type, scroll, key) does not match Steel's (click_mouse, type_text, scroll, press_key), so the function is the translation layer:

"left_click" | "right_click" | "middle_click" | "double_click" | "triple_click" => {
    SessionComputerParams::ClickMouse(ComputerActionRequestClickMouse {
        button: Some(button),
        coordinates: Some(vec![coords.0, coords.1]),
        num_clicks,
        screenshot: Some(true),
        ..
    })
}

screenshot: Some(true) tells Steel to attach a fresh PNG to the action's response, so the click and the screenshot that proves it landed are a single round-trip. That PNG goes straight back into the next tool_result as a base64 image source.

Two translation details worth knowing. Keys run through normalize_key before they reach Steel (CTRL to Control, ESC to Escape, UP to ArrowUp), and scroll_amount is converted to a pixel delta at 100px per step, with direction mapped onto delta_x / delta_y. Both mirror the Python recipe so behavior stays identical across languages.

The loop

Agent::execute_task seeds the transcript with the system prompt and the task, then repeats: call Anthropic, run any actions, append results.

let response = self.call_anthropic().await?;
let (text, has_actions) = self.process_response(response).await?;

if !has_actions {
    println!("Task complete - no further actions requested");
    final_text = text;
    break;
}

The tool definition declares computer_20251124 with display_width_px and display_height_px. Those must match the Steel session's dimensions (1280x768 here) or Claude's coordinates point at the wrong pixels. Both read from the same VIEWPORT_WIDTH / VIEWPORT_HEIGHT constants so they cannot drift.

Three things end the loop: Claude replies with text and no tool_use (done), the last assistant message overlaps a recent one by more than 80% on word content (detect_repetition, a cheap stall guard), or the hard MAX_ITERATIONS cap of 50 trips. The beta is opt-in per request through the anthropic-beta: computer-use-2025-11-24 header in call_anthropic.

Run it

cd examples/claude-computer-use-rs
cp .env.example .env          # set STEEL_API_KEY and ANTHROPIC_API_KEY
cargo run

Get keys from app.steel.dev and console.anthropic.com. The default TASK lives in .env; override it per run:

TASK="Find the current weather in New York City" cargo run

Your output varies. Structure looks like this:

Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to Steel.dev and look for the latest news.
computer({"action":"key","text":"ctrl+l"})
computer({"action":"type","text":"https://steel.dev"})
computer({"action":"key","text":"Enter"})
computer({"action":"screenshot"})
...
Task complete - no further actions requested

TASK EXECUTION COMPLETED
Duration: 78.4 seconds
Result: Steel's latest news includes ...

Releasing Steel session...

Expect 60 to 180 seconds and 10 to 30 iterations for a simple browse, plus Anthropic token cost. A run also spends a few cents of browser time. Steel bills per session-minute, so the cleanup call that releases the session is not optional: main runs the task inside an async block and calls agent.cleanup().await afterward whether it returned Ok or an error, so a failed task still frees the browser.

Make it yours

Change the task. Edit TASK in .env or pass it inline.
Tune the viewport. VIEWPORT_WIDTH / VIEWPORT_HEIGHT feed both the Steel dimensions and the tool definition. Keep them together.
Rework the prompt. browser_system_prompt holds the browsing conventions: date injection, the clear-then-type rule, black-screen recovery.
Raise the ceiling. MAX_ITERATIONS is the safety net for long tasks.
Persist a login. Pass a session context to sessions().create to resume with cookies and local storage. See credentials.

Anthropic computer use docs · Python version · Go version · scrape-rs

examples/claude-computer-use-go

Jun RyuUpdated Jun 23, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

Two typed unions meet in this recipe. Claude's Beta Messages API returns a computer tool call (left_click at [640, 412], type "claude opus", scroll down 3); Steel's Sessions Computer endpoint accepts a discriminated union of actions (click_mouse, type_text, scroll) and returns a screenshot. main.go is the agent loop that translates one into the other and feeds the screenshot back, using the official anthropic-sdk-go and steel-go SDKs end to end with no hand-rolled HTTP.

A Steel session is a headful Chromium in a VM. The Computer endpoint (client.Sessions.Computer) runs a mouse or keyboard action server-side and, when you pass Screenshot: true, returns a base64 PNG in the same call. So one round-trip both acts and observes.

Constructing a Steel action

Steel models its action request as a tagged union. In Go that is SessionComputerParams: a discriminator Action plus one pointer field per variant, all marshaled by the SDK based on the tag. You set the string and the matching struct, and leave the rest nil:

req := &steel.ComputerActionRequestClickMouse{
    Action:      "click_mouse",
    Button:      &button,
    Coordinates: &coords,
    Screenshot:  ptr(true),
}
resp, err := a.steelClient.Sessions.Computer(ctx, a.session.ID,
    steel.SessionComputerParams{Action: "click_mouse", ComputerActionRequestClickMouse: req})
img := resp.Base64Image // *string, base64 PNG

executeComputerAction is one big switch over Claude's action names that builds the right variant for each: left_click and friends become a ComputerActionRequestClickMouse (with NumClicks 2 or 3 for double and triple), type becomes ComputerActionRequestTypeText, scroll becomes a ComputerActionRequestScroll with pixel deltas. Two translation details carry over from the Python and TypeScript versions: scroll_amount is multiplied by 100 pixels per step, and key names like CTRL+A run through normalizeKey (CTRL to Control, ESC to Escape, UP to ArrowUp) before they reach press_key.

Most coordinate and key fields on these structs are pointers (*[]float64, *bool), so the ptr generic helper near the top of the file keeps the construction readable.

Reading Claude's turn

The response side is the other union. BetaMessage.Content is a slice of BetaContentBlockUnion; block.AsAny() returns the concrete variant for a type switch:

for _, block := range msg.Content {
    switch v := block.AsAny().(type) {
    case anthropic.BetaTextBlock:
        // narration; print it and echo it back as a text block
    case anthropic.BetaToolUseBlock:
        // v.Input is the action; execute it, return a screenshot
    }
}

BetaToolUseBlock.Input arrives as any. processResponse marshals it to JSON and unmarshals into a small computerAction struct to read action, coordinate, text, and the rest. The same Input value goes straight back into NewBetaToolUseBlock when echoing the assistant turn, so you never reconstruct it field by field.

Screenshots return to Claude as a tool_result whose content is a base64 image, built in screenshotResult. The anthropic-sdk-go ships NewBetaToolResultBlock for text results, but an image result needs the explicit struct: a BetaToolResultBlockParam whose Content holds a BetaImageBlockParam with a BetaBase64ImageSourceParam. The ToolUseID ties the screenshot back to the call that produced it.

The loop

executeTask seeds the history with the system prompt and the task, then on each turn calls the Beta Messages API and processes the response:

resp, err := a.anthropicClient.Beta.Messages.New(ctx, anthropic.BetaMessageNewParams{
    Model:     anthropic.ModelClaudeOpus4_7,
    MaxTokens: 4096,
    Messages:  a.messages,
    Tools:     a.tools,
    Betas:     []string{"computer-use-2025-11-24"},
})

The tool is declared once in NewAgent with anthropic.BetaToolUnionParamOfComputerUseTool20251124(viewportHeight, viewportWidth), which builds the computer_20251124 definition. Keep the 1280x768 viewport in sync with the Steel session's Dimensions or clicks land in the wrong place. Three conditions end the loop: Claude returns only text (task done), the last assistant messages overlap more than 80% by word content (wordOverlap, a cheap stall detector), or the iteration count hits maxIterations (50).

One SDK note worth its own line: anthropic-sdk-go v1.51.1 has no named constant for the computer-use-2025-11-24 beta yet (its newest is computer-use-2025-01-24). Because AnthropicBeta is a string alias, the raw string in Betas is correct and type-checks. Swap in the constant if a later SDK release adds one.

Run it

cd examples/claude-computer-use-go
cp .env.example .env          # set STEEL_API_KEY and ANTHROPIC_API_KEY
go run .

Get keys from app.steel.dev and console.anthropic.com. The default task lives in .env as TASK; override it per run:

TASK="Find the current weather in New York City" go run .

Your output varies. Structure looks like this:

Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to Steel.dev and look for the latest news.
computer({"action":"key","text":"ctrl+l"})
computer({"action":"type","text":"https://steel.dev"})
computer({"action":"key","text":"Return"})
computer({"action":"screenshot"})
...
Task complete - no further actions requested

============================================================
TASK EXECUTION COMPLETED
Duration: 78.4 seconds
Releasing Steel session...

Expect roughly 60 to 180 seconds and 10 to 40 loop iterations for a simple browsing task. A run costs a few cents of browser time plus the Anthropic tokens for each screenshot. Steel bills per session-minute, so the defer agent.cleanup(ctx) in main that releases the session is not optional: skip it and the browser runs until the 900000 ms timeout set in initialize.

Make it yours

Change the task. Edit TASK in .env or pass it inline.
Tune the viewport. viewportWidth and viewportHeight set both the Steel Dimensions and the tool's display_*_px. Keep them equal.
Rework the system prompt. browserSystemPrompt is where the browsing conventions live: date injection, the screenshot-after-submit rule, black-screen recovery.
Raise the ceiling. maxIterations is the safety net for long tasks.
Hand off auth. Pass SessionContext to Sessions.Create to start authenticated. See credentials and auth-context.

Python version · TypeScript version · OpenAI computer use in Go · Anthropic computer use docs

Drive a browser with Gemini Computer Use

Connect Google's Gemini Computer Use to a Steel browser session for autonomous web interactions.

Computer use

Nov 25, 2025

Drive a mobile browser with Claude Computer Use

Claude Computer Use with Steel for autonomous task execution in mobile browser environments.

Computer use Mobile

Oct 14, 2025

Drive a browser with OpenAI Computer Use

Connect OpenAI's Computer Use Assistant to a Steel browser session for autonomous web interactions.

Computer use

Mar 19, 2025