> Full docs index: https://docs.steel.dev/llms.txt # Drive a browser with Claude Computer Use URL: https://docs.steel.dev/cookbook/claude-computer-use **TypeScript** Claude sees the screen as an image and returns concrete actions at pixel coordinates: `left_click [640, 412]`, `type "claude 4.7 opus"`, `scroll down 3`. Something has to execute those actions against a real browser and send the next screenshot back. That "something" is the agent loop in `index.ts`, and the browser is a Steel session. The [Claude Computer Use integration](/integrations/claude-computer-use) covers install and setup. ## The loop The whole thing fits in one `while` block inside `Agent.executeTask`. Each iteration sends the growing message history plus the `computer` tool definition to Claude: ```typescript const response = await this.client.beta.messages.create({ model: this.model, max_tokens: 4096, messages: this.messages, tools: this.tools, betas: ["computer-use-2025-11-24"], }); ``` The tool definition declares `computer_20251124` with the viewport's `display_width_px` and `display_height_px`. Keep it consistent with the Steel session's `dimensions` (1280x768 here) or clicks land in the wrong place. `executeComputerAction` is the translation layer. Claude emits [computer-use actions](/cookbook/topics/computer-use) (`left_click`, `type`, `key`, `scroll`, `screenshot`, ...); Steel's Input API speaks a parallel vocabulary (`click_mouse`, `type_text`, `press_key`, `scroll`, `take_screenshot`): ```typescript case "left_click": case "right_click": case "middle_click": case "double_click": case "triple_click": { body = { action: "click_mouse", button: buttonMap[action], coordinates: coords, screenshot: true, }; break; } ``` Every action sets `screenshot: true`, so Steel returns a fresh base64 PNG after each interaction. That PNG becomes the content of a `tool_result` block in the next user message. A few translation details: - **Keys get normalized.** `normalizeKey` maps synonyms (`CTRL` to `Control`, `CMD` to `Meta`, `ENTER` to `Enter`) before sending to Steel. - **Scroll is delta-based.** Claude says `scroll_direction: "down", scroll_amount: 3`; Steel expects `delta_x`/`delta_y` in pixels. The code multiplies by 100 per step. - **Drags default from center.** `left_click_drag` only gives an end coordinate, so the start is the viewport center. ## Stop conditions - **No tool calls.** Claude wrote only text. Task is complete. - **Repetition.** `detectRepetition` compares the last assistant message against the previous three by word overlap (>80%). - **Iteration cap.** 50 iterations by default. The `finally` block in `main` always calls `agent.cleanup()`, which releases the Steel session. ## Run it ```bash cd examples/claude-computer-use-ts cp .env.example .env # set STEEL_API_KEY and ANTHROPIC_API_KEY npm install npm start ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [console.anthropic.com](https://console.anthropic.com/). Override the task inline: ```bash TASK="Find the current weather in New York City" npm start ``` Your output varies. Structure looks like this: ```text Steel Session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34... Executing task: Go to Steel.dev and find the latest news ============================================================ I'll navigate to Steel.dev and look for the latest news. computer({"action":"screenshot"}) computer({"action":"left_click","coordinate":[640,48]}) computer({"action":"type","text":"https://steel.dev"}) computer({"action":"key","text":"Return"}) ... Task complete - no further actions requested TASK EXECUTION COMPLETED Duration: 84.3 seconds Result: Steel's latest news includes ... ``` Expect ~60-120 seconds and 15-40 iterations for a simple browsing task. ## Make it yours - **Change the viewport.** `viewportWidth` and `viewportHeight` in the `Agent` constructor set both the Steel session dimensions and the tool definition's `display_width_px`/`display_height_px`. Keep them in sync. - **Tune the system prompt.** `BROWSER_SYSTEM_PROMPT` is where the browsing conventions live: date injection, screenshot-after-submit rule, black-screen recovery. - **Raise the ceiling.** Long tasks bump against the 50-iteration default in `executeTask`. - **Hand off auth.** Pair this recipe with Steel's [credentials](/cookbook/credentials) or [auth contexts](/cookbook/auth-context) to start the session authenticated. ## Related [Computer use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) · [Python version](/cookbook/claude-computer-use) · [Mobile variant](/cookbook/claude-computer-use-mobile) **Python** Computer use is Anthropic's primitive for giving Claude direct control of a screen. You declare a `computer` tool with a viewport size; Claude replies with actions like `left_click` at `(x, y)`, `type` with text, `scroll`, `key`. You execute each one and hand back a screenshot. Steel supplies the screen. A Steel session is a headful Chromium in a VM reachable over HTTPS, and the Input API (`sessions.computer`) executes mouse and keyboard actions and returns a PNG in the same call. ## The loop Everything in `main.py` hangs off a single loop in `Agent.execute_task`. Seed the conversation with a system prompt and the task, then on each turn: ```python response = self.client.beta.messages.create( model=self.model, max_tokens=4096, messages=self.messages, tools=self.tools, betas=["computer-use-2025-11-24"], ) text, has_actions = self.process_response(response) if not has_actions: break ``` `tools` declares the computer tool Claude is allowed to call: ```python self.tools = [ { "type": "computer_20251124", "name": "computer", "display_width_px": self.viewport_width, "display_height_px": self.viewport_height, "display_number": 1, } ] ``` The viewport (1280x768) has to match what Steel renders or clicks land in the wrong place. `tool_use` blocks go to `execute_computer_action`, which maps each Anthropic action name onto a Steel Input API call: ```python elif action in ("left_click", "right_click", "middle_click", "double_click", "triple_click"): body = { "action": "click_mouse", "button": button_map[action], "coordinates": [coords[0], coords[1]], "screenshot": True, } ``` `screenshot: True` tells Steel to attach a base64 PNG to the response, so a click and the screenshot that proves it landed are one round-trip. The PNG goes back into `messages` as a `tool_result` with the matching `tool_use_id`. Two normalization details: `key` / `hold_key` run names like `CTRL+A` through `normalize_key` (`CTRL` to `Control`, `ESC` to `Escape`, `UP` to `ArrowUp`), and `scroll_amount` is multiplied by 100 pixels per step. Two things end the loop: Claude responds with only text (task done), or the last two assistant messages overlap 80%+ on word content (`detect_repetition`). A hard cap of 50 iterations catches anything that slips past both. ## Run it ```bash cd examples/claude-computer-use-py cp .env.example .env # set STEEL_API_KEY and ANTHROPIC_API_KEY uv run main.py ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [console.anthropic.com](https://console.anthropic.com/). Default task lives in `.env` as `TASK`; you can override per-run: ```bash TASK="Find the current weather in New York City" python main.py ``` Your output varies. Structure looks like this: ```text Starting Steel session... Steel Session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34… Executing task: Go to Steel.dev and find the latest news ============================================================ I'll navigate to Steel.dev and look for the latest news. computer({"action": "key", "text": "ctrl+l"}) computer({"action": "type", "text": "https://steel.dev"}) computer({"action": "key", "text": "Return"}) computer({"action": "screenshot"}) … Task complete - no further actions requested TASK EXECUTION COMPLETED Duration: 74.3 seconds Result: Steel just shipped … Releasing Steel session... ``` A run typically takes 60-180 seconds and 10-30 loop iterations. ## Make it yours - **Change the task.** Edit `TASK` in `.env` or pass it per-run. - **Tune the viewport.** `viewport_width` / `viewport_height` in `Agent.__init__`. - **Rework the system prompt.** `BROWSER_SYSTEM_PROMPT` is where site-specific knowledge lives. - **Persist a login.** Pass `session_context` to `sessions.create` to resume with cookies and local storage. See [credentials](/cookbook/credentials). - **Raise the ceiling.** `max_iterations=50` in `execute_task` is the safety net. ## Related [Anthropic computer use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) · [TypeScript version](/cookbook/claude-computer-use) **Rust** There is no first-party Anthropic SDK for Rust, so the Messages API here is exactly what it is on the wire: one `POST https://api.anthropic.com/v1/messages` with `reqwest`, three headers, and a JSON body you assemble yourself. That turns out to be an advantage for computer use. The request body is dynamic (a growing transcript of text, `tool_use`, and screenshot `tool_result` blocks), so you build it with `serde_json::json!`; the response shape is fixed, so you decode it into a typed `enum`. The half that benefits from types gets them, the half that does not stays loose. The other half of the loop is the browser. A Steel session is a headful Chromium in a VM, and `client.sessions().computer(&id, action)` runs one mouse or keyboard action server-side and returns a base64 PNG in the same call. The `steel` crate models the action set as a `SessionComputerParams` enum, so the actions you send Steel are fully typed even though the actions you receive from Claude arrive as untyped JSON. ## Two type boundaries This recipe straddles two APIs with opposite typing stories, and `main.rs` leans into both. Claude's reply decodes into an internally tagged enum on the block's `type` field: ```rust #[derive(Debug, Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] enum ContentBlock { Text { text: String }, ToolUse { id: String, name: String, input: Value }, #[serde(other)] Other, } ``` `input` stays a `serde_json::Value` on purpose: it is the computer tool's arguments (`action`, `coordinate`, `text`, ...), and those vary per action. The `#[serde(other)]` arm means a new block type in a future API version deserializes instead of panicking. Going the other direction, `execute_computer_action` reads that loose `input` and constructs a typed Steel action. Claude's vocabulary (`left_click`, `type`, `scroll`, `key`) does not match Steel's (`click_mouse`, `type_text`, `scroll`, `press_key`), so the function is the translation layer: ```rust "left_click" | "right_click" | "middle_click" | "double_click" | "triple_click" => { SessionComputerParams::ClickMouse(ComputerActionRequestClickMouse { button: Some(button), coordinates: Some(vec![coords.0, coords.1]), num_clicks, screenshot: Some(true), .. }) } ``` `screenshot: Some(true)` tells Steel to attach a fresh PNG to the action's response, so the click and the screenshot that proves it landed are a single round-trip. That PNG goes straight back into the next `tool_result` as a base64 `image` source. Two translation details worth knowing. Keys run through `normalize_key` before they reach Steel (`CTRL` to `Control`, `ESC` to `Escape`, `UP` to `ArrowUp`), and `scroll_amount` is converted to a pixel delta at 100px per step, with direction mapped onto `delta_x` / `delta_y`. Both mirror the Python recipe so behavior stays identical across languages. ## The loop `Agent::execute_task` seeds the transcript with the system prompt and the task, then repeats: call Anthropic, run any actions, append results. ```rust let response = self.call_anthropic().await?; let (text, has_actions) = self.process_response(response).await?; if !has_actions { println!("Task complete - no further actions requested"); final_text = text; break; } ``` The tool definition declares `computer_20251124` with `display_width_px` and `display_height_px`. Those must match the Steel session's `dimensions` (1280x768 here) or Claude's coordinates point at the wrong pixels. Both read from the same `VIEWPORT_WIDTH` / `VIEWPORT_HEIGHT` constants so they cannot drift. Three things end the loop: Claude replies with text and no `tool_use` (done), the last assistant message overlaps a recent one by more than 80% on word content (`detect_repetition`, a cheap stall guard), or the hard `MAX_ITERATIONS` cap of 50 trips. The beta is opt-in per request through the `anthropic-beta: computer-use-2025-11-24` header in `call_anthropic`. ## Run it ```bash cd examples/claude-computer-use-rs cp .env.example .env # set STEEL_API_KEY and ANTHROPIC_API_KEY cargo run ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [console.anthropic.com](https://console.anthropic.com/). The default `TASK` lives in `.env`; override it per run: ```bash TASK="Find the current weather in New York City" cargo run ``` Your output varies. Structure looks like this: ```text Steel Session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34... Executing task: Go to Steel.dev and find the latest news ============================================================ I'll navigate to Steel.dev and look for the latest news. computer({"action":"key","text":"ctrl+l"}) computer({"action":"type","text":"https://steel.dev"}) computer({"action":"key","text":"Enter"}) computer({"action":"screenshot"}) ... Task complete - no further actions requested TASK EXECUTION COMPLETED Duration: 78.4 seconds Result: Steel's latest news includes ... Releasing Steel session... ``` Expect 60 to 180 seconds and 10 to 30 iterations for a simple browse, plus Anthropic token cost. A run also spends a few cents of browser time. Steel bills per session-minute, so the `cleanup` call that releases the session is not optional: `main` runs the task inside an `async` block and calls `agent.cleanup().await` afterward whether it returned `Ok` or an error, so a failed task still frees the browser. ## Make it yours - **Change the task.** Edit `TASK` in `.env` or pass it inline. - **Tune the viewport.** `VIEWPORT_WIDTH` / `VIEWPORT_HEIGHT` feed both the Steel `dimensions` and the tool definition. Keep them together. - **Rework the prompt.** `browser_system_prompt` holds the browsing conventions: date injection, the clear-then-type rule, black-screen recovery. - **Raise the ceiling.** `MAX_ITERATIONS` is the safety net for long tasks. - **Persist a login.** Pass a session context to `sessions().create` to resume with cookies and local storage. See [credentials](/cookbook/credentials). ## Related [Anthropic computer use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) · [Python version](/cookbook/claude-computer-use) · [Go version](/cookbook/claude-computer-use) · [scrape-rs](/cookbook/scrape) **Go** Two typed unions meet in this recipe. Claude's Beta Messages API returns a `computer` tool call (`left_click` at `[640, 412]`, `type "claude opus"`, `scroll down 3`); Steel's Sessions Computer endpoint accepts a discriminated union of actions (`click_mouse`, `type_text`, `scroll`) and returns a screenshot. `main.go` is the agent loop that translates one into the other and feeds the screenshot back, using the official `anthropic-sdk-go` and `steel-go` SDKs end to end with no hand-rolled HTTP. A Steel session is a headful Chromium in a VM. The Computer endpoint (`client.Sessions.Computer`) runs a mouse or keyboard action server-side and, when you pass `Screenshot: true`, returns a base64 PNG in the same call. So one round-trip both acts and observes. ## Constructing a Steel action Steel models its action request as a tagged union. In Go that is `SessionComputerParams`: a discriminator `Action` plus one pointer field per variant, all marshaled by the SDK based on the tag. You set the string and the matching struct, and leave the rest nil: ```go req := &steel.ComputerActionRequestClickMouse{ Action: "click_mouse", Button: &button, Coordinates: &coords, Screenshot: ptr(true), } resp, err := a.steelClient.Sessions.Computer(ctx, a.session.ID, steel.SessionComputerParams{Action: "click_mouse", ComputerActionRequestClickMouse: req}) img := resp.Base64Image // *string, base64 PNG ``` `executeComputerAction` is one big `switch` over Claude's action names that builds the right variant for each: `left_click` and friends become a `ComputerActionRequestClickMouse` (with `NumClicks` 2 or 3 for double and triple), `type` becomes `ComputerActionRequestTypeText`, `scroll` becomes a `ComputerActionRequestScroll` with pixel deltas. Two translation details carry over from the Python and TypeScript versions: `scroll_amount` is multiplied by 100 pixels per step, and key names like `CTRL+A` run through `normalizeKey` (`CTRL` to `Control`, `ESC` to `Escape`, `UP` to `ArrowUp`) before they reach `press_key`. Most coordinate and key fields on these structs are pointers (`*[]float64`, `*bool`), so the `ptr` generic helper near the top of the file keeps the construction readable. ## Reading Claude's turn The response side is the other union. `BetaMessage.Content` is a slice of `BetaContentBlockUnion`; `block.AsAny()` returns the concrete variant for a type switch: ```go for _, block := range msg.Content { switch v := block.AsAny().(type) { case anthropic.BetaTextBlock: // narration; print it and echo it back as a text block case anthropic.BetaToolUseBlock: // v.Input is the action; execute it, return a screenshot } } ``` `BetaToolUseBlock.Input` arrives as `any`. `processResponse` marshals it to JSON and unmarshals into a small `computerAction` struct to read `action`, `coordinate`, `text`, and the rest. The same `Input` value goes straight back into `NewBetaToolUseBlock` when echoing the assistant turn, so you never reconstruct it field by field. Screenshots return to Claude as a `tool_result` whose content is a base64 image, built in `screenshotResult`. The `anthropic-sdk-go` ships `NewBetaToolResultBlock` for text results, but an image result needs the explicit struct: a `BetaToolResultBlockParam` whose `Content` holds a `BetaImageBlockParam` with a `BetaBase64ImageSourceParam`. The `ToolUseID` ties the screenshot back to the call that produced it. ## The loop `executeTask` seeds the history with the system prompt and the task, then on each turn calls the Beta Messages API and processes the response: ```go resp, err := a.anthropicClient.Beta.Messages.New(ctx, anthropic.BetaMessageNewParams{ Model: anthropic.ModelClaudeOpus4_7, MaxTokens: 4096, Messages: a.messages, Tools: a.tools, Betas: []string{"computer-use-2025-11-24"}, }) ``` The tool is declared once in `NewAgent` with `anthropic.BetaToolUnionParamOfComputerUseTool20251124(viewportHeight, viewportWidth)`, which builds the `computer_20251124` definition. Keep the 1280x768 viewport in sync with the Steel session's `Dimensions` or clicks land in the wrong place. Three conditions end the loop: Claude returns only text (task done), the last assistant messages overlap more than 80% by word content (`wordOverlap`, a cheap stall detector), or the iteration count hits `maxIterations` (50). One SDK note worth its own line: `anthropic-sdk-go` v1.51.1 has no named constant for the `computer-use-2025-11-24` beta yet (its newest is `computer-use-2025-01-24`). Because `AnthropicBeta` is a string alias, the raw string in `Betas` is correct and type-checks. Swap in the constant if a later SDK release adds one. ## Run it ```bash cd examples/claude-computer-use-go cp .env.example .env # set STEEL_API_KEY and ANTHROPIC_API_KEY go run . ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [console.anthropic.com](https://console.anthropic.com/). The default task lives in `.env` as `TASK`; override it per run: ```bash TASK="Find the current weather in New York City" go run . ``` Your output varies. Structure looks like this: ```text Steel Session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34... Executing task: Go to Steel.dev and find the latest news ============================================================ I'll navigate to Steel.dev and look for the latest news. computer({"action":"key","text":"ctrl+l"}) computer({"action":"type","text":"https://steel.dev"}) computer({"action":"key","text":"Return"}) computer({"action":"screenshot"}) ... Task complete - no further actions requested ============================================================ TASK EXECUTION COMPLETED Duration: 78.4 seconds Releasing Steel session... ``` Expect roughly 60 to 180 seconds and 10 to 40 loop iterations for a simple browsing task. A run costs a few cents of browser time plus the Anthropic tokens for each screenshot. Steel bills per session-minute, so the `defer agent.cleanup(ctx)` in `main` that releases the session is not optional: skip it and the browser runs until the 900000 ms timeout set in `initialize`. ## Make it yours - **Change the task.** Edit `TASK` in `.env` or pass it inline. - **Tune the viewport.** `viewportWidth` and `viewportHeight` set both the Steel `Dimensions` and the tool's `display_*_px`. Keep them equal. - **Rework the system prompt.** `browserSystemPrompt` is where the browsing conventions live: date injection, the screenshot-after-submit rule, black-screen recovery. - **Raise the ceiling.** `maxIterations` is the safety net for long tasks. - **Hand off auth.** Pass `SessionContext` to `Sessions.Create` to start authenticated. See [credentials](/cookbook/credentials) and [auth-context](/cookbook/auth-context). ## Related [Python version](/cookbook/claude-computer-use) · [TypeScript version](/cookbook/claude-computer-use) · [OpenAI computer use in Go](/cookbook/openai-computer-use) · [Anthropic computer use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) ## Related recipes - [Drive a browser with Gemini Computer Use](/cookbook/gemini-computer-use): Connect Google's Gemini Computer Use to a Steel browser session for autonomous web interactions. - [Drive a mobile browser with Claude Computer Use](/cookbook/claude-computer-use-mobile): Claude Computer Use with Steel for autonomous task execution in mobile browser environments. - [Drive a browser with OpenAI Computer Use](/cookbook/openai-computer-use): Connect OpenAI's Computer Use Assistant to a Steel browser session for autonomous web interactions.