> Full docs index: https://docs.steel.dev/llms.txt # Drive a browser with OpenAI Computer Use URL: https://docs.steel.dev/cookbook/openai-computer-use **TypeScript** OpenAI's [computer-use](/cookbook/topics/computer-use) model ships as a single tool declaration: `{ type: "computer" }`. You hand it to the Responses API, send a screenshot, and the model returns `computer_call` items with actions like `click`, `type`, `keypress`, `scroll`. Your job is to execute them against a real browser, capture the next screenshot, and feed it back. Steel's [OpenAI Computer Use integration](/integrations/openai-computer-use) covers the same loop as a guide. ## The loop The Responses API threads conversation state server-side, so each turn carries only the new tool outputs plus a `previous_response_id`: ```typescript const response = await createResponse({ model: this.model, instructions: this.systemPrompt, input: nextInput, tools: this.tools, previous_response_id: previousResponseId, reasoning: { effort: "medium" }, truncation: "auto", }); ``` First turn: `nextInput` is `[{ role: "user", content: task }]`. Subsequent turns: `nextInput` is just the array of tool outputs from the previous iteration. The response's `output` array mixes three item types: - `message`: a plain text reply. The final message becomes the return value. - `reasoning`: the model's own summary; printed but not fed back. - `computer_call`: one or more actions to execute. Each call has a `call_id` that must be echoed back in the matching `computer_call_output`. ## Actions in, screenshots out `executeComputerAction` is the translation layer: ```typescript case "click": { const coords = this.toCoords(actionArgs.x, actionArgs.y); const button = this.mapButton(actionArgs.button); const clicks = this.toNumber(actionArgs.num_clicks, 1); body = { action: "click_mouse", button, coordinates: coords, ...(clicks > 1 ? { num_clicks: clicks } : {}), screenshot: true, }; break; } ``` The screenshot goes back as a `computer_call_output`, matched by `call_id`: ```typescript toolOutputs.push({ type: "computer_call_output", call_id: item.call_id, acknowledged_safety_checks: pendingChecks, output: { type: "computer_screenshot", image_url: `data:image/png;base64,${screenshotBase64}`, }, }); ``` A few translation details: - **`keypress` takes a list.** `normalizeKey` rewrites synonyms (`CTRL` to `Control`, `CMD` to `Meta`, `ENTER` to `Enter`). - **`scroll` is delta-based.** OpenAI sends `scroll_x`/`scroll_y` in pixels. Steel's `scroll` takes `delta_x`/`delta_y` directly. - **`drag` gives a path.** OpenAI provides the full point list in `path`; Steel's `drag_mouse` wants the same shape. - **Unknown actions fall through to `take_screenshot`.** ## Safety checks A `computer_call` can attach `pending_safety_checks` when the planned action looks sensitive. The call won't take effect until you echo those check IDs back in `acknowledged_safety_checks`. The starter auto-acknowledges everything; for production, flip `autoAcknowledgeSafety` to `false` and gate each check on a human approval. ## Run it ```bash cd examples/openai-computer-use-ts cp .env.example .env # set STEEL_API_KEY and OPENAI_API_KEY npm install npm start ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [platform.openai.com](https://platform.openai.com/api-keys). Override the task inline: ```bash TASK="Find the current weather in New York City" npm start ``` Your output varies. Structure looks like this: ```text Steel Session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34... Executing task: Go to Steel.dev and find the latest news ============================================================ I'll navigate to steel.dev and scan the landing page. click({"x":720,"y":48}) type({"text":"https://steel.dev"}) keypress({"keys":["Enter"]}) scroll({"x":720,"y":450,"scroll_y":600}) Steel's latest release adds ... ============================================================ TASK EXECUTION COMPLETED ============================================================ Duration: 71.4 seconds ``` Expect roughly 60-120 seconds and 15-40 turns for a simple browsing task. ## Make it yours - **Change the viewport.** `viewportWidth` and `viewportHeight` in the `Agent` constructor set the Steel session `dimensions`. - **Swap the model.** The default is `gpt-5.5`. Update `this.model` in the `Agent` constructor. - **Tune reasoning effort.** `reasoning: { effort: "medium" }` trades latency for planning quality. - **Rewrite the system prompt.** `BROWSER_SYSTEM_PROMPT` holds the browsing conventions. - **Persist a login.** Pass `sessionContext` to `sessions.create`. See [credentials](/cookbook/credentials) and [auth-context](/cookbook/auth-context). - **Turn off auto-ack.** Flip `autoAcknowledgeSafety` to `false` to make pending safety checks raise. ## Related [Computer use guide](https://platform.openai.com/docs/guides/tools-computer-use) · [Python version](/cookbook/openai-computer-use) · [Anthropic equivalent](/cookbook/claude-computer-use) **Python** OpenAI's Computer Use models expose one tool (`{"type": "computer"}`) and emit `computer_call` items containing an `action` the model wants performed on a screen. You execute the action, return a screenshot as a `computer_call_output`, and the next turn the model sees the result. The action vocabulary (`click`, `type`, `keypress`, `scroll`, `drag`, `wait`, `screenshot`) is fixed by OpenAI. This recipe uses OpenAI's **Responses API**, not Chat Completions. Responses keeps conversation state on OpenAI's side via `previous_response_id`, so each turn only sends the new tool outputs rather than the full screenshot history. ## The loop ```python params = { "model": self.model, "instructions": self.system_prompt, "input": next_input, "tools": self.tools, "reasoning": {"effort": "medium"}, "truncation": "auto", } if previous_response_id: params["previous_response_id"] = previous_response_id response = create_response(**params) previous_response_id = response.get("id") ``` Each `response["output"]` is a list of items with a `type`. The loop walks them: - `reasoning`: model's internal thinking, printed. - `message`: terminal prose; the agent stores the last one as the final result. - `computer_call`: one or more actions to execute. `execute_computer_action` maps OpenAI's action vocabulary onto Steel's Input API. Each branch builds a Steel request body and sends it through `self.steel.sessions.computer(...)` with `screenshot: True`: ```python elif action_type in ("click",): coords = self.to_coords(action_args.get("x"), action_args.get("y")) button = self.map_button(action_args.get("button")) num_clicks = int(self.to_number(action_args.get("num_clicks"), 1)) payload = { "action": "click_mouse", "button": button, "coordinates": [coords[0], coords[1]], "screenshot": True, } if num_clicks > 1: payload["num_clicks"] = num_clicks body = payload ``` `keypress` arrives with OpenAI names (`CTRL`, `ENTER`, `ESC`, `UP`); `normalize_key` rewrites them into the Steel / DOM vocabulary (`Control`, `Enter`, `Escape`, `ArrowUp`). The screenshot goes back as a `computer_call_output`: ```python tool_outputs.append({ "type": "computer_call_output", "call_id": item["call_id"], "acknowledged_safety_checks": pending_checks, "output": { "type": "computer_screenshot", "image_url": f"data:image/png;base64,{screenshot_base64}", }, }) ``` ## Safety checks A `computer_call` can include `pending_safety_checks`. You must echo them back in `acknowledged_safety_checks` on the next turn, or the model stalls. The default here is `auto_acknowledge_safety = True`, which suits a starter but is not what you want in production. Flip it to `False` and surface the check to a human before proceeding. ## Run it ```bash cd examples/openai-computer-use-py cp .env.example .env # set STEEL_API_KEY and OPENAI_API_KEY uv run main.py ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [platform.openai.com](https://platform.openai.com/api-keys). Override the task per run: ```bash TASK="Find the current weather in New York City" python main.py ``` Your output varies. Structure looks like this: ```text Starting Steel session... Steel Session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34… Executing task: Go to Steel.dev and find the latest news ============================================================ I'll open steel.dev and check the blog. keypress({"keys": ["CTRL", "L"]}) type({"text": "https://steel.dev"}) keypress({"keys": ["ENTER"]}) wait({"ms": 1500}) … Steel's latest release notes mention … TASK EXECUTION COMPLETED Duration: 62.8 seconds ``` A run typically takes 60-180 seconds and 10-30 iterations. Screenshots are cached between turns via `previous_response_id`, so per-turn input cost stays roughly flat even on long loops. The `finally` block in `main()` calls `sessions.release()`. ## Make it yours - **Change the task.** Edit `TASK` in `.env` or pass it inline. - **Swap the model.** The default is `gpt-5.5`. Update `self.model` in `Agent.__init__`. - **Tune the viewport.** `viewport_width` / `viewport_height` in `Agent.__init__` flow into `sessions.create(dimensions=...)`. - **Turn off auto-ack.** Flip `auto_acknowledge_safety = False` to make pending safety checks raise. - **Persist a login.** Pass `session_context` to `sessions.create`. See [credentials](/cookbook/credentials). - **Adjust reasoning.** `"effort": "medium"` trades latency for deeper plans. Drop to `"low"` for fast lookups, raise to `"high"` for multi-step research. ## Related [TypeScript version](/cookbook/openai-computer-use) · [Claude version](/cookbook/claude-computer-use) · [OpenAI Computer Use guide](https://platform.openai.com/docs/guides/tools-computer-use) · [Responses API reference](https://platform.openai.com/docs/api-reference/responses) **Rust** OpenAI's `computer-use-preview` model returns mouse and keyboard actions instead of text. This recipe executes those actions against a real Chromium running in Steel's cloud, feeds each resulting screenshot back, and loops until the model reports the task done. There is no official OpenAI Rust SDK, so it calls the Responses API directly with `reqwest` and drives the browser through Steel's server-side `computer` endpoint. It is the Rust counterpart to [openai-computer-use-py](/cookbook/openai-computer-use) and the OpenAI sibling of [claude-computer-use-rs](/cookbook/claude-computer-use). ## The loop The Responses API is stateful. Each turn you pass the previous `response.id` as `previous_response_id` and send only the new input, so the conversation never gets resent: ```rust let mut input = json!([{ "role": "user", "content": task }]); let mut previous_response_id: Option = None; for _ in 0..MAX_ITERATIONS { let response = self.call_openai(&input, &previous_response_id).await?; previous_response_id = Some(response.id); let mut next_input = Vec::new(); for item in &response.output { // message -> print it, reasoning -> print it, // computer_call -> run the action, screenshot, push a computer_call_output } if next_input.is_empty() { break; } // model returned only text: done input = Value::Array(next_input); } ``` Contrast [claude-computer-use-rs](/cookbook/claude-computer-use), where the Anthropic Messages API is stateless and you grow and resend a `messages` array every turn. Here the server holds the history and you send back only `computer_call_output` items. `MAX_ITERATIONS` caps the loop so a stuck model cannot run forever. ## From OpenAI action to Steel action A `computer_call` carries one `action` such as `{ "type": "click", "button": "left", "x": 412, "y": 280 }`. `execute_action` matches on `type` and builds the matching Steel `SessionComputerParams`: ```rust "click" => SessionComputerParams::ClickMouse(ComputerActionRequestClickMouse { button: Some(map_button(/* left | right | middle | back | forward */)), coordinates: Some(vec![x, y]), screenshot: Some(true), .. }), "type" => SessionComputerParams::TypeText(/* text */), "keypress" => SessionComputerParams::PressKey(/* normalized keys */), "scroll" => SessionComputerParams::Scroll(/* delta_x, delta_y */), ``` Every action sets `screenshot: true`, so Steel returns a fresh base64 PNG. That image goes back as a `computer_call_output` with `image_url: data:image/png;base64,...`, which is what the model sees for its next move. OpenAI key names (`ENTER`, `CTRL`, `ESC`) are normalized to the DOM vocabulary Steel expects (`Enter`, `Control`, `Escape`). When a turn includes a `pending_safety_check`, the recipe auto-acknowledges it by echoing it back in `acknowledged_safety_checks`. That is fine for a demo on a throwaway page. Read each check before letting an agent act on a real account. ## Run it ```bash cd examples/openai-computer-use-rs cp .env.example .env # set STEEL_API_KEY and OPENAI_API_KEY cargo run ``` Get a Steel key at [app.steel.dev/settings/api-keys](https://app.steel.dev/settings/api-keys) and an OpenAI key at [platform.openai.com/api-keys](https://platform.openai.com/api-keys). Set `TASK` in `.env` to change the goal. Your output varies. Structure looks like this: ```text Steel + OpenAI Computer Use Assistant ============================================================ Starting Steel session... View live session at: https://app.steel.dev/sessions/3f2a... Executing task: Go to Steel.dev and find the latest news ============================================================ click(button=left x=720 y=400) type(text="steel.dev blog") keypress(keys=["Enter"]) ... The latest Steel post is "...". ============================================================ TASK EXECUTION COMPLETED Duration: 48.2 seconds ``` A run drives a real session and a vision model across many turns, so it costs a few cents of browser time plus the OpenAI tokens for the loop. Steel bills per session-minute until `cleanup` releases the session, which always runs through a deferred release, even on error. ## Make it yours - **Change the task.** Set `TASK` in `.env`, or edit the default in `main`. - **Resize the viewport.** `VIEWPORT_WIDTH` and `VIEWPORT_HEIGHT` set both the Steel session dimensions and the `display_width`/`display_height` on the tool. Keep them in sync so the model's coordinates match the page. - **Gate safety checks.** Instead of auto-acknowledging every `pending_safety_check`, prompt a human or allowlist specific codes before echoing them back. - **Start authenticated.** Pass a session context or credentials to `sessions().create(...)` so the agent begins on a logged-in page. See [auth-context](/cookbook/auth-context) and [credentials](/cookbook/credentials). ## Related [openai-computer-use-go](/cookbook/openai-computer-use) is the same agent through the official `openai-go` SDK, which has a typed Responses API. [openai-computer-use-py](/cookbook/openai-computer-use) and [openai-computer-use-ts](/cookbook/openai-computer-use) are the Python and TypeScript versions. [claude-computer-use-rs](/cookbook/claude-computer-use) runs the same Steel action loop against Anthropic instead. **Go** This recipe wires two typed Go SDKs together so OpenAI's `computer-use-preview` model can drive a Steel cloud browser. The model emits a computer action through the official `github.com/openai/openai-go/v3` Responses API. You execute that action against a Steel session with `client.Sessions.Computer`, then hand the resulting screenshot back so the next turn sees what changed. That single exchange, repeated, is the whole agent. The interesting part in Go is the seam between the two SDKs, because each models the action vocabulary differently. openai-go gives you a *flattened* union: `ResponseComputerToolCallActionUnion` carries every possible field (`X`, `Y`, `Button`, `Keys`, `ScrollX`, `Text`, `Path`) on one struct, and you read whichever ones the `Type` discriminator says are live. Steel's Go SDK takes the opposite shape: `SessionComputerParams` is a *constructed* discriminated union where you set an `Action` string and attach the one matching `ComputerActionRequest*` pointer. `executeAction` is the translation layer between the two. ```go case "click": body := &steel.ComputerActionRequestClickMouse{ Action: steel.ComputerActionRequestClickMouseActionClickMouse, Button: ptr(mapButton(act.Button)), Coordinates: coords(), Screenshot: ptr(true), } return a.run(ctx, steel.SessionComputerParams{Action: "click_mouse", ComputerActionRequestClickMouse: body}) ``` Every branch sets `Screenshot: ptr(true)`, so the Steel call that performs the action also returns the screenshot in the same round trip. `run` reads `resp.Base64Image` and falls back to an explicit `take_screenshot` if a particular action did not capture one. ## Responses keeps the conversation, you keep the loop The Responses API stores conversation state server side. The first turn sends the task as a user message; every later turn sends only the new `computer_call_output` items and threads `PreviousResponseID` from the prior response. You never resend the screenshot history, so input size stays roughly flat even across a long run. ```go params := responses.ResponseNewParams{ Model: shared.ResponsesModelComputerUsePreview, Instructions: openai.String(systemPrompt()), Input: responses.ResponseNewParamsInputUnion{OfInputItemList: input}, Tools: []responses.ToolUnionParam{ responses.ToolParamOfComputerUsePreview(viewportHeight, viewportWidth, responses.ComputerUsePreviewToolEnvironmentBrowser), }, Reasoning: shared.ReasoningParam{Effort: shared.ReasoningEffortMedium}, Truncation: responses.ResponseNewParamsTruncationAuto, } if previousResponseID != "" { params.PreviousResponseID = openai.String(previousResponseID) } ``` `executeTask` walks `resp.Output`, switching on each item's `Type`. A `reasoning` item is the model thinking out loud and gets printed. A `message` item is terminal prose, stored as the final result. A `computer_call` item carries the actions to run. The model may batch several actions into one call, so `actionsFromCall` returns `call.Actions` when it is populated and the single `call.Action` otherwise, normalizing both into the same flat slice. When a turn produces no tool output, the loop stops. The model speaks OpenAI key names (`CTRL`, `ENTER`, `ESC`, `ArrowUp`). Steel expects DOM key names (`Control`, `Enter`, `Escape`). `normalizeKey` rewrites them before any `press_key` action goes out. ## Safety checks A `computer_call` can arrive with `PendingSafetyChecks`. You must echo each one back in `AcknowledgedSafetyChecks` on the matching `computer_call_output`, or the model stalls waiting for confirmation. This starter auto-acknowledges and prints each check: ```go for _, check := range call.PendingSafetyChecks { fmt.Printf("Auto-acknowledging safety check: %s\n", check.Message) acks = append(acks, responses.ResponseInputItemComputerCallOutputAcknowledgedSafetyCheckParam{ ID: check.ID, Code: openai.String(check.Code), Message: openai.String(check.Message), }) } ``` Auto-acknowledging suits a demo, not production. In a real deployment, surface the check's `Message` to a human and only acknowledge on approval. ## Run it ```bash cd examples/openai-computer-use-go cp .env.example .env # set STEEL_API_KEY and OPENAI_API_KEY go mod tidy go run . ``` Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [platform.openai.com](https://platform.openai.com/api-keys). The program prints a session viewer URL at startup. Open it in another tab to watch the browser run live. Override the task per run: ```bash TASK="Find the current weather in New York City" go run . ``` Your output varies. Structure looks like this: ```text Steel session created successfully! View live session at: https://app.steel.dev/sessions/ab12cd34... Executing task: Go to Steel.dev and find the latest news ============================================================ I'll open steel.dev and check for recent posts. keypress(keys=[Control l]) type(text="https://steel.dev") keypress(keys=[Enter]) ... Steel's latest update mentions ... ============================================================ TASK EXECUTION COMPLETED ============================================================ Duration: 78.4 seconds ``` A run typically takes 60-180 seconds across 10-30 model turns. Each turn is one Responses call plus one or more Steel computer actions, so a run costs a few cents of browser time on top of model tokens. Steel bills per session-minute, so the deferred `cleanup` that calls `Sessions.Release` is not optional: skip it and the browser keeps running until the session timeout (set to 900000 ms here in `initialize`). ## Make it yours - **Change the task.** Edit `TASK` in `.env` or pass it inline. - **Tune reasoning effort.** `shared.ReasoningEffortMedium` trades latency for deeper plans. Drop to `ReasoningEffortLow` for quick lookups, raise to `ReasoningEffortHigh` for multi-step research. - **Adjust the viewport.** `viewportWidth` and `viewportHeight` feed both the Steel session `Dimensions` and the `computer_use_preview` tool's display size. Keep the two in sync so the model's coordinates match the real screen. - **Raise or lower the ceiling.** `maxIterations` bounds the loop at 50 turns. Lower it to cap spend on a flaky task. - **Gate safety checks.** Replace the auto-acknowledge block with a prompt or an allowlist before appending to `acks`. - **Persist a login.** Pass `SessionContext` to `Sessions.Create` to reuse cookies across runs. See [credentials](/cookbook/credentials). ## Notes The published OpenAI Python and TypeScript computer-use recipes target the `{"type": "computer"}` tool on a newer general model. This Go port uses the dedicated `computer-use-preview` model and its `computer_use_preview` tool, which is the path the openai-go Responses API exposes today through `ToolParamOfComputerUsePreview`. The action vocabulary and the loop shape are identical; only the tool descriptor and model id differ. ## Related [Python version](/cookbook/openai-computer-use) · [TypeScript version](/cookbook/openai-computer-use) · [Claude on Steel in Go](/cookbook/claude-computer-use) · [OpenAI computer use guide](https://platform.openai.com/docs/guides/tools-computer-use) · [Responses API reference](https://platform.openai.com/docs/api-reference/responses) ## Related recipes - [Drive a browser with Gemini Computer Use](/cookbook/gemini-computer-use): Connect Google's Gemini Computer Use to a Steel browser session for autonomous web interactions. - [Drive a mobile browser with Claude Computer Use](/cookbook/claude-computer-use-mobile): Claude Computer Use with Steel for autonomous task execution in mobile browser environments. - [Drive a browser with Claude Computer Use](/cookbook/claude-computer-use): Connect Claude to a Steel browser session for autonomous web interactions.