Drive a browser with OpenAI Computer Use

Connect OpenAI's Computer Use Assistant to a Steel browser session for autonomous web interactions.

Jun Ryu, Nikola BalicUpdated Apr 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

OpenAI's computer-use model ships as a single tool declaration: { type: "computer" }. You hand it to the Responses API, send a screenshot, and the model returns computer_call items with actions like click, type, keypress, scroll. Your job is to execute them against a real browser, capture the next screenshot, and feed it back. Steel's OpenAI Computer Use integration covers the same loop as a guide.

The loop

The Responses API threads conversation state server-side, so each turn carries only the new tool outputs plus a previous_response_id:

const response = await createResponse({
  model: this.model,
  instructions: this.systemPrompt,
  input: nextInput,
  tools: this.tools,
  previous_response_id: previousResponseId,
  reasoning: { effort: "medium" },
  truncation: "auto",
});

First turn: nextInput is [{ role: "user", content: task }]. Subsequent turns: nextInput is just the array of tool outputs from the previous iteration.

The response's output array mixes three item types:

message: a plain text reply. The final message becomes the return value.
reasoning: the model's own summary; printed but not fed back.
computer_call: one or more actions to execute. Each call has a call_id that must be echoed back in the matching computer_call_output.

Actions in, screenshots out

executeComputerAction is the translation layer:

case "click": {
  const coords = this.toCoords(actionArgs.x, actionArgs.y);
  const button = this.mapButton(actionArgs.button);
  const clicks = this.toNumber(actionArgs.num_clicks, 1);
  body = {
    action: "click_mouse",
    button,
    coordinates: coords,
    ...(clicks > 1 ? { num_clicks: clicks } : {}),
    screenshot: true,
  };
  break;
}

The screenshot goes back as a computer_call_output, matched by call_id:

toolOutputs.push({
  type: "computer_call_output",
  call_id: item.call_id,
  acknowledged_safety_checks: pendingChecks,
  output: {
    type: "computer_screenshot",
    image_url: `data:image/png;base64,${screenshotBase64}`,
  },
});

A few translation details:

keypress takes a list. normalizeKey rewrites synonyms (CTRL to Control, CMD to Meta, ENTER to Enter).
scroll is delta-based. OpenAI sends scroll_x/scroll_y in pixels. Steel's scroll takes delta_x/delta_y directly.
drag gives a path. OpenAI provides the full point list in path; Steel's drag_mouse wants the same shape.
Unknown actions fall through to take_screenshot.

Safety checks

A computer_call can attach pending_safety_checks when the planned action looks sensitive. The call won't take effect until you echo those check IDs back in acknowledged_safety_checks. The starter auto-acknowledges everything; for production, flip autoAcknowledgeSafety to false and gate each check on a human approval.

Run it

cd examples/openai-computer-use-ts
cp .env.example .env          # set STEEL_API_KEY and OPENAI_API_KEY
npm install
npm start

Get keys from app.steel.dev and platform.openai.com. Override the task inline:

TASK="Find the current weather in New York City" npm start

Your output varies. Structure looks like this:

Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to steel.dev and scan the landing page.
click({"x":720,"y":48})
type({"text":"https://steel.dev"})
keypress({"keys":["Enter"]})
scroll({"x":720,"y":450,"scroll_y":600})
Steel's latest release adds ...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 71.4 seconds

Expect roughly 60-120 seconds and 15-40 turns for a simple browsing task.

Make it yours

Change the viewport. viewportWidth and viewportHeight in the Agent constructor set the Steel session dimensions.
Swap the model. The default is gpt-5.5. Update this.model in the Agent constructor.
Tune reasoning effort. reasoning: { effort: "medium" } trades latency for planning quality.
Rewrite the system prompt. BROWSER_SYSTEM_PROMPT holds the browsing conventions.
Persist a login. Pass sessionContext to sessions.create. See credentials and auth-context.
Turn off auto-ack. Flip autoAcknowledgeSafety to false to make pending safety checks raise.

Computer use guide · Python version · Anthropic equivalent

examples/openai-computer-use-py

Jun Ryu, Nikola BalicUpdated Apr 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

OpenAI's Computer Use models expose one tool ({"type": "computer"}) and emit computer_call items containing an action the model wants performed on a screen. You execute the action, return a screenshot as a computer_call_output, and the next turn the model sees the result. The action vocabulary (click, type, keypress, scroll, drag, wait, screenshot) is fixed by OpenAI.

This recipe uses OpenAI's Responses API, not Chat Completions. Responses keeps conversation state on OpenAI's side via previous_response_id, so each turn only sends the new tool outputs rather than the full screenshot history.

The loop

params = {
    "model": self.model,
    "instructions": self.system_prompt,
    "input": next_input,
    "tools": self.tools,
    "reasoning": {"effort": "medium"},
    "truncation": "auto",
}
if previous_response_id:
    params["previous_response_id"] = previous_response_id

response = create_response(**params)
previous_response_id = response.get("id")

Each response["output"] is a list of items with a type. The loop walks them:

reasoning: model's internal thinking, printed.
message: terminal prose; the agent stores the last one as the final result.
computer_call: one or more actions to execute.

execute_computer_action maps OpenAI's action vocabulary onto Steel's Input API. Each branch builds a Steel request body and sends it through self.steel.sessions.computer(...) with screenshot: True:

elif action_type in ("click",):
    coords = self.to_coords(action_args.get("x"), action_args.get("y"))
    button = self.map_button(action_args.get("button"))
    num_clicks = int(self.to_number(action_args.get("num_clicks"), 1))
    payload = {
        "action": "click_mouse",
        "button": button,
        "coordinates": [coords[0], coords[1]],
        "screenshot": True,
    }
    if num_clicks > 1:
        payload["num_clicks"] = num_clicks
    body = payload

keypress arrives with OpenAI names (CTRL, ENTER, ESC, UP); normalize_key rewrites them into the Steel / DOM vocabulary (Control, Enter, Escape, ArrowUp).

The screenshot goes back as a computer_call_output:

tool_outputs.append({
    "type": "computer_call_output",
    "call_id": item["call_id"],
    "acknowledged_safety_checks": pending_checks,
    "output": {
        "type": "computer_screenshot",
        "image_url": f"data:image/png;base64,{screenshot_base64}",
    },
})

Safety checks

A computer_call can include pending_safety_checks. You must echo them back in acknowledged_safety_checks on the next turn, or the model stalls. The default here is auto_acknowledge_safety = True, which suits a starter but is not what you want in production. Flip it to False and surface the check to a human before proceeding.

Run it

cd examples/openai-computer-use-py
cp .env.example .env          # set STEEL_API_KEY and OPENAI_API_KEY
uv run main.py

Get keys from app.steel.dev and platform.openai.com.

Override the task per run:

TASK="Find the current weather in New York City" python main.py

Your output varies. Structure looks like this:

Starting Steel session...
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34…

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll open steel.dev and check the blog.
keypress({"keys": ["CTRL", "L"]})
type({"text": "https://steel.dev"})
keypress({"keys": ["ENTER"]})
wait({"ms": 1500})
…
Steel's latest release notes mention …

TASK EXECUTION COMPLETED
Duration: 62.8 seconds

A run typically takes 60-180 seconds and 10-30 iterations. Screenshots are cached between turns via previous_response_id, so per-turn input cost stays roughly flat even on long loops. The finally block in main() calls sessions.release().

Make it yours

Change the task. Edit TASK in .env or pass it inline.
Swap the model. The default is gpt-5.5. Update self.model in Agent.__init__.
Tune the viewport. viewport_width / viewport_height in Agent.__init__ flow into sessions.create(dimensions=...).
Turn off auto-ack. Flip auto_acknowledge_safety = False to make pending safety checks raise.
Persist a login. Pass session_context to sessions.create. See credentials.
Adjust reasoning. "effort": "medium" trades latency for deeper plans. Drop to "low" for fast lookups, raise to "high" for multi-step research.

TypeScript version · Claude version · OpenAI Computer Use guide · Responses API reference

examples/openai-computer-use-rs

Jun RyuUpdated Jun 23, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

OpenAI's computer-use-preview model returns mouse and keyboard actions instead of text. This recipe executes those actions against a real Chromium running in Steel's cloud, feeds each resulting screenshot back, and loops until the model reports the task done. There is no official OpenAI Rust SDK, so it calls the Responses API directly with reqwest and drives the browser through Steel's server-side computer endpoint. It is the Rust counterpart to openai-computer-use-py and the OpenAI sibling of claude-computer-use-rs.

The loop

The Responses API is stateful. Each turn you pass the previous response.id as previous_response_id and send only the new input, so the conversation never gets resent:

let mut input = json!([{ "role": "user", "content": task }]);
let mut previous_response_id: Option<String> = None;

for _ in 0..MAX_ITERATIONS {
    let response = self.call_openai(&input, &previous_response_id).await?;
    previous_response_id = Some(response.id);

    let mut next_input = Vec::new();
    for item in &response.output {
        // message -> print it, reasoning -> print it,
        // computer_call -> run the action, screenshot, push a computer_call_output
    }
    if next_input.is_empty() { break; } // model returned only text: done
    input = Value::Array(next_input);
}

Contrast claude-computer-use-rs, where the Anthropic Messages API is stateless and you grow and resend a messages array every turn. Here the server holds the history and you send back only computer_call_output items. MAX_ITERATIONS caps the loop so a stuck model cannot run forever.

From OpenAI action to Steel action

A computer_call carries one action such as { "type": "click", "button": "left", "x": 412, "y": 280 }. execute_action matches on type and builds the matching Steel SessionComputerParams:

"click" => SessionComputerParams::ClickMouse(ComputerActionRequestClickMouse {
    button: Some(map_button(/* left | right | middle | back | forward */)),
    coordinates: Some(vec![x, y]),
    screenshot: Some(true),
    ..
}),
"type"     => SessionComputerParams::TypeText(/* text */),
"keypress" => SessionComputerParams::PressKey(/* normalized keys */),
"scroll"   => SessionComputerParams::Scroll(/* delta_x, delta_y */),

Every action sets screenshot: true, so Steel returns a fresh base64 PNG. That image goes back as a computer_call_output with image_url: data:image/png;base64,..., which is what the model sees for its next move. OpenAI key names (ENTER, CTRL, ESC) are normalized to the DOM vocabulary Steel expects (Enter, Control, Escape).

When a turn includes a pending_safety_check, the recipe auto-acknowledges it by echoing it back in acknowledged_safety_checks. That is fine for a demo on a throwaway page. Read each check before letting an agent act on a real account.

Run it

cd examples/openai-computer-use-rs
cp .env.example .env          # set STEEL_API_KEY and OPENAI_API_KEY
cargo run

Get a Steel key at app.steel.dev/settings/api-keys and an OpenAI key at platform.openai.com/api-keys. Set TASK in .env to change the goal. Your output varies. Structure looks like this:

Steel + OpenAI Computer Use Assistant
============================================================

Starting Steel session...
View live session at: https://app.steel.dev/sessions/3f2a...
Executing task: Go to Steel.dev and find the latest news
============================================================
click(button=left x=720 y=400)
type(text="steel.dev blog")
keypress(keys=["Enter"])
...
The latest Steel post is "...".
============================================================
TASK EXECUTION COMPLETED
Duration: 48.2 seconds

A run drives a real session and a vision model across many turns, so it costs a few cents of browser time plus the OpenAI tokens for the loop. Steel bills per session-minute until cleanup releases the session, which always runs through a deferred release, even on error.

Make it yours

Change the task. Set TASK in .env, or edit the default in main.
Resize the viewport. VIEWPORT_WIDTH and VIEWPORT_HEIGHT set both the Steel session dimensions and the display_width/display_height on the tool. Keep them in sync so the model's coordinates match the page.
Gate safety checks. Instead of auto-acknowledging every pending_safety_check, prompt a human or allowlist specific codes before echoing them back.
Start authenticated. Pass a session context or credentials to sessions().create(...) so the agent begins on a logged-in page. See auth-context and credentials.

openai-computer-use-go is the same agent through the official openai-go SDK, which has a typed Responses API. openai-computer-use-py and openai-computer-use-ts are the Python and TypeScript versions. claude-computer-use-rs runs the same Steel action loop against Anthropic instead.

examples/openai-computer-use-go

Jun RyuUpdated Jun 23, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

This recipe wires two typed Go SDKs together so OpenAI's computer-use-preview model can drive a Steel cloud browser. The model emits a computer action through the official github.com/openai/openai-go/v3 Responses API. You execute that action against a Steel session with client.Sessions.Computer, then hand the resulting screenshot back so the next turn sees what changed. That single exchange, repeated, is the whole agent.

The interesting part in Go is the seam between the two SDKs, because each models the action vocabulary differently. openai-go gives you a flattened union: ResponseComputerToolCallActionUnion carries every possible field (X, Y, Button, Keys, ScrollX, Text, Path) on one struct, and you read whichever ones the Type discriminator says are live. Steel's Go SDK takes the opposite shape: SessionComputerParams is a constructed discriminated union where you set an Action string and attach the one matching ComputerActionRequest* pointer. executeAction is the translation layer between the two.

case "click":
    body := &steel.ComputerActionRequestClickMouse{
        Action:      steel.ComputerActionRequestClickMouseActionClickMouse,
        Button:      ptr(mapButton(act.Button)),
        Coordinates: coords(),
        Screenshot:  ptr(true),
    }
    return a.run(ctx, steel.SessionComputerParams{Action: "click_mouse", ComputerActionRequestClickMouse: body})

Every branch sets Screenshot: ptr(true), so the Steel call that performs the action also returns the screenshot in the same round trip. run reads resp.Base64Image and falls back to an explicit take_screenshot if a particular action did not capture one.

Responses keeps the conversation, you keep the loop

The Responses API stores conversation state server side. The first turn sends the task as a user message; every later turn sends only the new computer_call_output items and threads PreviousResponseID from the prior response. You never resend the screenshot history, so input size stays roughly flat even across a long run.

params := responses.ResponseNewParams{
    Model:        shared.ResponsesModelComputerUsePreview,
    Instructions: openai.String(systemPrompt()),
    Input:        responses.ResponseNewParamsInputUnion{OfInputItemList: input},
    Tools: []responses.ToolUnionParam{
        responses.ToolParamOfComputerUsePreview(viewportHeight, viewportWidth, responses.ComputerUsePreviewToolEnvironmentBrowser),
    },
    Reasoning:  shared.ReasoningParam{Effort: shared.ReasoningEffortMedium},
    Truncation: responses.ResponseNewParamsTruncationAuto,
}
if previousResponseID != "" {
    params.PreviousResponseID = openai.String(previousResponseID)
}

executeTask walks resp.Output, switching on each item's Type. A reasoning item is the model thinking out loud and gets printed. A message item is terminal prose, stored as the final result. A computer_call item carries the actions to run. The model may batch several actions into one call, so actionsFromCall returns call.Actions when it is populated and the single call.Action otherwise, normalizing both into the same flat slice. When a turn produces no tool output, the loop stops.

The model speaks OpenAI key names (CTRL, ENTER, ESC, ArrowUp). Steel expects DOM key names (Control, Enter, Escape). normalizeKey rewrites them before any press_key action goes out.

Safety checks

A computer_call can arrive with PendingSafetyChecks. You must echo each one back in AcknowledgedSafetyChecks on the matching computer_call_output, or the model stalls waiting for confirmation. This starter auto-acknowledges and prints each check:

for _, check := range call.PendingSafetyChecks {
    fmt.Printf("Auto-acknowledging safety check: %s\n", check.Message)
    acks = append(acks, responses.ResponseInputItemComputerCallOutputAcknowledgedSafetyCheckParam{
        ID: check.ID, Code: openai.String(check.Code), Message: openai.String(check.Message),
    })
}

Auto-acknowledging suits a demo, not production. In a real deployment, surface the check's Message to a human and only acknowledge on approval.

Run it

cd examples/openai-computer-use-go
cp .env.example .env          # set STEEL_API_KEY and OPENAI_API_KEY
go mod tidy
go run .

Get keys from app.steel.dev and platform.openai.com. The program prints a session viewer URL at startup. Open it in another tab to watch the browser run live.

Override the task per run:

TASK="Find the current weather in New York City" go run .

Your output varies. Structure looks like this:

Steel session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll open steel.dev and check for recent posts.
keypress(keys=[Control l])
type(text="https://steel.dev")
keypress(keys=[Enter])
...
Steel's latest update mentions ...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.4 seconds

A run typically takes 60-180 seconds across 10-30 model turns. Each turn is one Responses call plus one or more Steel computer actions, so a run costs a few cents of browser time on top of model tokens. Steel bills per session-minute, so the deferred cleanup that calls Sessions.Release is not optional: skip it and the browser keeps running until the session timeout (set to 900000 ms here in initialize).

Make it yours

Change the task. Edit TASK in .env or pass it inline.
Tune reasoning effort. shared.ReasoningEffortMedium trades latency for deeper plans. Drop to ReasoningEffortLow for quick lookups, raise to ReasoningEffortHigh for multi-step research.
Adjust the viewport. viewportWidth and viewportHeight feed both the Steel session Dimensions and the computer_use_preview tool's display size. Keep the two in sync so the model's coordinates match the real screen.
Raise or lower the ceiling. maxIterations bounds the loop at 50 turns. Lower it to cap spend on a flaky task.
Gate safety checks. Replace the auto-acknowledge block with a prompt or an allowlist before appending to acks.
Persist a login. Pass SessionContext to Sessions.Create to reuse cookies across runs. See credentials.

Notes

The published OpenAI Python and TypeScript computer-use recipes target the {"type": "computer"} tool on a newer general model. This Go port uses the dedicated computer-use-preview model and its computer_use_preview tool, which is the path the openai-go Responses API exposes today through ToolParamOfComputerUsePreview. The action vocabulary and the loop shape are identical; only the tool descriptor and model id differ.

Python version · TypeScript version · Claude on Steel in Go · OpenAI computer use guide · Responses API reference

Drive a browser with Gemini Computer Use

Connect Google's Gemini Computer Use to a Steel browser session for autonomous web interactions.

Computer use

Nov 25, 2025

Drive a mobile browser with Claude Computer Use

Claude Computer Use with Steel for autonomous task execution in mobile browser environments.

Computer use Mobile

Oct 14, 2025

Drive a browser with Claude Computer Use

Connect Claude to a Steel browser session for autonomous web interactions.

Computer use

Jul 16, 2025