Drive a browser with Gemini Computer Use

Connect Google's Gemini Computer Use to a Steel browser session for autonomous web interactions.

Jun RyuUpdated Apr 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

Gemini exposes computer use as a built-in tool type, not a hand-written schema. You set config.tools = [{ computerUse: { environment: Environment.ENVIRONMENT_BROWSER } }] on a generateContent call and the model plans against a fixed action vocabulary (click_at, type_text_at, navigate, scroll_document, search, drag_and_drop, key_combination, ...) with coordinates in a normalized 0-1000 grid.

The model defaults to gemini-3-flash-preview. Conversation state lives entirely on your side, appended to this.contents turn by turn. The Gemini Computer Use integration covers install and setup.

Coordinates and action mapping

Gemini plans in a 1000x1000 normalized grid regardless of the browser dimensions; denormalizeX and denormalizeY scale back to pixels off viewportWidth/viewportHeight (1440x900 by default).

private denormalizeX(x: number): number {
  return Math.round((x / MAX_COORDINATE) * this.viewportWidth);
}

Several of Gemini's actions are compound; the starter expands them. type_text_at fans into click, Ctrl+A, Backspace, type_text, Enter, wait, screenshot. navigate and search skip the URL bar hunt by doing the Chrome Ctrl+L trick: focus the address bar, type, press Enter, wait. key_combination arrives as a +-joined string like "Control+Enter"; splitKeys and normalizeKey break it apart and rewrite synonyms (CTRL to Control, CMD to Meta, ARROWUP to ArrowUp).

Every mapped action sets screenshot: true on the Steel call. The PNG comes back in the same response.

Sending frames back

Each completed call produces two parts in a single user-role turn: a functionResponse that names the call and echoes the current URL, then an inlineData part carrying the screenshot as raw base64.

const functionResponse: FunctionResponse = {
  name: fc.name ?? "",
  response: { url: result.url ?? this.currentUrl },
};
parts.push({ functionResponse });

parts.push({
  inlineData: {
    mimeType: "image/png",
    data: result.screenshotBase64,
  },
});

The loop

Four exits, in rough order of frequency:

Text, no function calls. The model wrote a final message.
Empty turn. No calls, no text. consecutiveNoActions increments. Three in a row stops the loop.
MALFORMED_FUNCTION_CALL with nothing else. A known quirk of the preview model; the loop continues to the next iteration.
Iteration cap. 50 turns by default.

The finally in main calls agent.cleanup(), which releases the Steel session.

Run it

cd examples/gemini-computer-use-ts
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
npm install
npm start

Get keys from app.steel.dev and aistudio.google.com. Override the task inline:

TASK="Find the current weather in New York City" npm start

Your output varies. Structure looks like this:

Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to steel.dev and scan the landing page for news.
navigate({"url":"https://steel.dev"})
scroll_document({"direction":"down"})
click_at({"x":520,"y":410})
Steel's latest release adds ...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.2 seconds

Expect roughly 60-120 seconds and 15-40 turns for a simple browsing task.

Make it yours

Resize the viewport. viewportWidth / viewportHeight in the Agent constructor feed both the Steel session dimensions and the denormalizeX / denormalizeY math.
Swap the model. this.model = "gemini-3-flash-preview" is the only version string.
Tune the system prompt. BROWSER_SYSTEM_PROMPT carries the browsing conventions: today's date via formatToday(), clear-before-typing, batch-actions-when-possible, black-screen recovery.
Gate safety decisions. Replace the auto-acknowledgement branch with a human approval before the next executeComputerAction fires.
Hand off auth. Pair this recipe with Steel's credentials or auth contexts to start the session already logged in.

Computer use docs · Python version · Anthropic equivalent · OpenAI equivalent

examples/gemini-computer-use-py

Jun RyuUpdated Apr 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

Gemini's computer use ships through google.genai as a single built-in tool: Tool(computer_use=ComputerUse(environment=ENVIRONMENT_BROWSER)). Setting ENVIRONMENT_BROWSER unlocks a fixed vocabulary of browser function calls (click_at, type_text_at, scroll_document, scroll_at, navigate, search, key_combination, drag_and_drop, hover_at, go_back, go_forward, open_web_browser, wait_5_seconds).

Steel supplies the screen. A Steel session is a headful Chromium in a VM, and sessions.computer(session_id, action=...) runs mouse and keyboard actions with a base64 PNG attached to the response.

The loop

Agent.execute_task seeds two user-role Parts (BROWSER_SYSTEM_PROMPT and the task) into self.contents, then calls generate_content in a loop:

response = self.client.models.generate_content(
    model=self.model,
    contents=self.contents,
    config=self.config,
)

candidate = response.candidates[0]
if candidate.content:
    self.contents.append(candidate.content)

reasoning = self.extract_text(candidate)
function_calls = self.extract_function_calls(candidate)

Gemini doesn't keep server-side conversation state, so every turn resends the full contents list including every prior screenshot.

Coordinates live in a 0-1000 canvas

Gemini never emits pixel coordinates. Every spatial argument (x, y, destination_x, destination_y, magnitude) is scaled against MAX_COORDINATE = 1000 regardless of viewport. denormalize_x and denormalize_y rescale onto Steel's viewport before each action:

def denormalize_x(self, x: int) -> int:
    return int(x / MAX_COORDINATE * self.viewport_width)

Sending screenshots back

Gemini expects each function response as two Parts in a user-role Content: a FunctionResponse with metadata, then an inline_data Blob carrying the PNG.

function_response = FunctionResponse(
    name=fc.name or "",
    response={"url": url or self.current_url},
)
parts.append(Part(function_response=function_response))

parts.append(
    Part(
        inline_data=types.Blob(
            mime_type="image/png",
            data=screenshot_base64,
        )
    )
)

Stopping conditions

execute_task ends one of three ways:

1Gemini emits only text, no function calls.
2Three consecutive iterations produce neither text nor function calls.
3max_iterations=50 caps total turns.

Run it

cd examples/gemini-computer-use-py
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
uv run main.py

Steel keys live at app.steel.dev/settings/api-keys. Gemini keys come from aistudio.google.com/apikey.

Override the task per run:

TASK="Find the current weather in New York City" python main.py

Your output varies. Structure looks like this:

Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================

I'll open steel.dev and look for the latest news.
navigate({"url": "https://steel.dev"})
scroll_document({"direction": "down"})
click_at({"x": 512, "y": 340})
...
Task complete - model provided final response

TASK EXECUTION COMPLETED
Duration: 78.4 seconds
Result: Steel's latest release notes mention ...

A run typically takes 60-180 seconds and 10-30 iterations. Because generate_content has no server-side state, every new turn resends the full self.contents list including every prior Blob. The finally block in main() calls sessions.release().

Make it yours

Change the task. Edit TASK in .env or pass it inline.
Swap the model. self.model = "gemini-3-flash-preview" in Agent.__init__.
Tune the viewport. viewport_width and viewport_height in Agent.__init__ flow into sessions.create(dimensions=...).
Gate safety confirmations. Replace the auto-acknowledge branch in execute_task with a human prompt.
Persist a login. Pass session_context to sessions.create to resume with cookies and local storage. See credentials.
Raise the ceiling. max_iterations=50 in execute_task bounds a single task.

TypeScript version · Gemini Computer Use guide · google-genai SDK

examples/gemini-computer-use-rs

Jun RyuUpdated Jun 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

There is no official Gemini Rust SDK, so this port talks to the generateContent REST endpoint directly over reqwest. The request body is a hand-built serde_json::Value: contents accumulates the conversation turn by turn, and tools carries a single { "computerUse": { "environment": "ENVIRONMENT_BROWSER" } } entry that switches Gemini into its built-in computer-use vocabulary. The browser itself is a Steel cloud session driven through the steel-rs crate, the same sessions().computer(...) surface the Anthropic and OpenAI Rust recipes use.

The model is gemini-3-flash-preview, the viewport is 1440x900, and the agent caps out at 50 iterations.

REST plumbing and coordinates

Everything in the request and response is camelCase JSON, so the two response structs (Candidate, GenerateContentResponse) carry #[serde(rename_all = "camelCase")] and the rest is read straight off serde_json::Value with .get(...). Auth is the x-goog-api-key header rather than a bearer token.

Gemini plans on a fixed 0-1000 grid regardless of the real viewport. denormalize_x and denormalize_y scale those numbers back to pixels off VIEWPORT_WIDTH / VIEWPORT_HEIGHT before any coordinate reaches Steel. Several actions are compound and get expanded locally: type_text_at fans into click, Ctrl+A, Backspace, type, optional Enter, and a wait; navigate and search skip the address-bar hunt with the Chrome Ctrl+L trick (focus the bar, type the URL, press Enter, wait). key_combination arrives as a +-joined string such as "Control+Enter", which split_keys and normalize_key break apart and rewrite to canonical names (CTRL to Control, CMD to Meta, ARROWUP to ArrowUp).

Sending frames back

In REST the screenshot stays a base64 string the whole way through. Each completed call appends two parts to a single user-role turn: a functionResponse naming the call and echoing the current URL, then an inlineData part with mimeType image/png and the base64 PNG as data. The bytes are never decoded.

The loop has four exits: a text-only turn (the model wrote its final answer), three empty turns in a row, a MALFORMED_FUNCTION_CALL finish reason with nothing else (a known preview-model quirk, retried on the next iteration), and the 50-iteration cap. A call may carry a safety_decision arg requesting confirmation; the agent logs it and auto-acknowledges before running the action.

Run it

cd examples/gemini-computer-use-rs
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
cargo run

Get keys from app.steel.dev and aistudio.google.com. Override the task with the TASK env var:

TASK="Find the current weather in New York City" cargo run

Output varies. The shape looks like this:

Steel + Gemini Computer Use Assistant
============================================================

Starting Steel session...
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...
Steel session started!
Executing task: Go to Steel.dev and find the latest news
============================================================

I'll navigate to steel.dev and scan the landing page for news.
navigate({"url":"https://steel.dev"})
scroll_document({"direction":"down"})
click_at({"x":520,"y":410})
Steel's latest release adds ...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.2 seconds
Task: Go to Steel.dev and find the latest news
Result:
Steel's latest release adds ...
============================================================
Releasing Steel session...
Session completed. View replay at https://app.steel.dev/sessions/ab12cd34...

Expect roughly 60 to 120 seconds and 15 to 40 turns for a simple browsing task.

Make it yours

Resize the viewport. VIEWPORT_WIDTH / VIEWPORT_HEIGHT feed both the Steel session dimensions and the denormalize_x / denormalize_y math, so they stay in sync.
Swap the model. GEMINI_URL is the only place the version string gemini-3-flash-preview appears.
Tune the system prompt. browser_system_prompt carries the browsing conventions: today's date via format_today, clear-before-typing, batch-actions-when-possible, black-screen recovery.
Gate safety decisions. Replace the auto-acknowledge branch with a human approval before the next execute_computer_action fires.
Cap the run. MAX_ITERATIONS bounds the loop; lower it for cheaper experiments.

Gemini computer use docs · TypeScript version · Python version · Anthropic equivalent · OpenAI equivalent

examples/gemini-computer-use-go

Jun RyuUpdated Jun 24, 2026

Terminal

Scaffolds a starter project locally. Requires the Steel CLI.

google.golang.org/genai exposes computer use as a typed tool on the request config: set config.Tools = []*genai.Tool{{ComputerUse: &genai.ComputerUse{Environment: genai.EnvironmentBrowser}}} and client.Models.GenerateContent(ctx, "gemini-3-flash-preview", contents, config) starts planning against a fixed browser vocabulary (click_at, type_text_at, navigate, scroll_document, search, drag_and_drop, key_combination, hover_at, go_back, go_forward, open_web_browser, wait_5_seconds). Coordinates arrive in a normalized 0-1000 grid.

Steel runs the screen. A session is a headful Chromium in a VM, and client.Sessions.Computer(ctx, sessionID, body) takes the action as its body and returns a *SessionComputerResponse whose Base64Image carries the resulting PNG.

Two typed surfaces, one bytes gotcha

The Steel computer endpoint accepts the action as an any body, so each action is a distinct struct: steel.ClickMouse, steel.MoveMouse, steel.PressKey, steel.TypeText, steel.Scroll, steel.DragMouse, steel.Wait, steel.TakeScreenshot. The agent's switch fc.Name builds the right one per Gemini call, and run reads *resp.Base64Image back out (pointer fields, nil-checked).

Gemini's args land in a map[string]any with JSON types: numbers are float64, so argInt casts before denormalizeX / denormalizeY scale 0-1000 onto the 1440x900 viewport. The one trap worth naming: genai.Blob.Data is []byte, not a base64 string. Steel hands back base64 text, so every screenshot is run through base64.StdEncoding.DecodeString before it becomes an InlineData part.

data, err := base64.StdEncoding.DecodeString(shots[i])
parts = append(parts, &genai.Part{
    InlineData: &genai.Blob{MIMEType: "image/png", Data: data},
})

Several Gemini actions are compound and get expanded locally. type_text_at fans into click, Ctrl+A, Backspace, type, optional Enter, a one-second wait, then a screenshot. navigate and search skip hunting for the URL bar by doing the Ctrl+L focus trick in openURL. key_combination arrives as a +-joined string; splitKeys and normalizeKey break it apart and rewrite synonyms (CTRL to Control, CMD to Meta, ARROWUP to ArrowUp).

The loop

executeTask seeds two user Parts (the system prompt and the task) into contents, then loops on GenerateContent. genai keeps no server-side state, so the full contents slice, every prior screenshot included, is resent each turn. Each turn appends the model's Content, then a user Content pairing one FunctionResponse (name plus current URL) with one InlineData screenshot per call. Four exits:

Text and no function calls: the model wrote its final answer.
Three consecutive empty turns (no text, no calls): stop.
FinishReasonMalformedFunctionCall with nothing else: a preview-model quirk, skip to the next iteration.
The 50-iteration cap.

main defers agent.cleanup, which releases the Steel session.

Run it

cd examples/gemini-computer-use-go
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
go mod tidy
go run .

Steel keys live at app.steel.dev/settings/api-keys; Gemini keys at aistudio.google.com/apikey. Override the task per run:

TASK="Find the current weather in New York City" go run .

Output varies. The shape is:

Steel + Gemini Computer Use Assistant
============================================================

Starting Steel session...
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...
Executing task: Go to Steel.dev and find the latest news
============================================================

I'll open steel.dev and scan the page for recent news.
navigate({"url":"https://steel.dev"})
scroll_document({"direction":"down"})
click_at({"x":512,"y":340})
Task complete - model provided final response
Releasing Steel session...
Session completed. View replay at https://app.steel.dev/sessions/ab12cd34...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.4 seconds
Task: Go to Steel.dev and find the latest news
Result:
Steel's latest release notes mention ...
============================================================

A run usually takes 60-180 seconds across 10-30 iterations.

Make it yours

Change the task. Edit TASK in .env or pass it inline.
Swap the model. The model constant is the only version string.
Resize the viewport. viewportWidth / viewportHeight feed both the Steel Dimensions and the denormalize math.
Gate safety decisions. Replace the auto-acknowledge branch in executeTask with a human approval before the action fires.
Hand off auth. Pass SessionContext to Sessions.Create to resume with cookies and local storage. See credentials.

TypeScript version · Python version · Anthropic equivalent · OpenAI equivalent · google.golang.org/genai

Drive a mobile browser with Claude Computer Use

Claude Computer Use with Steel for autonomous task execution in mobile browser environments.

Computer use Mobile

Oct 14, 2025

Drive a browser with Claude Computer Use

Connect Claude to a Steel browser session for autonomous web interactions.

Computer use

Jul 16, 2025

Drive a browser with OpenAI Computer Use

Connect OpenAI's Computer Use Assistant to a Steel browser session for autonomous web interactions.

Computer use

Mar 19, 2025