> Full docs index: https://docs.steel.dev/llms.txt

# Drive a browser with Gemini Computer Use
URL: https://docs.steel.dev/cookbook/gemini-computer-use


**TypeScript**

Gemini exposes [computer use](/cookbook/topics/computer-use) as a built-in tool type, not a hand-written schema. You set `config.tools = [{ computerUse: { environment: Environment.ENVIRONMENT_BROWSER } }]` on a `generateContent` call and the model plans against a fixed action vocabulary (`click_at`, `type_text_at`, `navigate`, `scroll_document`, `search`, `drag_and_drop`, `key_combination`, ...) with coordinates in a normalized 0-1000 grid.

The model defaults to `gemini-3-flash-preview`. Conversation state lives entirely on your side, appended to `this.contents` turn by turn. The [Gemini Computer Use integration](/integrations/gemini-computer-use) covers install and setup.

## Coordinates and action mapping

Gemini plans in a 1000x1000 normalized grid regardless of the browser dimensions; `denormalizeX` and `denormalizeY` scale back to pixels off `viewportWidth`/`viewportHeight` (1440x900 by default).

```typescript
private denormalizeX(x: number): number {
  return Math.round((x / MAX_COORDINATE) * this.viewportWidth);
}
```

Several of Gemini's actions are compound; the starter expands them. `type_text_at` fans into click, Ctrl+A, Backspace, type_text, Enter, wait, screenshot. `navigate` and `search` skip the URL bar hunt by doing the Chrome `Ctrl+L` trick: focus the address bar, type, press Enter, wait. `key_combination` arrives as a `+`-joined string like `"Control+Enter"`; `splitKeys` and `normalizeKey` break it apart and rewrite synonyms (`CTRL` to `Control`, `CMD` to `Meta`, `ARROWUP` to `ArrowUp`).

Every mapped action sets `screenshot: true` on the Steel call. The PNG comes back in the same response.

## Sending frames back

Each completed call produces two parts in a single user-role turn: a `functionResponse` that names the call and echoes the current URL, then an `inlineData` part carrying the screenshot as raw base64.

```typescript
const functionResponse: FunctionResponse = {
  name: fc.name ?? "",
  response: { url: result.url ?? this.currentUrl },
};
parts.push({ functionResponse });

parts.push({
  inlineData: {
    mimeType: "image/png",
    data: result.screenshotBase64,
  },
});
```

## The loop

Four exits, in rough order of frequency:

- **Text, no function calls.** The model wrote a final message.
- **Empty turn.** No calls, no text. `consecutiveNoActions` increments. Three in a row stops the loop.
- **`MALFORMED_FUNCTION_CALL` with nothing else.** A known quirk of the preview model; the loop continues to the next iteration.
- **Iteration cap.** 50 turns by default.

The `finally` in `main` calls `agent.cleanup()`, which releases the Steel session.

## Run it

```bash
cd examples/gemini-computer-use-ts
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
npm install
npm start
```

Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [aistudio.google.com](https://aistudio.google.com/apikey). Override the task inline:

```bash
TASK="Find the current weather in New York City" npm start
```

Your output varies. Structure looks like this:

```text
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================
I'll navigate to steel.dev and scan the landing page for news.
navigate({"url":"https://steel.dev"})
scroll_document({"direction":"down"})
click_at({"x":520,"y":410})
Steel's latest release adds ...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.2 seconds
```

Expect roughly 60-120 seconds and 15-40 turns for a simple browsing task.

## Make it yours

- **Resize the viewport.** `viewportWidth` / `viewportHeight` in the `Agent` constructor feed both the Steel session `dimensions` and the `denormalizeX` / `denormalizeY` math.
- **Swap the model.** `this.model = "gemini-3-flash-preview"` is the only version string.
- **Tune the system prompt.** `BROWSER_SYSTEM_PROMPT` carries the browsing conventions: today's date via `formatToday()`, clear-before-typing, batch-actions-when-possible, black-screen recovery.
- **Gate safety decisions.** Replace the auto-acknowledgement branch with a human approval before the next `executeComputerAction` fires.
- **Hand off auth.** Pair this recipe with Steel's [credentials](/cookbook/credentials) or [auth contexts](/cookbook/auth-context) to start the session already logged in.

## Related

[Computer use docs](https://ai.google.dev/gemini-api/docs/computer-use) · [Python version](/cookbook/gemini-computer-use) · [Anthropic equivalent](/cookbook/claude-computer-use) · [OpenAI equivalent](/cookbook/openai-computer-use)

**Python**

Gemini's computer use ships through `google.genai` as a single built-in tool: `Tool(computer_use=ComputerUse(environment=ENVIRONMENT_BROWSER))`. Setting `ENVIRONMENT_BROWSER` unlocks a fixed vocabulary of browser function calls (`click_at`, `type_text_at`, `scroll_document`, `scroll_at`, `navigate`, `search`, `key_combination`, `drag_and_drop`, `hover_at`, `go_back`, `go_forward`, `open_web_browser`, `wait_5_seconds`).

Steel supplies the screen. A Steel session is a headful Chromium in a VM, and `sessions.computer(session_id, action=...)` runs mouse and keyboard actions with a base64 PNG attached to the response.

## The loop

`Agent.execute_task` seeds two user-role `Part`s (`BROWSER_SYSTEM_PROMPT` and the task) into `self.contents`, then calls `generate_content` in a loop:

```python
response = self.client.models.generate_content(
    model=self.model,
    contents=self.contents,
    config=self.config,
)

candidate = response.candidates[0]
if candidate.content:
    self.contents.append(candidate.content)

reasoning = self.extract_text(candidate)
function_calls = self.extract_function_calls(candidate)
```

Gemini doesn't keep server-side conversation state, so every turn resends the full `contents` list including every prior screenshot.

## Coordinates live in a 0-1000 canvas

Gemini never emits pixel coordinates. Every spatial argument (`x`, `y`, `destination_x`, `destination_y`, `magnitude`) is scaled against `MAX_COORDINATE = 1000` regardless of viewport. `denormalize_x` and `denormalize_y` rescale onto Steel's viewport before each action:

```python
def denormalize_x(self, x: int) -> int:
    return int(x / MAX_COORDINATE * self.viewport_width)
```

## Sending screenshots back

Gemini expects each function response as two `Part`s in a user-role `Content`: a `FunctionResponse` with metadata, then an `inline_data` `Blob` carrying the PNG.

```python
function_response = FunctionResponse(
    name=fc.name or "",
    response={"url": url or self.current_url},
)
parts.append(Part(function_response=function_response))

parts.append(
    Part(
        inline_data=types.Blob(
            mime_type="image/png",
            data=screenshot_base64,
        )
    )
)
```

## Stopping conditions

`execute_task` ends one of three ways:

1. Gemini emits only text, no function calls.
2. Three consecutive iterations produce neither text nor function calls.
3. `max_iterations=50` caps total turns.

## Run it

```bash
cd examples/gemini-computer-use-py
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
uv run main.py
```

Steel keys live at [app.steel.dev/settings/api-keys](https://app.steel.dev/settings/api-keys). Gemini keys come from [aistudio.google.com/apikey](https://aistudio.google.com/apikey).

Override the task per run:

```bash
TASK="Find the current weather in New York City" python main.py
```

Your output varies. Structure looks like this:

```text
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...

Executing task: Go to Steel.dev and find the latest news
============================================================

I'll open steel.dev and look for the latest news.
navigate({"url": "https://steel.dev"})
scroll_document({"direction": "down"})
click_at({"x": 512, "y": 340})
...
Task complete - model provided final response

TASK EXECUTION COMPLETED
Duration: 78.4 seconds
Result: Steel's latest release notes mention ...
```

A run typically takes 60-180 seconds and 10-30 iterations. Because `generate_content` has no server-side state, every new turn resends the full `self.contents` list including every prior `Blob`. The `finally` block in `main()` calls `sessions.release()`.

## Make it yours

- Change the task. Edit `TASK` in `.env` or pass it inline.
- Swap the model. `self.model = "gemini-3-flash-preview"` in `Agent.__init__`.
- Tune the viewport. `viewport_width` and `viewport_height` in `Agent.__init__` flow into `sessions.create(dimensions=...)`.
- Gate safety confirmations. Replace the auto-acknowledge branch in `execute_task` with a human prompt.
- Persist a login. Pass `session_context` to `sessions.create` to resume with cookies and local storage. See [credentials](/cookbook/credentials).
- Raise the ceiling. `max_iterations=50` in `execute_task` bounds a single task.

## Related

[TypeScript version](/cookbook/gemini-computer-use) · [Gemini Computer Use guide](https://ai.google.dev/gemini-api/docs/computer-use) · [google-genai SDK](https://googleapis.github.io/python-genai/)

**Rust**

There is no official Gemini Rust SDK, so this port talks to the `generateContent` REST endpoint directly over `reqwest`. The request body is a hand-built `serde_json::Value`: `contents` accumulates the conversation turn by turn, and `tools` carries a single `{ "computerUse": { "environment": "ENVIRONMENT_BROWSER" } }` entry that switches Gemini into its built-in computer-use vocabulary. The browser itself is a Steel cloud session driven through the `steel-rs` crate, the same `sessions().computer(...)` surface the Anthropic and OpenAI Rust recipes use.

The model is `gemini-3-flash-preview`, the viewport is 1440x900, and the agent caps out at 50 iterations.

## REST plumbing and coordinates

Everything in the request and response is camelCase JSON, so the two response structs (`Candidate`, `GenerateContentResponse`) carry `#[serde(rename_all = "camelCase")]` and the rest is read straight off `serde_json::Value` with `.get(...)`. Auth is the `x-goog-api-key` header rather than a bearer token.

Gemini plans on a fixed 0-1000 grid regardless of the real viewport. `denormalize_x` and `denormalize_y` scale those numbers back to pixels off `VIEWPORT_WIDTH` / `VIEWPORT_HEIGHT` before any coordinate reaches Steel. Several actions are compound and get expanded locally: `type_text_at` fans into click, Ctrl+A, Backspace, type, optional Enter, and a wait; `navigate` and `search` skip the address-bar hunt with the Chrome `Ctrl+L` trick (focus the bar, type the URL, press Enter, wait). `key_combination` arrives as a `+`-joined string such as `"Control+Enter"`, which `split_keys` and `normalize_key` break apart and rewrite to canonical names (`CTRL` to `Control`, `CMD` to `Meta`, `ARROWUP` to `ArrowUp`).

## Sending frames back

In REST the screenshot stays a base64 string the whole way through. Each completed call appends two parts to a single user-role turn: a `functionResponse` naming the call and echoing the current URL, then an `inlineData` part with `mimeType` `image/png` and the base64 PNG as `data`. The bytes are never decoded.

The loop has four exits: a text-only turn (the model wrote its final answer), three empty turns in a row, a `MALFORMED_FUNCTION_CALL` finish reason with nothing else (a known preview-model quirk, retried on the next iteration), and the 50-iteration cap. A call may carry a `safety_decision` arg requesting confirmation; the agent logs it and auto-acknowledges before running the action.

## Run it

```bash
cd examples/gemini-computer-use-rs
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
cargo run
```

Get keys from [app.steel.dev](https://app.steel.dev/settings/api-keys) and [aistudio.google.com](https://aistudio.google.com/apikey). Override the task with the `TASK` env var:

```bash
TASK="Find the current weather in New York City" cargo run
```

Output varies. The shape looks like this:

```text
Steel + Gemini Computer Use Assistant
============================================================

Starting Steel session...
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...
Steel session started!
Executing task: Go to Steel.dev and find the latest news
============================================================

I'll navigate to steel.dev and scan the landing page for news.
navigate({"url":"https://steel.dev"})
scroll_document({"direction":"down"})
click_at({"x":520,"y":410})
Steel's latest release adds ...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.2 seconds
Task: Go to Steel.dev and find the latest news
Result:
Steel's latest release adds ...
============================================================
Releasing Steel session...
Session completed. View replay at https://app.steel.dev/sessions/ab12cd34...
```

Expect roughly 60 to 120 seconds and 15 to 40 turns for a simple browsing task.

## Make it yours

- **Resize the viewport.** `VIEWPORT_WIDTH` / `VIEWPORT_HEIGHT` feed both the Steel session `dimensions` and the `denormalize_x` / `denormalize_y` math, so they stay in sync.
- **Swap the model.** `GEMINI_URL` is the only place the version string `gemini-3-flash-preview` appears.
- **Tune the system prompt.** `browser_system_prompt` carries the browsing conventions: today's date via `format_today`, clear-before-typing, batch-actions-when-possible, black-screen recovery.
- **Gate safety decisions.** Replace the auto-acknowledge branch with a human approval before the next `execute_computer_action` fires.
- **Cap the run.** `MAX_ITERATIONS` bounds the loop; lower it for cheaper experiments.

## Related

[Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use) · [TypeScript version](/cookbook/gemini-computer-use) · [Python version](/cookbook/gemini-computer-use) · [Anthropic equivalent](/cookbook/claude-computer-use) · [OpenAI equivalent](/cookbook/openai-computer-use)

**Go**

`google.golang.org/genai` exposes computer use as a typed tool on the request config: set `config.Tools = []*genai.Tool{{ComputerUse: &genai.ComputerUse{Environment: genai.EnvironmentBrowser}}}` and `client.Models.GenerateContent(ctx, "gemini-3-flash-preview", contents, config)` starts planning against a fixed browser vocabulary (`click_at`, `type_text_at`, `navigate`, `scroll_document`, `search`, `drag_and_drop`, `key_combination`, `hover_at`, `go_back`, `go_forward`, `open_web_browser`, `wait_5_seconds`). Coordinates arrive in a normalized 0-1000 grid.

Steel runs the screen. A session is a headful Chromium in a VM, and `client.Sessions.Computer(ctx, sessionID, body)` takes the action as its body and returns a `*SessionComputerResponse` whose `Base64Image` carries the resulting PNG.

## Two typed surfaces, one bytes gotcha

The Steel computer endpoint accepts the action as an `any` body, so each action is a distinct struct: `steel.ClickMouse`, `steel.MoveMouse`, `steel.PressKey`, `steel.TypeText`, `steel.Scroll`, `steel.DragMouse`, `steel.Wait`, `steel.TakeScreenshot`. The agent's `switch fc.Name` builds the right one per Gemini call, and `run` reads `*resp.Base64Image` back out (pointer fields, nil-checked).

Gemini's args land in a `map[string]any` with JSON types: numbers are `float64`, so `argInt` casts before `denormalizeX` / `denormalizeY` scale 0-1000 onto the 1440x900 viewport. The one trap worth naming: `genai.Blob.Data` is `[]byte`, not a base64 string. Steel hands back base64 text, so every screenshot is run through `base64.StdEncoding.DecodeString` before it becomes an `InlineData` part.

```go
data, err := base64.StdEncoding.DecodeString(shots[i])
parts = append(parts, &genai.Part{
    InlineData: &genai.Blob{MIMEType: "image/png", Data: data},
})
```

Several Gemini actions are compound and get expanded locally. `type_text_at` fans into click, Ctrl+A, Backspace, type, optional Enter, a one-second wait, then a screenshot. `navigate` and `search` skip hunting for the URL bar by doing the Ctrl+L focus trick in `openURL`. `key_combination` arrives as a `+`-joined string; `splitKeys` and `normalizeKey` break it apart and rewrite synonyms (`CTRL` to `Control`, `CMD` to `Meta`, `ARROWUP` to `ArrowUp`).

## The loop

`executeTask` seeds two user `Part`s (the system prompt and the task) into `contents`, then loops on `GenerateContent`. genai keeps no server-side state, so the full `contents` slice, every prior screenshot included, is resent each turn. Each turn appends the model's `Content`, then a user `Content` pairing one `FunctionResponse` (name plus current URL) with one `InlineData` screenshot per call. Four exits:

- Text and no function calls: the model wrote its final answer.
- Three consecutive empty turns (no text, no calls): stop.
- `FinishReasonMalformedFunctionCall` with nothing else: a preview-model quirk, skip to the next iteration.
- The 50-iteration cap.

`main` defers `agent.cleanup`, which releases the Steel session.

## Run it

```bash
cd examples/gemini-computer-use-go
cp .env.example .env          # set STEEL_API_KEY and GEMINI_API_KEY
go mod tidy
go run .
```

Steel keys live at [app.steel.dev/settings/api-keys](https://app.steel.dev/settings/api-keys); Gemini keys at [aistudio.google.com/apikey](https://aistudio.google.com/apikey). Override the task per run:

```bash
TASK="Find the current weather in New York City" go run .
```

Output varies. The shape is:

```text
Steel + Gemini Computer Use Assistant
============================================================

Starting Steel session...
Steel Session created successfully!
View live session at: https://app.steel.dev/sessions/ab12cd34...
Executing task: Go to Steel.dev and find the latest news
============================================================

I'll open steel.dev and scan the page for recent news.
navigate({"url":"https://steel.dev"})
scroll_document({"direction":"down"})
click_at({"x":512,"y":340})
Task complete - model provided final response
Releasing Steel session...
Session completed. View replay at https://app.steel.dev/sessions/ab12cd34...

============================================================
TASK EXECUTION COMPLETED
============================================================
Duration: 78.4 seconds
Task: Go to Steel.dev and find the latest news
Result:
Steel's latest release notes mention ...
============================================================
```

A run usually takes 60-180 seconds across 10-30 iterations.

## Make it yours

- Change the task. Edit `TASK` in `.env` or pass it inline.
- Swap the model. The `model` constant is the only version string.
- Resize the viewport. `viewportWidth` / `viewportHeight` feed both the Steel `Dimensions` and the denormalize math.
- Gate safety decisions. Replace the auto-acknowledge branch in `executeTask` with a human approval before the action fires.
- Hand off auth. Pass `SessionContext` to `Sessions.Create` to resume with cookies and local storage. See [credentials](/cookbook/credentials).

## Related

[TypeScript version](/cookbook/gemini-computer-use) · [Python version](/cookbook/gemini-computer-use) · [Anthropic equivalent](/cookbook/claude-computer-use) · [OpenAI equivalent](/cookbook/openai-computer-use) · [google.golang.org/genai](https://pkg.go.dev/google.golang.org/genai)

## Related recipes

- [Drive a mobile browser with Claude Computer Use](/cookbook/claude-computer-use-mobile): Claude Computer Use with Steel for autonomous task execution in mobile browser environments.
- [Drive a browser with Claude Computer Use](/cookbook/claude-computer-use): Connect Claude to a Steel browser session for autonomous web interactions.
- [Drive a browser with OpenAI Computer Use](/cookbook/openai-computer-use): Connect OpenAI's Computer Use Assistant to a Steel browser session for autonomous web interactions.