Connect with Selenium

Our Selenium integration is in its early stages and is not at feature parity with our Puppeteer and Playwright integrations. Some features like CAPTCHA solving and proxy support are currently unavailable. More details are provided below.

Steel sessions are designed to be easily driven by Selenium, allowing you to run your existing Selenium scripts in the cloud with minimal changes.

This guide shows you how to drive Steel's cloud browser sessions using Selenium with Python.

Quick Start: Want to jump right in? Skip to example project.

Limitations

Before we begin, please note that the following features are not yet supported in our Selenium integration:

• CAPTCHA Solving: Automatic CAPTCHA solving is not available.

• Proxy Support: Custom proxy configurations are currently unsupported.

• Advanced Session Management: Features like session cloning and cookie manipulation are limited.

• Live Session Viewer: While sessions are logged in the Steel Cloud app, we don’t currently have support for the live session viewer.

Connecting to Steel with Selenium

Most Selenium scripts start with a simple WebDriver setup that looks something like this:

1
from selenium import webdriver
2

3
driver = webdriver.Chrome()  # or Firefox(), Safari(), etc.
4
driver.get('https://example.com')

To run your script with Steel, you'll need to:

• Create a session with Selenium support enabled

• Set up custom header handling (required for authentication)

• Connect using Steel's dedicated Selenium URL

Here's what that looks like:

First, create a custom connection handler for Steel-specific headers:

1
from selenium.webdriver.remote.remote_connection import RemoteConnection
2

3
class CustomRemoteConnection(RemoteConnection):
4
    def __init__(self, remote_server_addr: str, session_id: str):
5
        super().__init__(remote_server_addr)
6
        self._session_id = session_id
7

8
    def get_remote_connection_headers(self, parsed_url, keep_alive=False):
9
        headers = super().get_remote_connection_headers(parsed_url, keep_alive)
10
        headers.update({
11
            'steel-api-key': os.environ.get("STEEL_API_KEY"),
12
            'session-id': self._session_id
13
        })
14
        return headers
15

Then use it to connect to Steel:

1
from steel import Steel
2
from selenium import webdriver
3
import os
4

5
client = Steel(
6
    steel_api_key=os.getenv('STEEL_API_KEY'),
7
)
8

9
def main():
10
    # Create a session with Selenium support
11
    session = client.sessions.create(
12
        is_selenium=True,  # Required for Selenium sessions
13
    )
14

15
    # Connect using the custom connection handler
16
    driver = webdriver.Remote(
17
        command_executor=CustomRemoteConnection(
18
            remote_server_addr='http://connect.steelbrowser.com/selenium',
19
            session_id=session.id
20
        ),
21
        options=webdriver.ChromeOptions()
22
    )
23

24
    # Run your automation
25
    driver.get('https://example.com')
26

27
    # Clean up when done
28
    driver.quit()
29
    client.sessions.release(session.id)
30

31
if __name__ == "__main__":
32
    main()

Important: Sessions remain active until explicitly released or timed out. It’s best practise to call client.sessions.release() when finished instead of relying on timeout.

Why Custom Headers?

Unlike Puppeteer and Playwright, Selenium doesn't natively support adding the headers required by Steel (session-id and steel-api-key). That's why we need to create a custom connection handler to include these headers with each request.

Example Project: Scraping Hacker News

Here's a working example that scrapes Hacker News with proper error handling and session management:

Starter code that scrapes Hacker News for top 5 stories using Steel's Python SDK and Selenium.

To run it:

• Add your STEEL_API_KEY to the secrets pane. It's located under "Tools" on the left hand pane.

• Hit Run

The example includes:

• Complete session configuration options

• Error handling best practices

• A working Hacker News scraper example

You can also clone it on Github or Replit to start editing it yourself!