The Browser Tool: Multi-Backend

+15 Mana ✨
Hermes Browser Tools Reference(DOCUMENTATION)

Introduction

When web_extract is not enough, Hermes can drive a real browser. The browser toolset includes navigate, snapshot, click, type, scroll, vision, and more. What is interesting is that this toolset is not tied to one browser engine: Hermes supports multiple backends, and you choose based on where the work runs and how much realism you need.

Key Concepts

  • Browser toolset: A bundle of tools (browser_navigate, browser_snapshot, browser_click, browser_type, browser_vision, ...) that together let the agent operate a web browser.
  • Backend: The actual browser the toolset drives. Options include Browserbase (managed cloud), Browser Use (agent-friendly headless), and local Chrome or Chromium via CDP.
  • CDP: Chrome DevTools Protocol. Some advanced browser tools (raw CDP, dialog handling) require a CDP-capable backend.

Real World Context

If you ask Hermes to log into the staging dashboard and screenshot the user table, that is browser work. web_extract cannot log in. The agent uses browser_navigate to open the login page, browser_type to enter credentials, browser_click to submit, then browser_snapshot and browser_vision to capture the result. The choice of backend then decides whether this happens in a managed cloud sandbox, a headless browser, or a real Chrome window on your machine.

Deep Dive

The browser toolset is the heaviest in Hermes by step count: a single user request can produce ten or twenty browser calls (navigate, snapshot, click, type, scroll, repeat). That is the cost of working through a real UI. So the browser is the wrong tool until you are sure you need it.

Backend trade-offs:

  • Browserbase: Managed cloud browsers. No local install, scales horizontally, good for production gateways. You pay per session.
  • Browser Use: An agent-tuned browser environment. Friendly to LLM-driven control. Good for cloud or hosted scenarios.
  • Local Chrome or Chromium: Drives a real browser via CDP on your machine. Best for development, login flows that need your existing cookies, and debugging.

The browser toolset itself is the same in each case. What changes is where the page is rendered and what credentials are available. CDP-only tools (browser_cdp, browser_dialog) require a backend that exposes a CDP endpoint, which currently means local Chrome or a Browserbase session configured for it.

When the browser earns its overhead:

  • The page is a single-page app and web_extract returns an empty shell.
  • The work requires logging in or interacting with a form.
  • You need to see the page visually (use browser_vision after browser_snapshot).
  • You are reproducing a user-facing flow end to end.

When it does not:

  • The page is static and the content is in the HTML. web_extract is faster.
  • You only need a list of search results. web_search is cheaper.
  • You need an image described. vision_analyze works directly without launching a browser.

Common Pitfalls

  1. Defaulting to browser: A 30-step browser run when a 1-step web_extract would do is the most common waste in agent sessions.
  2. Mismatched backend for the task: Trying to use local Chrome for a production gateway is brittle. Trying to use Browserbase for read my logged-in dashboard at localhost:3000 does not work because the sandbox cannot reach your machine.

Best Practices

  1. Escalate, do not default: Try web_extract first. Reach for the browser only when it is the wrong answer.
  2. Match the backend to the deployment: Local for dev, Browserbase or Browser Use for hosted gateways. Pick once, configure clearly, do not switch mid-session.

Summary

  • The browser toolset is a set of tools (navigate, snapshot, click, type, vision, ...) that drive a real browser.
  • Hermes supports multiple backends: Browserbase, Browser Use, local Chrome or Chromium.
  • The tools are identical across backends; the backend changes where the page is rendered.
  • Use the browser when content needs JavaScript, login, or interaction. Otherwise prefer web_extract.

Code Examples

yaml
# A short browser session: open a SPA, snapshot, click
tool_call_1:
  name: browser_navigate
  arguments:
    url: https://app.example.com/dashboard

tool_call_2:
  name: browser_snapshot
  # returns an accessibility tree with ref IDs for interactive elements

tool_call_3:
  name: browser_click
  arguments:
    ref: ref-1234  # from the snapshot above

tool_call_4:
  name: browser_vision
  # screenshot the page and analyze it visually
✓ Completed