Decision Tree: When Each Web and Media Tool Wins

+15 Mana ✨
Hermes Tools Overview(DOCUMENTATION)

Introduction

Five tools (web_search, web_extract, the browser toolset, vision_analyze, image_generate) cover most of the web and media surface in Hermes. They overlap enough to be confusing. A short decision tree fixes that.

Key Concepts

  • Decision tree: A sequence of yes or no questions whose leaves are tools.
  • Cheapest first: Always try the lower-overhead tool before escalating. Reads before writes, extract before browser.
  • Capability fit: Sometimes only one tool can do the thing (only the browser can log in). The tree shortcuts straight to that tool when the requirement is clear.

Real World Context

In practice, the choice between these five tools is the second-most-common decision the model makes (after file or shell). Codifying it as a tree saves time and tokens.

Deep Dive

A usable decision tree:

text
Goal involves an image you already have?
 └─ yes → vision_analyze

Goal is to produce a new image from a prompt?
 └─ yes → image_generate

Goal involves a web page or PDF?
 ├─ Do you know the exact URL?
 │   ├─ no  → web_search (then web_extract on a result)
 │   └─ yes ↓
 │
 ├─ Does the page require JS, login, or interaction?
 │   ├─ no  → web_extract
 │   └─ yes → browser toolset (navigate, snapshot, click, type, vision)

Goal involves audio output?
 └─ yes → text_to_speech

The ordering matters. Asking is this an image? first short-circuits the rest of the tree for image work. Asking do you know the URL? before does it need JS? saves a web_search call when the user already supplied the URL.

Three higher-order tips:

  1. Trust the agent on web_extract vs browser: If web_extract returns suspiciously short content (like a SPA shell), the agent should escalate to the browser on the next step. Building this fallback into the prompt or instructions can be helpful.
  2. Prefer vision_analyze over browser_vision when you already have the image: Skipping the browser saves overhead.
  3. For research-heavy sessions, expect a search and extract chain: One search per topic, one or more extracts on the best results. That is the rhythm.

Common Pitfalls

  1. Skipping web_extract and jumping to the browser: Most pages are static enough for web_extract. Try it first.
  2. Running web_search when the URL is already in the conversation: Wasted call, wasted tokens.

Best Practices

  1. Think in cost order: vision_analyze and web_extract are cheap. The browser is expensive. image_generate has real per-call cost.
  2. Let the agent escalate when needed: A good prompt allows the agent to fall back to the browser if web_extract clearly fails.

Summary

  • The web and media tools form a small decision tree.
  • The cheapest tool that can answer the question is usually the right one.
  • Browser is the heavyweight; reach for it only when JS, login, or interaction is required.
  • Match the tool to whether the artifact already exists (analyze) or needs to be created (generate).

Code Examples

text
User: 'what changed in Next.js 16?'
  step 1: web_search('Next.js 16 changes')
  step 2: web_extract(top result URL)
  step 3: final text response

User: 'log into the dashboard and screenshot the user table'
  step 1: browser_navigate(dashboard URL)
  step 2: browser_type(username, password)
  step 3: browser_click(submit)
  step 4: browser_navigate(/users)
  step 5: browser_vision('extract the user table')

User: 'describe this screenshot' (with attached PNG)
  step 1: vision_analyze(image=PNG, question='describe this screenshot')
  step 2: final text response
✓ Completed