web_extract: For Targeted Pulls

+15 Mana ✨
Hermes Web Tools Reference(DOCUMENTATION)

Introduction

Once the agent has a URL it wants to read, web_extract is the right tool. It fetches the page, converts it to markdown, and returns it to the model. For most static pages and PDFs, web_extract is faster, cheaper, and more reliable than launching a browser.

Key Concepts

  • Targeted pull: Fetching a known URL because you want its content, not a list of candidates.
  • Markdown conversion: web_extract strips boilerplate (nav bars, ads) and returns a clean markdown view of the article.
  • Summarization threshold: Small pages (under roughly 5000 characters) return as full markdown. Larger pages may be LLM-summarized to keep them in context.

Real World Context

The most common Hermes workflow involving the web is: user asks a question, agent runs web_search to find candidate URLs, then runs web_extract on the top result to pull the actual content. web_extract is the workhorse. It also handles PDFs, which is useful for research papers, datasheets, and changelogs.

Deep Dive

web_extract accepts a URL and returns markdown. Its behavior:

  • HTML pages: Stripped of navigation, ads, and most non-article chrome. The result is a clean prose view.
  • PDFs: Parsed and returned as markdown. Tables become tables, headings become headings.
  • Large pages: If the markdown exceeds the threshold, Hermes summarizes via an auxiliary LLM call so the result fits in context. You can disable this by passing the right argument, but it is on by default.

When web_extract is the right tool:

  • The page is static (the content is in the HTML, not painted in by JavaScript).
  • You do not need to log in or interact with the page.
  • You want the article content, not a screenshot.

When web_extract is the wrong tool:

  • The page is a SPA where content only appears after JS runs.
  • You need to authenticate, fill a form, or click through a UI.
  • You need to see the page (visual layout matters).

In those cases, the browser tools (next lesson) are the right escalation.

Common Pitfalls

  1. Reaching for the browser too soon: Many pages that look interactive are actually static. Try web_extract first. It is faster, cheaper, and produces cleaner context.
  2. Ignoring the summarization: For long pages, the agent may receive a summary instead of the raw markdown. If precision matters (citing exact wording, parsing a table), pass the option to skip summarization.

Best Practices

  1. Pair with web_search for discovery, then web_extract for the read: The standard pattern.
  2. Use directly when a URL is already known: Skip the search step when the user supplies a URL.

Summary

  • web_extract fetches a URL and returns clean markdown.
  • Works on HTML pages and PDFs.
  • Faster, cheaper, and cleaner than the browser tools for static content.
  • Long pages may be summarized to fit context.

Code Examples

yaml
# Pulling a specific page's content as markdown
tool_call:
  name: web_extract
  arguments:
    url: https://hermes-agent.nousresearch.com/docs/user-guide/features/tools

# Result is markdown, stripped of nav and ads, ready to read.
# If the page is small enough, full content is returned.
# Larger pages may return an LLM-generated summary.
✓ Completed