Cloud vs Local: The First Fork

+15 Mana ✨
Hermes Integrations(DOCUMENTATION)

Introduction

Before any other provider decision, you will face one fork: cloud or local. Both are first-class in Hermes, every other tool, gateway, and skill behaves the same regardless of which side of the fork you choose. But the operational reality is profoundly different, and picking the wrong side is the most expensive setup mistake to undo later.

Key Concepts

  • Cloud provider: Hermes calls a hosted API. Examples include Nous Portal (Nous Research's own endpoint, optimized for the Hermes models), Anthropic (the first-party API for the Claude family), OpenAI (the first-party GPT API), and OpenRouter (an aggregator that brokers requests across many providers behind one key). You pay per token; latency depends on the network.
  • Local provider: Hermes calls a model running on your own hardware. Examples include Ollama (the most common local runtime, abstracting model downloads), llama.cpp, vLLM, or any self-hosted endpoint. You pay for hardware; latency depends on your GPU.
  • Hybrid: Many users combine, a strong cloud model for hard tasks, a local model for routine ones.

Real World Context

A developer working on a laptop with a 16GB consumer GPU can run small local models for everyday Q&A but hits a wall on hard reasoning. The same developer keeps a cloud provider configured for the moments local cannot carry the weight. From the user's perspective, they are talking to "Hermes", Hermes is just routing under the hood.

Deep Dive

Cloud, the practical default

text
Pros                                  Cons
─────────────────────────────────     ───────────────────────────────────────
Best capability available             Each token has a price
No hardware requirement                Prompts leave your machine
Constant updates / new models          Subject to rate limits and outages

Local, the privacy default

text
Pros                                  Cons
─────────────────────────────────     ───────────────────────────────────────
No data leaves your machine            Requires capable hardware
Marginal cost per token = 0            Capability gap vs frontier models
No external rate limits                Slower first-token on small machines

How Hermes makes the choice manageable

Hermes raises socket read timeouts for local providers automatically (from 120s to 1800s) because local prefill can be slow. It also lets you mix providers in a single fallback chain, if a cloud provider is rate-limited, Hermes can fall back to a local one.

Common Pitfalls

  1. Choosing local for privacy without measuring the capability gap, A privacy gain is worth nothing if the agent cannot do the work. Test on your real tasks first.
  2. Choosing cloud for speed without checking your network, Cloud first-token latency depends on geography. From a remote region, local can win.

Best Practices

  1. Start cloud, add local later, Cloud gets you to a working agent fastest. Add a local fallback once you know what you would offload.
  2. Measure on your own tasks, A 10-prompt benchmark on your actual workload tells you more than any marketing leaderboard.

Summary

  • Cloud vs local is the first provider fork; both are first-class in Hermes.
  • Cloud wins on capability and time-to-running; local wins on privacy and per-token cost.
  • Hermes is built for hybrid setups, the choice is rarely binary.
  • Measure on your own prompts before committing.

Code Examples

yaml
# A hybrid setup: cloud for hard turns, local for everyday burn
providers:
  - id: anthropic         # cloud, capability-heavy
    model: claude-sonnet-4
  - id: ollama-local      # local, privacy-heavy
    model: hermes-3-llama-8b
    timeout_s: 1800       # Hermes raises this automatically for local
fallback_chain: [anthropic, ollama-local]
✓ Completed