Local vs Cloud: Privacy, Cost, Latency

+15 Mana ✨
Use Voice Mode with Hermes(DOCUMENTATION)

Introduction

Every voice stage offers a choice between running on your own machine and calling a cloud service. This is not a quality ranking. It is a trade-off across three axes: privacy, cost, and latency. Choosing well means knowing which axis matters most for your situation.

Key Concepts

  • Local provider: An engine that runs entirely on your hardware. faster-whisper is the recommended local STT engine (a separate pip install faster-whisper), and NeuTTS is a local TTS option.
  • Cloud provider: A hosted service reached over the network, such as Groq or OpenAI for STT, or ElevenLabs for TTS.
  • The three axes: Privacy (does your audio leave the machine), cost (per-use billing versus free), and latency (network round trip versus local compute).
  • API key: A credential a cloud provider requires. Local providers need none.

Real World Context

A contractor working on a client codebase under NDA cannot send recordings of themselves discussing that code to a third-party API. For them, local is not a preference, it is a requirement. A different developer on a slow laptop finds that local Whisper transcription takes several seconds per turn, and a cloud engine that responds in half a second transforms the experience. Both made the right call. They were optimizing different axes.

Deep Dive

Look at each axis honestly.

Privacy. Local providers keep audio on your machine. Nothing is uploaded, no third party sees or stores your speech, and there is no API to leak a key for. Cloud providers, by definition, transmit your audio off the device. For sensitive work this single axis can decide everything.

Cost. Local providers are free to run after the one-time model download. There is no per-minute meter. Cloud STT and premium cloud TTS bill per use. For light personal use the cost is negligible, but for heavy or automated use it adds up, and it is a recurring dependency on a billing relationship.

Latency. This is the axis where the answer is not obvious. Local STT latency depends on your hardware: a strong machine transcribes quickly, a weak one lags. Cloud STT latency depends on the network and the provider, and the fastest cloud engines are genuinely fast (on the order of half a second). So cloud can be faster than local on weak hardware and slower than local on strong hardware. There is no universal winner.

The practical heuristic Hermes itself recommends as a starting point: local STT with free Edge TTS. It costs nothing and needs no API keys. Edge TTS is a free cloud voice, so only the reply text leaves the machine, never your audio; local STT keeps your actual speech on-device. You only reach for cloud when a specific axis pushes you there: latency on weak hardware, or a quality requirement that a premium cloud voice satisfies.

One more subtlety. The choice is per stage. You might run local STT for privacy while using a cloud TTS voice because you prefer how it sounds. The axes are evaluated independently for each stage of the pipeline.

Common Pitfalls

  1. Assuming cloud is always faster: On capable hardware, local STT can beat the network round trip of a cloud call. Latency depends on your machine, not on the word "cloud."
  2. Forgetting the privacy cost of cloud: A cloud provider receives your raw audio. For confidential work that disqualifies it regardless of speed or price.

Best Practices

  1. Default to local STT, escalate deliberately: Start with local STT and free Edge TTS. Move a stage to a paid or different provider only when a named axis (latency, quality) demands it.
  2. Decide per stage: Evaluate privacy, cost, and latency separately for STT and for TTS. A mixed setup is normal and often optimal.

Summary

  • Local versus cloud is a trade-off across privacy, cost, and latency, not a quality ranking.
  • Local keeps audio on-device, is free, and needs no API key.
  • Cloud can be faster on weak hardware but transmits your audio and bills per use.
  • Hermes recommends starting with local STT and free Edge TTS, and the choice is made independently for each stage.

Code Examples

yaml
# A mixed setup: local STT for privacy, cloud TTS for voice quality
stt:
  provider: "local"        # audio never leaves the machine
  local:
    model: "base"
tts:
  provider: "elevenlabs"   # premium cloud voice, billed per use
# The two stages are evaluated independently against privacy, cost, latency
✓ Completed