Voice Mode Architecture: STT to LLM to TTS

+15 Mana ✨
Hermes Voice Mode(DOCUMENTATION)

Introduction

Voice mode feels like one feature, but it is really three independent stages chained together. Understanding the chain is what lets you debug it, tune it, and predict its latency. When something goes wrong, you will not ask "is voice broken?" You will ask "which stage broke?"

Key Concepts

  • STT (speech-to-text): The stage that turns recorded audio into a text transcript. Hermes uses Whisper-based engines for this.
  • LLM: The unchanged core of Hermes. It receives text and produces text, exactly as it does when you type.
  • TTS (text-to-speech): The stage that turns the agent's text reply into spoken audio.
  • Pipeline: The ordered path audio takes: microphone, STT, LLM, TTS, speaker.

Real World Context

A developer reports that voice mode "does not work." That description is useless until you locate the stage. If Hermes transcribes your words correctly but stays silent, STT is fine and TTS is the suspect. If it never reacts to speech at all, the microphone or STT is the suspect. The pipeline mental model turns a vague complaint into a precise diagnosis.

Deep Dive

Here is the full path a single voice turn takes:

text
You speak
     |
  [ Microphone ]  captures raw audio
     |
  [ STT ]         Whisper transcribes audio into text
     |
  [ LLM ]         the Hermes agent reasons over the text
     |
  [ TTS ]         the reply text is synthesized into audio
     |
  [ Speaker ]     you hear the response

The critical insight: the LLM stage is identical to text mode. Hermes does not run a special "voice agent." STT is a front door that converts speech into the same text the keyboard would have produced, and TTS is a back door that converts the reply into audio. Everything in between (tools, memory, skills, personality) behaves exactly as it always does.

That design has three consequences worth internalizing.

First, STT and TTS are decoupled. You can run speech-to-text without speaking the reply back, or vice versa. Hermes exposes this directly: voice input can be on while spoken output is off.

Second, latency is additive. Each stage adds its own delay. Total response time is roughly STT time plus LLM time plus TTS time. If a turn feels slow, you can attack whichever stage dominates rather than blaming "voice" as a whole.

Third, each stage has its own provider choice. STT has its own set of engines, TTS has another, and they are configured separately. A slow STT engine and a fast TTS engine is a perfectly valid combination.

Because the stages are independent, you tune them independently. A noisy room is an STT problem. A robotic-sounding reply is a TTS problem. A wrong answer is an LLM problem and has nothing to do with voice at all.

One setup consequence follows from this bolt-on design: voice is not part of base Hermes. It ships as an optional install extra that you add explicitly when you want speech in and speech out, rather than something bundled by default.

Common Pitfalls

  1. Treating voice as a monolith: "Voice is broken" hides which of three stages failed. Always name the stage before debugging.
  2. Assuming STT and TTS are linked: They are separate. You can transcribe speech without speaking replies, which is often the right default.

Best Practices

  1. Debug stage by stage: When voice misbehaves, confirm the transcript first (STT), then the reply text (LLM), then the audio (TTS). Isolate before you fix.
  2. Budget latency per stage: If turns feel slow, measure which stage dominates and tune that one, rather than switching the whole feature off.

Summary

  • Voice mode is a three-stage pipeline: STT, then the LLM, then TTS.
  • The LLM stage is identical to text mode. STT and TTS only translate at the edges.
  • STT and TTS are decoupled and configured separately.
  • Latency is additive, so debug and tune one stage at a time.

Code Examples

yaml
# ~/.hermes/config.yaml
# Each pipeline stage has its own independent block
stt:
  provider: "local"      # speech-to-text engine
  local:
    model: "base"
tts:
  provider: "edge"       # text-to-speech engine, chosen separately
  edge:
    voice: "en-US-AriaNeural"
✓ Completed