STT: From Audio to Tokens

+15 Mana ✨
Hermes Voice Mode(DOCUMENTATION)

Introduction

Speech-to-text is the front door of voice mode. Everything downstream depends on the transcript it produces. If you understand what STT actually does (and what it cannot do), you will set realistic expectations and configure it well.

Key Concepts

  • STT (speech-to-text): The process of converting recorded audio into a text transcript.
  • Whisper: The speech recognition model family Hermes builds on. faster-whisper is the local, optimized implementation Hermes ships.
  • Model size: Whisper comes in sizes (tiny, base, small, medium, large-v3) that trade accuracy against speed and memory.
  • Transcript: The plain text STT hands to the LLM. From the agent's point of view it is indistinguishable from typed input.

Real World Context

A developer dictates "add a null check to the parser" and the agent acts on "add a null check to the parser." The voice never reached the LLM. By the time the agent saw the request, it was already text. This is why a course on voice spends real time on STT: the transcript is the only thing the agent ever sees, so transcription quality is the ceiling on everything else.

Deep Dive

Here is the journey from sound to tokens.

You speak. The microphone records a waveform. Hermes detects when you have stopped (silence detection, covered later) and hands the recorded clip to the STT engine. The engine runs a Whisper model over the audio and emits a string. That string is injected into the conversation exactly as if you had typed it.

The local provider uses faster-whisper, which you install separately with pip install faster-whisper (the hermes-agent[voice] extra installs audio capture and playback, not the transcription engine). It is a re-implementation of OpenAI's Whisper optimized for speed and lower memory use, and it runs entirely on your machine with no API key. The default model is base, which is around 150 MB on disk: small enough to download quickly, accurate enough for clear speech.

The model sizes form a ladder:

text
tiny     -> fastest, smallest, least accurate
base     -> Hermes default, good balance (~150 MB)
small    -> noticeably better accuracy, slower
medium   -> strong accuracy, heavier
large-v3 -> best accuracy, slowest, most memory

Moving up the ladder buys accuracy and costs speed and memory. There is no universally correct rung. base is the sensible default; you climb only if your speech is being mistranscribed and you have the hardware budget.

The key mental model: STT is lossy translation, not perfect capture. It guesses. It guesses very well for clear speech in a quiet room, and less well for accents, jargon, fast speech, or background noise. Because the transcript is all the LLM receives, an STT error is indistinguishable from you having said the wrong thing. The agent cannot recover information that STT never captured.

Common Pitfalls

  1. Expecting perfect transcription: STT is a probabilistic guess. Plan for occasional errors and design your voice habits (covered later) to recover from them gracefully.
  2. Defaulting to the largest model: large-v3 is slow and memory-hungry. Start with base and climb only if accuracy is genuinely failing you.

Best Practices

  1. Start at base, climb on evidence: Use the default model until you observe real mistranscriptions, then move one rung up rather than jumping to the top.
  2. Remember the transcript is the contract: The agent acts on text, not sound. Treat a clear transcript as the goal of the STT stage.

Summary

  • STT converts recorded audio into a text transcript the LLM treats as typed input.
  • The local provider uses faster-whisper (installed separately with pip install faster-whisper); base is the default model.
  • Model sizes trade accuracy against speed and memory; base is the balanced default.
  • STT is lossy. The transcript is the only thing the agent sees, so transcription quality caps everything downstream.

Code Examples

yaml
# ~/.hermes/config.yaml
stt:
  provider: "local"      # faster-whisper (separate: pip install faster-whisper)
  local:
    model: "base"        # tiny | base | small | medium | large-v3
# 'base' is ~150 MB and balances accuracy against speed
✓ Completed