Verifying End to End: A Mental Model of Failure Points

+15 Mana ✨
Use Voice Mode with Hermes(DOCUMENTATION)

Introduction

When voice mode does not work, the temptation is to reinstall everything and hope. A better approach: know the handful of places voice can fail, and check them in order. Most voice problems are not in the agent at all. They are in the audio plumbing around it.

Key Concepts

  • System dependency: An OS-level package the voice pipeline relies on, separate from the Hermes Python package. Examples include portaudio and ffmpeg.
  • Audio device: The microphone and speaker the operating system exposes to Hermes.
  • Failure point: A specific stage where a voice turn can break: capture, transcription, reasoning, or playback.
  • End-to-end check: Confirming a single full turn works before trusting voice for real work.

Real World Context

A developer installs the voice extra, starts Hermes, presses the record key, and nothing happens. They assume voice mode is buggy. The real cause: portaudio was never installed, so Hermes has no way to reach the microphone. The Python package installed fine; the OS-level audio library did not. Knowing the failure points turns a frustrating dead end into a thirty-second fix.

Deep Dive

Voice mode depends on more than the Hermes package. It needs OS-level audio libraries. On macOS that typically means portaudio and ffmpeg; on Debian or Ubuntu the equivalents are portaudio19-dev and ffmpeg. These are not optional extras. Without them the pipeline has no working audio path.

Walk the pipeline and name what can fail at each point:

Capture. No audio device found, missing portaudio, the wrong microphone selected, or the OS denying microphone permission to the terminal. Symptom: pressing the record key does nothing, or Hermes never reacts to speech.

Transcription (STT). The local engine was never installed (faster-whisper is a separate pip install, not part of the [voice] extra), the Whisper model failed to download, or a cloud STT provider is configured but its API key is missing. Symptom: recording happens but no transcript appears.

Reasoning (LLM). This is ordinary Hermes. If the transcript is correct but the answer is wrong, voice is working fine and the problem is a normal agent issue.

Playback (TTS). No reply audio because ffmpeg is missing, a cloud TTS key is absent, or the audio output device is wrong. Symptom: the transcript and reply text are both correct but you hear nothing.

The verification habit is simple: after install, run one full turn before trusting voice for real work. Speak a short sentence, confirm the transcript matches, confirm the agent replies, and confirm you hear it. If any of those four checks fails, you have already localized the problem to one stage.

Think of it as a checklist that runs left to right. The first failing check is your bug. There is no need to look further right until you have fixed it.

Common Pitfalls

  1. Forgetting OS-level audio dependencies: The voice extra installs the Python side only. Missing portaudio or ffmpeg breaks capture or playback even though pip reported success.
  2. Skipping the end-to-end check: Trusting voice before running one verified turn means you discover the broken stage mid-task instead of at setup time.

Best Practices

  1. Install system dependencies alongside the extra: Treat portaudio and ffmpeg as part of voice setup, not an afterthought.
  2. Verify left to right: Check capture, then transcript, then reply, then playback. The first failing check is the bug; fix it before moving on.

Summary

  • Voice mode needs OS-level audio libraries (portaudio, ffmpeg) beyond the Python package.
  • The failure points map to pipeline stages: capture, transcription, reasoning, playback.
  • A correct transcript with a wrong answer is an ordinary LLM issue, not a voice issue.
  • Run one full verified turn after install, checking each stage left to right.

Code Examples

bash
# macOS: install the OS-level audio libraries voice mode needs
brew install portaudio ffmpeg

# Ubuntu/Debian equivalent
# sudo apt install portaudio19-dev ffmpeg

# Local STT engine (separate from the [voice] extra)
pip install faster-whisper

# Then run one full turn and check each stage:
#  1. capture   -> did recording start?
#  2. transcript-> does the text match what you said?
#  3. reply     -> did the agent answer?
#  4. playback  -> did you hear it?
✓ Completed