Resilience: Fallback Chains and Credential Pools

+15 Mana ✨
Configuration: Fallback and Credential Pools(DOCUMENTATION)

Introduction

Cloud providers go down. Keys hit rate limits. Networks blip. A real Hermes setup needs to keep working through all of this, and Hermes ships two complementary mechanisms that handle it: fallback chains for cross-provider failover and credential pools for in-provider rotation. Configured well, your agent feels uninterrupted even when the underlying providers are misbehaving.

Key Concepts

  • Fallback chain: An ordered list of providers Hermes tries when the primary fails after retries.
  • Credential pool: Multiple API keys for the same provider, rotated by a strategy.
  • hermes fallback: The CLI subcommand that manages your fallback chain (add, remove, clear).
  • hermes auth: The CLI subcommand that manages credentials and pools (add, list, remove, status).

Real World Context

You are in the middle of a debug session and your primary provider returns 429 (rate limit). With no resilience, the agent stalls and you context-switch out, never to come back. With a credential pool, Hermes silently switches to your second key. With a fallback chain, if the whole provider is down, Hermes switches to a backup provider mid-message. The user experience is "the agent kept working." The configuration was a one-time setup.

Deep Dive

Credential pools, same provider, multiple keys

A credential pool is for load balancing within one provider. Strategies include:

text
Strategy      Behavior
────────────  ──────────────────────────────────────────────
fill_first    Use key A until exhausted, then B, then C (default)
round_robin   Cycle through keys evenly
least_used    Always pick the least-utilized key
random        Pick randomly each call

You add credentials to a pool with:

bash
hermes auth add openrouter      # adds another key to the openrouter pool
hermes auth list                # show pools and per-key status

Fallback chains, different providers

A fallback chain is for failover across providers. When the primary fails after agent.api_max_retries (default 3), Hermes moves to the next provider in the chain.

bash
hermes fallback add anthropic openrouter ollama
hermes fallback list

The order matters: cheaper or faster fallbacks come first.

How they compose

A typical resilient setup combines both:

  • Primary: Anthropic, with a 3-key credential pool (round_robin).
  • Fallback 1: OpenRouter, with a 2-key pool.
  • Fallback 2: Local Ollama (no key needed).

If Anthropic is rate-limited on one key → pool rotates. If all keys are limited → fallback fires. If OpenRouter is down → local model takes over.

Common Pitfalls

  1. Confusing pools with chains, Pools rotate keys for one provider; chains switch between providers. You typically need both.
  2. Putting expensive providers as fallbacks, A fallback that costs 10x the primary will bite you on a bad day. Order chains by what you are willing to spend.

Best Practices

  1. Always have at least one fallback, Even a local model is useful when the cloud is down.
  2. Test by simulating failure, Pause your primary provider's billing for a test session and watch the fallback fire. It is the only way to know the chain works before you need it.

Summary

  • hermes fallback manages cross-provider failover.
  • hermes auth manages credential pools for in-provider key rotation.
  • Pools handle rate limits within a provider; chains handle the provider being unavailable.
  • A mature setup composes both, your agent keeps working through real-world failures.

Code Examples

bash
# Build a resilient provider config from scratch
hermes auth add anthropic            # primary key #1
hermes auth add anthropic            # primary key #2 (pool)
hermes auth add openrouter           # fallback provider key

hermes fallback add anthropic openrouter ollama
hermes fallback list
# 1. anthropic    (pool: 2 keys, strategy=round_robin)
# 2. openrouter   (pool: 1 key)
# 3. ollama       (local, no key needed)
✓ Completed