Failover Chains & Resilience

+15 Mana ✨
OpenClaw Failover & Resilience(DOCUMENTATION)

Introduction

Production agents need to handle model outages, rate limits, and billing failures gracefully. OpenClaw provides a failover system with primary and fallback models, exponential cooldown mechanics, and session stickiness to ensure consistent behavior. This lesson covers how to build resilient agent configurations that keep working even when providers go down.

Key Concepts

  • Failover Chain: A configuration that defines a primary model and an ordered list of fallback models to try when the primary is unavailable.
  • Cooldown Mechanics: Exponential backoff timers applied after failures, preventing repeated requests to a failing provider.
  • Session Stickiness: The behavior where an auth profile is pinned to a session until a reset, compaction, or cooldown event occurs.
  • Billing Failure Handling: A separate, more aggressive cooldown schedule applied when failures are caused by billing issues rather than transient errors.
  • Model Fallback Triggers: The specific failure types that cause failover: auth failure, rate limit, and timeout.

Real World Context

A production chatbot serving thousands of users cannot afford downtime when Anthropic's API has an outage. By configuring a failover chain with OpenAI as a fallback, the bot continues serving users seamlessly. The cooldown system prevents hammering a failing provider with retries, while session stickiness ensures a user's conversation stays on the same model and credentials throughout their session for consistency.

Deep Dive

A failover chain is configured in the agent's model settings. Here is a complete resilience configuration:

bash
# View current model configuration
openclaw agent config get code-reviewer --key model

# Output:
# {
#   "primary": "anthropic:claude-sonnet-4-20250514",
#   "fallbacks": [
#     "openai:gpt-4o",
#     "openrouter:anthropic/claude-3-haiku"
#   ]
# }

The configuration above shows a primary model (Claude Sonnet) with two fallbacks. The system tries the primary model first. If the primary fails due to an auth failure, rate limit, or timeout, it triggers the failover chain. These are the only three failure types that trigger failover — other errors like malformed requests do not cause model switching.

Cooldown mechanics use exponential backoff to avoid overwhelming a failing provider. For standard (non-billing) failures, the cooldown schedule is:

bash
# Standard cooldown progression
# Failure 1: 1 minute cooldown
# Failure 2: 5 minutes cooldown
# Failure 3: 25 minutes cooldown
# Failure 4+: 60 minutes cooldown (maximum)

# Check cooldown status for an agent
openclaw agent status code-reviewer --show-cooldowns

# Output:
# Provider: anthropic
# Status: cooling down
# Cooldown remaining: 4m 32s
# Failures: 1
# Next cooldown: 5m

The cooldown starts at 1 minute after the first failure. Each subsequent failure multiplies the cooldown by 5, up to a maximum of 60 minutes (1 hour). This prevents the system from endlessly retrying a provider that is experiencing a sustained outage.

Billing failures receive a much more aggressive cooldown schedule because they indicate a fundamental account issue that will not resolve quickly:

bash
# Billing failure cooldown progression
# Failure 1: 5 hours cooldown (300 minutes)
# Failure 2: 10 hours cooldown
# Failure 3: 20 hours cooldown
# Failure 4+: 24 hours cooldown (maximum)

# Billing failures are identified by specific error codes
# from providers (e.g., 402 Payment Required, billing_error)

Billing failures start with a 5-hour cooldown and double with each subsequent failure, capping at 24 hours. This aggressive schedule reflects the reality that billing issues require human intervention — retrying sooner wastes resources.

Session stickiness ensures that once an auth profile is selected for a session, it remains pinned for the duration. This is important for conversation continuity. The profile stays pinned until one of three events occurs:

bash
# Session stickiness reset triggers:
# 1. Manual reset: openclaw session reset <sessionId>
# 2. Compaction: when conversation history is compressed
# 3. Cooldown: when the pinned profile enters cooldown

# Force reset a session's auth profile
openclaw session reset abc123-session

# Override model for current chat session
/model openai:gpt-4o

The /model command shown above allows users to override the model selection for their current chat session. This is useful for testing or when a user needs a specific model's capabilities for a particular conversation.

When the pinned auth profile enters cooldown due to a failure, the session stickiness is broken and the system selects a new profile following the standard selection priority chain. The new profile then becomes sticky for the remainder of the session.

Common Pitfalls

  • Configuring fallbacks with the same provider: If your primary is Anthropic and all fallbacks are also Anthropic, a provider-wide outage takes down your entire chain. Use different providers for fallbacks.
  • Ignoring billing failure cooldowns: Billing cooldowns are intentionally long. If you see a 5-hour cooldown, fix the billing issue rather than trying to work around it.
  • Assuming all errors trigger failover: Only auth failures, rate limits, and timeouts trigger the failover chain. A malformed request error will not cause model switching.

Best Practices

  • Diversify providers in your fallback chain: Use at least two different providers (e.g., Anthropic primary, OpenAI first fallback, OpenRouter second fallback).
  • Monitor cooldown events: Set up alerts for cooldown activations so you can investigate provider issues proactively.
  • Test your failover chain regularly: Simulate failures in a staging environment to verify that fallbacks work as expected before relying on them in production.

Summary

  • Failover chains define a primary model and ordered fallback models, triggered by auth failures, rate limits, or timeouts.
  • Standard cooldown uses exponential backoff: 1 min, 5 min, 25 min, up to a maximum of 60 minutes.
  • Billing failure cooldowns are more aggressive: starting at 5 hours, doubling up to 24 hours maximum.
  • Session stickiness pins an auth profile to a session until reset, compaction, or cooldown occurs.
  • The /model command allows users to override model selection within a chat session.
✓ Completed