Server Routing

Introduction

With multiple MCP servers connected, your application needs to route tool calls to the correct server, monitor connection health, and handle failures gracefully. This lesson covers building a server registry, directing tool calls, implementing health checks, reconnection strategies, and graceful degradation.

Key Concepts

• Server Registry: A data structure that maps tool names (or server identifiers) to their corresponding Client instances, enabling correct routing of tool calls.
• Health Checking: Periodically verifying that server connections are alive and responsive, typically by sending a lightweight request like listTools().
• Reconnection Strategy: Automatically re-establishing a connection when a server becomes unavailable, using patterns like exponential backoff.
• Graceful Degradation: Continuing to function with reduced capabilities when one or more servers are down, rather than failing entirely.

Real World Context

A production assistant connects to five MCP servers. During normal operation, the GitHub server crashes and restarts. The application detects the disconnection, removes GitHub tools from the available set, notifies the user that GitHub features are temporarily unavailable, and begins reconnection attempts with exponential backoff. When the server comes back, tools are re-registered and the user is notified that full functionality is restored.

Deep Dive

A server registry centralizes the mapping between tools and their owning clients. It also tracks server status.

The following code implements a basic server registry:

typescript
interface ServerEntry {
  client: Client;
  config: ServerConfig;
  tools: string[];
  status: 'connected' | 'disconnected' | 'reconnecting';
}

class ServerRegistry {
  private servers = new Map<string, ServerEntry>();
  private toolToServer = new Map<string, string>();

  register(name: string, entry: ServerEntry) {
    this.servers.set(name, entry);
    for (const tool of entry.tools) {
      this.toolToServer.set(tool, name);
    }
  }

  getClientForTool(toolName: string): Client | null {
    const serverName = this.toolToServer.get(toolName);
    if (!serverName) return null;
    const entry = this.servers.get(serverName);
    if (!entry || entry.status !== 'connected') return null;
    return entry.client;
  }

  getAvailableTools(): string[] {
    const tools: string[] = [];
    for (const [name, entry] of this.servers) {
      if (entry.status === 'connected') {
        tools.push(...entry.tools);
      }
    }
    return tools;
  }
}

The registry tracks each server's status and only returns tools from connected servers. The getClientForTool method returns null for tools whose server is unavailable.

Routing tool calls through the registry ensures calls go to the correct server.

Here is a routing function that uses the registry:

typescript
async function routeToolCall(
  registry: ServerRegistry,
  toolName: string,
  args: Record<string, unknown>
) {
  const client = registry.getClientForTool(toolName);
  if (!client) {
    return {
      content: [{ type: 'text', text: `Tool '${toolName}' is currently unavailable.` }],
      isError: true
    };
  }

  try {
    return await client.callTool({ name: toolName, arguments: args });
  } catch (error) {
    return {
      content: [{ type: 'text', text: `Error calling ${toolName}: ${error.message}` }],
      isError: true
    };
  }
}

The function handles both missing servers (returning an error result) and call failures (catching exceptions and returning a structured error).

Health checking verifies that connections are still alive. A simple approach pings each server periodically.

The following code sets up periodic health checks:

typescript
async function healthCheck(registry: ServerRegistry) {
  for (const [name, entry] of registry.servers) {
    if (entry.status !== 'connected') continue;
    try {
      await entry.client.listTools();
    } catch (error) {
      console.error(`Server ${name} health check failed:`, error.message);
      entry.status = 'disconnected';
      scheduleReconnect(name, entry, registry);
    }
  }
}

// Run health checks every 30 seconds
setInterval(() => healthCheck(registry), 30_000);

If a listTools() call fails, the server is marked as disconnected and a reconnection is scheduled.

Reconnection uses exponential backoff to avoid overwhelming a struggling server.

Here is a reconnection function with backoff:

typescript
async function scheduleReconnect(
  name: string,
  entry: ServerEntry,
  registry: ServerRegistry,
  attempt = 1
) {
  entry.status = 'reconnecting';
  const delay = Math.min(1000 * Math.pow(2, attempt), 30_000);

  setTimeout(async () => {
    try {
      const newClient = new Client({ name: 'app', version: '1.0.0' });
      await newClient.connect(
        new StdioClientTransport({
          command: entry.config.command,
          args: entry.config.args
        })
      );
      const { tools } = await newClient.listTools();
      entry.client = newClient;
      entry.tools = tools.map(t => t.name);
      entry.status = 'connected';
      registry.register(name, entry);
      console.error(`Reconnected to ${name}`);
    } catch {
      scheduleReconnect(name, entry, registry, attempt + 1);
    }
  }, delay);
}

The backoff starts at 2 seconds and doubles up to a maximum of 30 seconds. On successful reconnection, the server's tools are refreshed in case they changed.

Common Pitfalls

1. Not cleaning up disconnected server tools: When a server goes down, its tools must be removed from the available set. Otherwise the LLM will attempt to call unavailable tools.
2. Unbounded reconnection attempts: Without a maximum retry limit or backoff cap, reconnection logic can consume excessive resources.
3. Health check intervals too aggressive: Checking every second wastes resources and can overwhelm servers. Every 30 seconds is a reasonable default for most applications.

Best Practices

1. Always route tool calls through a centralized registry rather than maintaining ad-hoc client references throughout your code.
2. Implement graceful degradation by informing the LLM which tools are temporarily unavailable instead of silently dropping them.
3. Refresh the tool list after reconnection since the server may have been updated while it was down.

Summary

• A server registry maps tool names to Client instances and tracks connection status.
• Route all tool calls through the registry for consistent error handling and correct server targeting.
• Implement periodic health checks using lightweight calls like listTools() to detect failures.
• Use exponential backoff for reconnection to avoid overwhelming recovering servers.
• Gracefully degrade by removing unavailable tools and informing the LLM, rather than crashing on failures.

class ServerRegistry {
  private servers = new Map<string, ServerEntry>();
  private toolToServer = new Map<string, string>();

  register(name: string, entry: ServerEntry) {
    this.servers.set(name, entry);
    for (const tool of entry.tools) {
      this.toolToServer.set(tool, name);
    }
  }

  getClientForTool(toolName: string): Client | null {
    const serverName = this.toolToServer.get(toolName);
    if (!serverName) return null;
    const entry = this.servers.get(serverName);
    if (!entry || entry.status !== 'connected') return null;
    return entry.client;
  }
}

async function scheduleReconnect(name, entry, registry, attempt = 1) {
  entry.status = 'reconnecting';
  const delay = Math.min(1000 * Math.pow(2, attempt), 30_000);
  setTimeout(async () => {
    try {
      const newClient = new Client({ name: 'app', version: '1.0.0' });
      await newClient.connect(
        new StdioClientTransport({ command: entry.config.command, args: entry.config.args })
      );
      entry.client = newClient;
      entry.status = 'connected';
    } catch {
      scheduleReconnect(name, entry, registry, attempt + 1);
    }
  }, delay);
}